Trends For University Web Site Search Engines

Surveys of Search Engines Used on UK University Web Sites

What search engines are Universities using on their Web sites? This was a question we sought to answer about ten years ago,with the intention of identifying trends and providing evidence which could be used to inform the development of best practices.

Search engines used across UK Universities in 1999

An analysis of the first survey findings was published in Ariadne in September 1999. As can be seen from the accompanying pie chart a significant number (59 of 160 institutions, or 37%) of University Web sites did not provide a search function. Of those that did the three most widely used search engines were ht://Dig (25 sites, 15.6%), Excite (19 sites, 11.9%) and a Microsoft indexing tool (12 sites, 7.7%).

Perhaps the most interesting observation to be made is the diversity of tool which were being used back then.  On addition to the tools I’ve mentioned universities were also using Harvest, Ultraseek, SWISH, Webinator, Netscape, WWWWais and Freefind together with an ever larger number of tools which were in use at a single institution.

The survey was repeated every six months for a number of years. A year after the initial finding had been published there had been a growth in use of the open source ht://Dig application (from 25 to 44 institutions) and a decrease in the number of institutions which did not provide a search function (down from 59 to 37).

This survey, published in July 2001 was also interesting as it provided evidence of a new search engine tool which was starting to be used: Google, which was being used at the following six institutions: Glasgow School of Arts – Lampeter – Leeds – Manchester Business School – Nottingham – St Mark and St John.

Two years later the survey showed that ht://Dig was still popular, showing a slight increase to use across 54 institutions.  However this time the second most popular tool was Google, which was being used in 21 institutions. Interestingly it was note that a small number of institutions were providing access to multiple search engines such as ht://Dig and Google. It was probably around this time that the discussion began as to whether one should use an externally-hosted solution (due to concerns regarding the sustainability of the provider, the loss of administrative control and use of a proprietary solution when open source solutions – particularly ht://Dig – were being widely used across the sector).

These surveys stopped in 2003. However two years later Lucy Anscombe of Thames Valley University carried out a similar survey in order to inform decision-making at her host institution. Lucy was willing to share this information to others in the sector, and so the data has been hosted on the UKOLN Web site, thus providing our most recently survey of search engine usage across UK Universities.

This time we find that Google is now the leading provider across the sector, being used in 44 of the  109 institutions which were surveyed. That figure can be increased of the five institutions which were using the Google Search Appliance are included in the total.

What’s Being Used Today?

A survey of Web site search engines used on Russell Group University Web sites was carried out recently. The results are given below.

	Institution	Search Engine	Search
1	University of Birmingham	Google Search Appliance	Search University of Birmingham for “Search Engine”
2	University of Bristol	ht://Dig	Search University of Bristol for “Search Engine”
3	University of Cambridge	Ultraseek	Search University of Cambridge for “Search Engine”
4	Cardiff University	Google Custom Search	Search Cardiff University for “Search Engine”
5	University of Edinburgh	Google Custom Search	Search University of Edinburgh for “Search Engine”
6	University of Glasgow	Google Custom Search(?)	Search University of Glasgow for “Search Engine”
7	Imperial College	Google	Search Imperial College for “Search Engine”
8	King’s College London	Google	Search KCL for “Search Engine”
9	University of Leeds	Google Search Appliance	Search University of Leeds for “Search Engine”
10	University of Liverpool	Google	Search University of Liverpool for “Search Engine”
11	LSE	Funnelback	Search LSE for “Search Engine”
12	University of Manchester	Google	Search University of Manchester for “Search Engine”
13	Newcastle University	Google Search Appliance	Search Newcastle University of for “Search Engine”
14	University of Nottingham	Google Search Appliance	Search University of Nottingham for “Search Engine”
15	University of Oxford	Google Search Appliance	Search University of Oxford for “Search Engine”
16	Queen’s University Belfast	Google Search Appliance	Search Queen’s University Belfast for “Search Engine”
17	University of Sheffield	Google Search Appliance	Search University of Sheffield for “Search Engine”
18	University of Southampton	Sharepoint	Search University of Southampton for “Search Engine“
19	University College London	Google Search Appliance	Search University College London for “Search Engine”
20	University of Warwick	Sitebuilder	Search University of Warwick for “Search Engine”

In brief 15 Russell Groups institutions (75%) use Google to provide their main institutional Web site search facility, with no other search engine being used more than once.

Note that Google provide a number of solutions including the Google Search Appliance, the Google Mini and the public Google search. Mike Nolan pointed out to me that “you can customise with API or XSLT to make [Google search results] look different” so I have only named a specific solution if this has been given on the Web site or I have been provided with additional information (note that I can update the table if I receive additional information).

Discussion

Over ten years ago there was a large diversity of search engine solutions being used across the sector. The discussions at the time tended to focus on use of open source solutions, with the argument occasionally being made that since ht://Dig was open source there was no need to look any further. There was also a suggestion that the open source Search Maestro solution, developed at Charles University and deployed at the University of Dundee could have an important role to play in the sector.

However in today’s environment it seems that a Google Search solution is now regarded as the safe option and this seems to have been corroborated with a survey carried out by Mike Nolan in December 2008. The potential of Google Custom Search will have been enhanced by the announcement, two days ago, of developments to metadata search capabilities.

There has, however, been some discussion recently on the web-support JISCMail list on software alternatives to the Google Search Appliance.Another discussion on the website-info-mgt JISCMail list has shown some interest in the Funnelback software. But, interestingly, open source solutions has not been mentioned in the discussions.

We might conclude that, in the case of Web site search engines, after ten years of the ‘bazaar’ the sector has moved to Google’s cathedral. What, I wonder, might be the lessons to be learnt from the evidence of the solutions which are used across the sector? Might it be that the HE sector has moved towards cost-effective solutions provided by Google’s free solutions or the richness of the licenced Google Search Appliance or Google Mini? And might this be used to demonstrate that the HE sector has been successful in identifying and deploying cost-effective search solutions?