UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

Trends For University Web Site Search Engines

Posted by Brian Kelly on 15 Dec 2010

Surveys of Search Engines Used on UK University Web Sites

What search engines are Universities using on their Web sites? This was a question we sought to answer about ten years ago,with the intention of identifying trends and providing evidence which could be used to inform the development of best practices.

Search engines used across UK Universities in 1999

An analysis of the first survey findings was published in Ariadne in September 1999. As can be seen from the accompanying pie chart a significant number (59 of 160 institutions, or 37%) of University Web sites did not provide a search function. Of those that did the three most widely used search engines were ht://Dig (25 sites, 15.6%), Excite (19 sites, 11.9%) and a Microsoft indexing tool (12 sites, 7.7%).

Perhaps the most interesting observation to be made is the diversity of tool which were being used back then.  On addition to the tools I’ve mentioned universities were also using Harvest, Ultraseek, SWISH, Webinator, Netscape, WWWWais and Freefind together with an ever larger number of tools which were in use at a single institution.

The survey was repeated every six months for a number of years. A year after the initial finding had been published there had been a growth in use of the open source ht://Dig application (from 25 to 44 institutions) and a decrease in the number of institutions which did not provide a search function (down from 59 to 37).

This survey, published in July 2001 was also interesting as it provided evidence of a new search engine tool which was starting to be used: Google, which was being used at the following six institutions: Glasgow School of Arts – Lampeter – Leeds – Manchester Business School – Nottingham – St Mark and St John.

Two years later the survey showed that ht://Dig was still popular, showing a slight increase to use across 54 institutions.  However this time the second most popular tool was Google, which was being used in 21 institutions. Interestingly it was note that a small number of institutions were providing access to multiple search engines such as ht://Dig and Google. It was probably around this time that the discussion began as to whether one should use an externally-hosted solution (due to concerns regarding the sustainability of the provider, the loss of administrative control and use of a proprietary solution when open source solutions – particularly ht://Dig – were being widely used across the sector).

These surveys stopped in 2003. However two years later Lucy Anscombe of Thames Valley University carried out a similar survey in order to inform decision-making at her host institution. Lucy was willing to share this information to others in the sector, and so the data has been hosted on the UKOLN Web site, thus providing our most recently survey of search engine usage across UK Universities.

This time we find that Google is now the leading provider across the sector, being used in 44 of the  109 institutions which were surveyed. That figure can be increased of the five institutions which were using the Google Search Appliance are included in the total.

What’s Being Used Today?

A survey of Web site search engines used on Russell Group University Web sites was carried out recently. The results are given below.

Institution Search Engine Search
1 University of Birmingham Google Search Appliance Search University of Birmingham for “Search Engine”
2 University of Bristol ht://Dig Search University of Bristol for “Search Engine”
3 University of Cambridge Ultraseek Search University of Cambridge for “Search Engine”
4 Cardiff University Google Custom Search Search Cardiff University for “Search Engine”
5 University of Edinburgh Google Custom Search Search University of Edinburgh for “Search Engine”
6 University of Glasgow Google Custom Search(?) Search University of Glasgow for “Search Engine”
7 Imperial College Google Search Imperial College for “Search Engine”
8 King’s College London Google Search KCL for “Search Engine”
9 University of Leeds Google Search Appliance Search University of Leeds for “Search Engine”
10 University of Liverpool Google Search University of Liverpool for “Search Engine”
11 LSE Funnelback Search LSE for “Search Engine”
12 University of Manchester Google Search University of Manchester for “Search Engine”
13 Newcastle University Google Search Appliance Search Newcastle University of for “Search Engine”
14 University of Nottingham Google Search Appliance Search University of Nottingham for “Search Engine”
15 University of Oxford Google Search Appliance Search University of Oxford for “Search Engine”
16 Queen’s University Belfast Google Search Appliance Search Queen’s University Belfast for “Search Engine”
17 University of Sheffield Google Search Appliance Search University of Sheffield for “Search Engine”
18 University of Southampton Sharepoint Search University of Southampton for “Search Engine
19 University College London Google Search Appliance Search University College London for “Search Engine”
20 University of Warwick Sitebuilder Search University of Warwick for “Search Engine”

In brief 15 Russell Groups institutions (75%) use Google to provide their main institutional Web site search facility, with no other search engine being used more than once.

Note that Google provide a number of solutions including the Google Search Appliance, the Google Mini and the public Google search. Mike Nolan pointed out to me that “you can customise with API or XSLT to make [Google search results] look different” so I have only named a specific solution if this has been given on the Web site or I have been provided with additional information (note that I can update the table if I receive additional information).


Over ten years ago there was a large diversity of search engine solutions being used across the sector. The discussions at the time tended to focus on use of open source solutions, with the argument occasionally being made that since ht://Dig was open source there was no need to look any further. There was also a suggestion that the open source Search Maestro solution, developed at Charles University and deployed at the University of Dundee could have an important role to play in the sector.

However in today’s environment it seems that a Google Search solution is now regarded as the safe option and this seems to have been corroborated with a survey carried out by Mike Nolan in December 2008. The potential of Google Custom Search will have been enhanced by the announcement, two days ago, of developments to metadata search capabilities.

There has, however, been some discussion recently on the web-support JISCMail list on software alternatives to the Google Search Appliance.Another discussion on the website-info-mgt JISCMail list has shown some interest in the Funnelback software. But, interestingly, open source solutions has not been mentioned in the discussions.

We might conclude that, in the case of Web site search engines, after ten years of the ‘bazaar’ the sector has moved to Google’s cathedral. What, I wonder, might be the lessons to be learnt from the evidence of the solutions which are used across the sector? Might it be that the HE sector has moved towards cost-effective solutions provided by Google’s free solutions or the richness of the licenced Google Search Appliance or Google Mini? And might this be used to demonstrate that the HE sector has been successful in identifying and deploying cost-effective search solutions?


13 Responses to “Trends For University Web Site Search Engines”

  1. Tony Hirst said

    Interesting… because the feeling I get from…. dunno… recent confs, feeds, discussion etc is that APache Solr is something to watch…

  2. […] This post was mentioned on Twitter by Brian Kelly and marquibeck, marquibeck. marquibeck said: via @briankelly good data to compare trends in search engine use in UK Universities 1999 vs today […]

  3. Simon Bains said

    What happened to the University of Edinburgh link in the table?

  4. Thanks for the post Brian, this is very interesting.

    For more information on LSE can I point your readers to the following Case Study on the Funnelback site:

    And please watch this space for news on other UK Universities that are currently in-build on Funnelback!

  5. Jon Warbrick said

    I’d make two points:

    1) I think that Google Search Appliance, Google Custom Search and the public Google search actually represent significantly different solutions (not least in terms of up front cost) and so the 75% ‘Google’ slice is somewhat misleading. The results of a better breakdown would be interesting.

    2) I’ve always considered that Open Source has failed to deliver good web search engines because a) they are hard to write (especially if they are to work on an enterprise scale), and b) there’s a sufficiently small market for them that there isn’t a critical mass of developers.

  6. I think Jon make a good point about the nature of Google search tools. You get what you pay for.

    For me, more interesting than the technology is how it’s being used. Do you have any data on how university search engines are being managed? From a user experience point of view, that is.

    Even the smartest search engine can’t do it all for you, especially when you’re relying on content written by a multitude of non-specialists as most universities are.

    I blogged about a collection of podcasts and resources I’ve come across recently: I recommend the John Ferrera presentation in particular.

  7. Chris Limb said

    Is there an independently produced breakdown of the cost to an institution of each of the solutions currently in use?

  8. […] carried out for the twenty Russell Group Universities in December 2010. As described in the post on Trends For University Web Site Search Engines we found that “15 Russell Groups institutions (75%) use Google to provide their main […]

  9. […] Trends For University Web Site Search Engines – Brian Kelly […]

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: