Trends For University Web Site Search Engines
Posted by Brian Kelly on 15 Dec 2010
Surveys of Search Engines Used on UK University Web Sites
What search engines are Universities using on their Web sites? This was a question we sought to answer about ten years ago,with the intention of identifying trends and providing evidence which could be used to inform the development of best practices.
An analysis of the first survey findings was published in Ariadne in September 1999. As can be seen from the accompanying pie chart a significant number (59 of 160 institutions, or 37%) of University Web sites did not provide a search function. Of those that did the three most widely used search engines were ht://Dig (25 sites, 15.6%), Excite (19 sites, 11.9%) and a Microsoft indexing tool (12 sites, 7.7%).
Perhaps the most interesting observation to be made is the diversity of tool which were being used back then. On addition to the tools I’ve mentioned universities were also using Harvest, Ultraseek, SWISH, Webinator, Netscape, WWWWais and Freefind together with an ever larger number of tools which were in use at a single institution.
The survey was repeated every six months for a number of years. A year after the initial finding had been published there had been a growth in use of the open source ht://Dig application (from 25 to 44 institutions) and a decrease in the number of institutions which did not provide a search function (down from 59 to 37).
This survey, published in July 2001 was also interesting as it provided evidence of a new search engine tool which was starting to be used: Google, which was being used at the following six institutions: Glasgow School of Arts – Lampeter – Leeds – Manchester Business School – Nottingham – St Mark and St John.
Two years later the survey showed that ht://Dig was still popular, showing a slight increase to use across 54 institutions. However this time the second most popular tool was Google, which was being used in 21 institutions. Interestingly it was note that a small number of institutions were providing access to multiple search engines such as ht://Dig and Google. It was probably around this time that the discussion began as to whether one should use an externally-hosted solution (due to concerns regarding the sustainability of the provider, the loss of administrative control and use of a proprietary solution when open source solutions – particularly ht://Dig – were being widely used across the sector).
These surveys stopped in 2003. However two years later Lucy Anscombe of Thames Valley University carried out a similar survey in order to inform decision-making at her host institution. Lucy was willing to share this information to others in the sector, and so the data has been hosted on the UKOLN Web site, thus providing our most recently survey of search engine usage across UK Universities.
This time we find that Google is now the leading provider across the sector, being used in 44 of the 109 institutions which were surveyed. That figure can be increased of the five institutions which were using the Google Search Appliance are included in the total.
What’s Being Used Today?
A survey of Web site search engines used on Russell Group University Web sites was carried out recently. The results are given below.
In brief 15 Russell Groups institutions (75%) use Google to provide their main institutional Web site search facility, with no other search engine being used more than once.
Note that Google provide a number of solutions including the Google Search Appliance, the Google Mini and the public Google search. Mike Nolan pointed out to me that “you can customise with API or XSLT to make [Google search results] look different” so I have only named a specific solution if this has been given on the Web site or I have been provided with additional information (note that I can update the table if I receive additional information).
Discussion
Over ten years ago there was a large diversity of search engine solutions being used across the sector. The discussions at the time tended to focus on use of open source solutions, with the argument occasionally being made that since ht://Dig was open source there was no need to look any further. There was also a suggestion that the open source Search Maestro solution, developed at Charles University and deployed at the University of Dundee could have an important role to play in the sector.
However in today’s environment it seems that a Google Search solution is now regarded as the safe option and this seems to have been corroborated with a survey carried out by Mike Nolan in December 2008. The potential of Google Custom Search will have been enhanced by the announcement, two days ago, of developments to metadata search capabilities.
There has, however, been some discussion recently on the web-support JISCMail list on software alternatives to the Google Search Appliance.Another discussion on the website-info-mgt JISCMail list has shown some interest in the Funnelback software. But, interestingly, open source solutions has not been mentioned in the discussions.
We might conclude that, in the case of Web site search engines, after ten years of the ‘bazaar’ the sector has moved to Google’s cathedral. What, I wonder, might be the lessons to be learnt from the evidence of the solutions which are used across the sector? Might it be that the HE sector has moved towards cost-effective solutions provided by Google’s free solutions or the richness of the licenced Google Search Appliance or Google Mini? And might this be used to demonstrate that the HE sector has been successful in identifying and deploying cost-effective search solutions?
Tony Hirst said
Interesting… because the feeling I get from…. dunno… recent confs, feeds, discussion etc is that APache Solr is something to watch… http://lucene.apache.org/solr/
Brian Kelly (UK Web Focus) said
Hmm, yes, I’ve heard people talking about Solr. But this doesn’t seem to have had any significant impact on those responsible for policies on search engine software deployed across institutional Web sit. Anybody know of examples where it is being used across the enterprise?
Timo Selvaraj said
SearchBlox is another open source based provider and it is free. SearchBlox is based on the open source Apache Lucene api. Here is a comparison between SearchBlox and Google Mini. – http://www.searchblox.com/comparison-of-searchblox-vs-google-mini
Tweets that mention Trends For University Web Site Search Engines « UK Web Focus -- Topsy.com said
[…] This post was mentioned on Twitter by Brian Kelly and marquibeck, marquibeck. marquibeck said: via @briankelly good data to compare trends in search engine use in UK Universities 1999 vs today http://bit.ly/gOn8hs […]
Simon Bains said
What happened to the University of Edinburgh link in the table?
Brian Kelly (UK Web Focus) said
Oops. sorry. Now fixed.
Matt Taylor said
Thanks for the post Brian, this is very interesting.
For more information on LSE can I point your readers to the following Case Study on the Funnelback site:
http://www.funnelback.com/news-and-events/news/posts/funnelback-seminar-london-school-of-economics-case-study
And please watch this space for news on other UK Universities that are currently in-build on Funnelback!
Jon Warbrick said
I’d make two points:
1) I think that Google Search Appliance, Google Custom Search and the public Google search actually represent significantly different solutions (not least in terms of up front cost) and so the 75% ‘Google’ slice is somewhat misleading. The results of a better breakdown would be interesting.
2) I’ve always considered that Open Source has failed to deliver good web search engines because a) they are hard to write (especially if they are to work on an enterprise scale), and b) there’s a sufficiently small market for them that there isn’t a critical mass of developers.
Neil Allison said
I think Jon make a good point about the nature of Google search tools. You get what you pay for.
For me, more interesting than the technology is how it’s being used. Do you have any data on how university search engines are being managed? From a user experience point of view, that is.
Even the smartest search engine can’t do it all for you, especially when you’re relying on content written by a multitude of non-specialists as most universities are.
I blogged about a collection of podcasts and resources I’ve come across recently: http://bit.ly/cIueJK. I recommend the John Ferrera presentation in particular.
Chris Limb said
Is there an independently produced breakdown of the cost to an institution of each of the solutions currently in use?
Brian Kelly (UK Web Focus) said
Not that I’m aware of, I’m afraid.
Do We Want Technical Diversity or Harmonisation? « UK Web Focus said
[…] carried out for the twenty Russell Group Universities in December 2010. As described in the post on Trends For University Web Site Search Engines we found that “15 Russell Groups institutions (75%) use Google to provide their main […]
Navigation & Labeling Reference Links | kabayview.com said
[…] Trends For University Web Site Search Engines – Brian Kelly […]