UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

Monitoring Web Server Usage Across A Community

Posted by Brian Kelly on 19 Jun 2007

How many public Web servers are there at your University? And how have the numbers changed over the past 5 years? Are you running more servers, as the range of services you provide grows, or have the numbers of servers decreased due to rationalisation in order to avoid duplication of effort across the institution?

Server numbers in March 2002I published an article on A Survey Of Numbers of UK University Web Servers in June 2000, with a follow-up article on An Update Of A Survey Of The Numbers of UK University Web Servers which was published in March 2002.  

The survey was carried out using the online Netcraft service, by using a wildcard (* to obtain details of the numbers of Web servers in, in this case, the Oxford university domain.  This process was repeated manually for all (~160) UK HEIs. A histogram for the results of the 2002 survey is illustrated.

Netcraft listing of Oxford Web serversHow have things changed in the past 5 years? It would be possible to repeat the manual survey – as can be seen, the online Netcraft survey service is still available.

However in a Web 2.0 environment in which many lightweight Web-based tools are available it would not be sensible to repeat the methodology.  It strikes me that the Netcraft results page is well-suited for screen-scraping (immediately after the “Results for *” text is a line which says “Found 356 sites“. So while this interface remains, the data can be programatically extracted, stored and displayed, possibly in a graphical format). 

The Dapper application could, perhaps, could be used for this purpose. After all, as I’ve described previously, Dapper has been used to create Blotter, which scrapes Technorati ranking data on a daily basis, stores this data and display the trends graphically.

But rather than doing this myself, I’d like to suggest that this might be a suitable example for the IWMW 2007 Innovation Competition – this should be lightweight and user-focussed (providing data which can detect trends across the community). It could be possible to provide an interface for a user to supply their own domain name, although another approach might be to take the domain names for the community (or perhaps a regional subset of the community) and display variations across the community – that, I think would be cool (and ‘coolness’ is one of the criteria for the competition). 

4 Responses to “Monitoring Web Server Usage Across A Community”

  1. Mike Nolan said

    Is this a sensible metric for comparison? The number of domain names listed on Netcraft can be wildly inaccurate and even if it was, does it mean much? I could set up a blog system similar to which gave 10,000 students Alternatively an institution could use a proxying system to map content from dozens of servers onto a single hostname.

    I agree that it would be interesting to find out more about the number of physical servers[1] and the services institutions offer online but I’m not sure an automated survey can discover the answers.

    [1] Although this is also reducing in importance as more people adopt server virtualisation – I have half a dozen VMs running on a single server, and expect that to increase significantly this year.

  2. Hi Mike
    After I announced one of the server usage surveys a few years ago, someone suggested that real soon now every institution would have thousands of servers, as desktop PCs would act as servers. That hasn’t happened – but it might be interesting to observe this if it started to happen (or the approach started to take off.
    Your point about physical servers is very interesting. I hadn’t considered that institutions may wish to monitor growth of physical servers as this could potentially be in conflict with an institutions environmental policy.

  3. […] Monitoring Web Server Usage Across A Community « UK Web Focus UK Web Focus Reflections On The Web Home About Blog Experiments Blog Policies Brian Kelly, UK Web Focus Welcome to Brian Kelly’s UK Web Focus blog. This blog gives my thoughts on Web developments, with an emphasis on best practices and areas of innovation […]

  4. Any updates on the stats that you published? Would be interesting!

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: