UK Web Focus

Innovation and best practices for the Web

Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers

Posted by Brian Kelly on 9 June 2011

Numbers matter. Or, as the JISC-funded report on Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources put it in its list of recommendations: “The media and the public are influenced by numbers and metrics“.

An example of how the media makes use of numbers in its reports can be seen by the recent story on Apple’s series of announcements at the WWDC event. Despite there being no new iPhone or iPad announced the media picked up on the statistics which Steve Jobs presented to highlight his view of the importance of various Apple developments with the BBC effectively providing an advertisment for the company:

Sales of 25 million iPads in just 14 months. 200 million iOS devices in total. 15 billion songs downloaded since 2003. 130 million books. 14 billion apps downloaded from a store that now runs to 425,000 apps.

Note how the figures are nicely rounded: 25 million iPads and not 24,987,974; 130 million books and not 131,087,459.  Similarly there are no caveats about the difficulties in gathering accurate figures.   There is simply a clear understanding from the private sector that such such approaches can be valuable in marketing activities to help ensure the growth and sustainability of the company – and the JISC report recommends that development work in the higher education sector learns from these approaches.

This is a reason why I feel we should be more willing to gather and use statistics related to the services we develop within the sector. Unfortunately, as described in a recent post on A Pilot Survey of the Numbers of Full-Text Items in Institutional Repositories it seems that we can’t provide a summary of the form:

UK repositories grew by xxxx  in just 14 months. yyy items in total. zzz full text items downloaded since 2003.

because, although we have such data and it is under our control, we do not make it available. There will be reasons for this: the complexity of the data; the difficulties in interpretting the data and, perhaps most importantly, the fears that the data will be used against the host institution or those directly involved in providing the service.

Yet data about institutional use of Social Web services is, in many case, freely available, as has been shown in recent posts on
Institutional Use of Twitter by the 1994 Group of UK Universities, Use of Facebook by Russell Group Universities, Institutional Use of Twitter by Russell Group Universities and How is the UK HE Sector Using YouTube?.

UKOLN is hosting a one-day workshop on “Metrics and Social Web Services: Quantitative Evidence for their Use and Impact” which will be held at the Open University on 11 July (and a few spaces are still available) which will seek to explore ways in which metrics related to use of Social Web services can be related to value.  Since the event is being hosted at the Open University it would seem appropriate to provide the following summary of the institution’s usage of popular services:

Open University’s
Social Network

Presence
Metric Comments
Facebook Likes
46,192
Twitter Followers Tweets
13,274 3,058
YouTube Channel Views Total Upload Views Subscribers
390,313 515,581 4,004 3 other channels available
iTunes No statistics readily available

The statistics for Facebook, Twitter and YouTube are easily obtained – although I am not aware of a way of automating the gathering of such statistics across all UK University presences which would be helpful if we wished to provide a national picture of how UK Universities are using these services.  I do suspect, however, that institutions may well be employing Social Media consultants to provide advice on strategies for making use of such social media services, which will include an audit of findings for peer institutions.  It would therefore be cost-effective for the sector if such information were to be gathered and published centrally so that money is not being used to replicate such activities unnecessarily.

It should be noted that it does not appear possible to obtain statistics from the iTunes service so in this case we are reliant on the information published by the Open University, such as the press release which announced in November 2009 that “The 10 millionth Open University track on iTunes U, a dedicated area within the iTunes Store (www.itunes.com), was downloaded this week, making the OU a top provider of free university content on iTunes U. The Open University launched its first piece of educational content on iTunes U in June 2008 and now has an average of 375,000 downloads a week.” The press release went on to add that “Tracks from the OU’s 260 collections are consistently in the Top Twenty downloads and this week one in four of the top 100 downloads on iTunes U is from The Open University“.

But whilst the Open University clearly benefits from such marketing, the sector itself is failing to demonstrate how collectively we are making use of innovative IT developments – whether in the area of social media, institutional repositories or in other areas –  to support its teaching and learning and research activities.

The concerns mentioned previously (such as “the information could  be used against us”)  can lead to the sector facing a form of the prisoner’s dilemma, whereby people (or organisations) fail to collaborate even if it is their interest to do so.

A danger for the sector is that if we fail to provide (and exploit) evidence of our services, others may make use of Freedom of Information (FOI) requests in order to gather such information and use it against the sector, as the Daily Telegraph did in an article entitled “Universities spending millions on websites which students rate as inadequate” which I described in a blog post last year year.

In a plenary session at UKOLN’s IWMW 2011 event to be held at the University of Reading on 26-27 July Amber Thomas, a JISC Programme Manger will give a talk on “Marketing and Other Dirty Words“. The talk will seek answers to the questions “How [Web services] can have maximum impact? How can they be effectively presented to aid in marketing and recruitment, and to increase engagement with the world outside the university? This session will bring together key messages from marketing, social media around content, usage tracking and strategy, with ideas for how we can present our intellectual assets online to get maximum effect.

The JISC-funded report on Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources which I described at the beginning of this post described how:

Being able to demonstrate your impact numerically can be a means of convincing others to visit your resource, and thus increase the resource’s future impact. For instance, the amount of traffic and size of iTunesU featured prominently in early press reports.

I agree. But let’s not just do this on an institutional basis, let’s open up access to our usage data so that the value of use of IT across the higher education sector can be demonstrated.  And let’s ensure that usage data for in-house services is made available (ideally in an open and easily reusable format) so that we won’t be reliant solely on usage data which is provide by commercial services (with uncertainties as to whether such data will continue to be provided for free)  – so that there won’t be a need for FOI requests for such data.

About these ads

12 Responses to “Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers”

  1. Hi Brian – are these stats on iTunesU ‘impact’ from the OU KMI of interest? http://projects.kmi.open.ac.uk/itunesu/impact/

  2. lescarr said

    It’s interesting that our main concerns in providing statistics thus far have been “will the authors like it”. There has been a trong argument that “we shouldn’t provide statistics that show up some people as weak, or their work unpopular and unregarded”.

  3. dkernohan said

    I’m worried about this trend… numbers (especially in the technology world) are just noise. Why should I care how many applications I can download from the iOS store compared with Android Marketplace, or how many people own each brand of handset. What does that information tell me, compared to one decent conversation with someone who understands both what I need and what is available?

    Surely the same applies to education, with the caveat that we (and the people we are talking to) are supposed to be a bit smarter than fanboys waving their smartphones about?

  4. Chris Rusbridge said

    I have been trying to measure a few key statistics for a small group of repositories (across the 3 major platforms), and it’s not easy!

    – deposits. There’s what looks like really good data in ROAR for most repositories with an OAI-PMH interface. The data allows calculation of daily or monthly or annual uploads. There are a few glitches; sudden lurches downwards in the numbers etc that are hard to explain. The biggest difficulty is that in some cases the headline numbers don’t match what the repository seems to hold when you examine it directly (although not all platforms allow you to do that easily as an end user). Perhaps this is accounted for by dark items? I don’t know!

    – accesses, which I wanted split into views (of the metadata/splash pages) and downloads (of the items/bitstreams). No getting away from the difficulties here. First, despite projects like IRS (http://irs.eprints.org/about.html) each platform does this differently. Second, you need to get your data from the particular site, and each site does it differently (eg some with Google Analytics, others with repository internals, etc). Third, to make sense you need to exclude the effects of spiders and robots, which are legion. And you need to do this in a comparable way across all your comparators. I concluded that the access data wasn’t worth much.

    – inlinks, in place of citations (since where the item is a secondary version, citations would be to the primary version, eg journal). These are tricky to measure in a replicatable way. I found a method using Yahoo that seemed robust, but some people have suggested that it really counts linking sites rather than links, and hence greatly under-estimates.

    I think it would be a good idea to gather these data; @Dkernohan seems to me to be missing the point…

  5. […] up full of good intentions to do so “proper” work today, but a a post by Brian Kelly on Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers triggered the hacker ethic in me and so I went of down a rabbit hole… And here’s what I […]

  6. Pat said

    Has the rest of CETIS told you about paradata yet?

  7. […] Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers […]

  8. […] Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers […]

  9. […] Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers « UK Web Fo…. […]

  10. […] accesses to usage data associated with the practices. This was an idea I explored in a post on Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers in which I highlighted the comment published in JISC-funded report on Splashes and Ripples: […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: