UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

Archive for the ‘Evidence’ Category

Paper Accepted for OR12: Can LinkedIn and Academia.edu Enhance Access to Open Repositories?

Posted by Brian Kelly on 3 Jul 2012

I’m pleased to say that a paper by myself and Jenny Delasalle, Academic Services Manager (Research) at the University of Warwick, which asked “Can LinkedIn and Academia.edu Enhance Access to Open Repositories?” has been accepted for the Open Repositories conference, OR 2012.

This paper, which is available from the University of Bath institutional repository, is based on work initially published on this blog.

A blog post entitled “How Researchers Can Use Inbound Linking Strategies to Enhance Access to Their Papers” published on 2 March 2012 described an Inbound linking strategy to get to the top listing on google fast. It occurred to me that my willingness to make use of researcher profiling services such as Academia.edu, ResearcherID, Scopus, Researchergate, Mendeley, Microsoft Academic Search and Google Scholar Citations may have helped to enhance the visibility of my research papers which are hosted in the University of Bath repository. The blog post went on to describe how I found that I was author of 15 of the most downloaded papers in the repository from my department.

More recent investigations reveal that, as illustrated, I have the largest number of downloads of any author at the University of Bath! This was recently brought to the attention of the PVC for Research who, in a departmental meeting, informed me that a University of Bath Research Group had discussed these figures and asked me to share the approaches with other researchers at Bath. In response I mentioned that the approaches I’d taken, the evidence I’d gathered, the hypothesis I had proposed for explaining the evidence, possible alternative hypotheses, the limitations of the approaches, the implications of the findings and areas for further work had been submitted to the Open Repositories 2012 conference – and if the paper was accepted the findings would be available to all, and not just researchers at my host institution.

The paper explores other possible reasons for the high visibility of these papers – and one possibility worthy of further investigation is the provision of many papers in HTML formats and not just PDF and MS Word. However the use of popular researcher profiling services such as LinkedIn and Academia.edu are felt to be worth recommending to researchers in order (a) to ensure that their research papers can be more easily found by their peers on these services and (b) so that links to the paper on their institutional repository can enhance the visibility to Google of the papers as well as enhancing the Google ranking of the repository itself.

Of course it probably needs to be said that that the number of downloads is not necessarily an indicator of quality. However the converse is also true: just because a paper in a repository is seldom viewed does not indicate that it must be a great paper! I am quite happy to promote the use of such approaches since increased numbers of views, especially for the target communities, can help to both embed the ideas given in the papers by practitioners and increase the likelihood that the papers will be cited by other researchers. In my case I’m pleased that, according to Google Scholar Citations, my most cited papers have been cited 87, 67, 54 and 40 times.

My co-author Jenny Delasalle has been investigating use of researcher profiling service at the University of Warwick, her host institution. It was interesting that in Jenny’s research she found that a number of commercial publishers encourage their authors to use services such as LinkedIn and Academia.edu to link to their papers hosted behind the publishers paywalls – and yet we are not seeing institutional views of the benefits of coordinated use of such services by their researchers. Institutional repository managers, research support staff and librarians could be prompting their institutions to make the most of these externally provided services, to enhance the visibility of their researchers’ work in institutional repositories.

Surely it is time for the research community to develop inbound linking strategies to their research work, especially as this can be done so simply. Indeed the OR12 conference organisers have invited us to summarise the ideas described in a poster and a one-minute presentation. The ideas have been summarised using the Pixton cartoon generation tool in four strips.

[link to source]
[link to source]
[link to source]
[link to source]

I’m not sure if it will be possible to use PowerPoint during the one-minute madness but I have prepared some slides which are available on Slideshare and embedded below.

NOTE: A one minute summary of this paper was given on the opening day of the OR 12 conference. A video recording of the summary is available on Vimeo and embedded below.

Also note that a slightly modified version of this post was published on the LSE Impact of Social Sciences blog on Thursday 23 August 2012. You can also view the bit.ly statistics for access to the post via the bit.ly URL.


Twitter conversation from Topsy: [View]

Posted in Evidence, Repositories, Web2.0 | 7 Comments »

“Conferences don’t end at the end anymore”: What IWMW 2012 Still Offers

Posted by Brian Kelly on 25 Jun 2012

IWMW 2012 Is Over: Long Live IWMW 2012!

Conferences don’t end at the end anymoretweeted @markpower two days after IWMW 2012 delegates had left Edinburgh and returned home.  This has always been the case: conferences organisers will have evaluation forms to analyse and invoices to chase.  But the point Mark was making related to the continuing discussions about the ideas discussed at an event and the accompanying resources, resources which increasingly these days may have been created during the event and support for the participants, which can help to ensure that an event is not just an collection of individuals who are co-located for a few days but, as I described in a recent post, a sustainable and thriving community of practice.  A related point was made recently in a post on “#mLearnCon 2012 Backchannel – Curated Resources” in which David Kelly described how “The backchannel is an excellent resource for learning from a conference or event that you are unable to attend in-person” and went on to add that he finds “collecting and reviewing backchannel resources to be a valuable learning experience …, even when [he is] attending a conference in person. Sharing these collections on this blog has shown that others find value in the collections as well.” But what are the resources from the IWMW 2012 which may be of interest to others, where can they be found and what value may they provide?

Key Resources

Slideshare

The slides used by the plenary speakers were uploaded to Slideshare in advance of the talks in order to allow the slides to be embedded in relevant Web pages and enable a remote audience to view the slides.  It should also be added that this also allowed participants at the event to view the slides if they were not able to view the main display of the slides. The slides have been tagged with the “iwmw12” tag on Slideshare.  This enables the collection of slides to be accessed by a search for this string or by  browsing slideshows which use this tag.  Note that in previous years an event tag had been used, but this service was discontinued recently, after Slideshare had been bought by LinkedIn.

Creating a collection of slides used at the event enables a Slideshare presentation pack to be created, as illustrated, thus making it easy to access all slides used at the event which have been made available. As can be seen from the IWMW 2012 web site, the presentation pack can be embedded in Web pages. This service is being used since participants at IWMW have frequently asked to be able to access slides, including slides used in parallel sessions which they were not able to attend. Using Slideshare makes it easy to respond to this user need. In addition it helps to raise the profile  and visibility of speakers at the event.

Lanyrd

The IWMW 2012 Lanyrd page was set up in advance to provide a social directory for participants at the event so they could see who else was attending. The value of this grows as Lanyrd is used across a number of events: from my Lanyrd, profile, for example, I can see that I have appeared at events on 12 occasions with my colleagues Marieke Guy and on 5 occasions with Paul Boag, Tony Hirst, Andy Powell, Keith Doyle and  Mike Nolan. In addition to the social dimension. Lanyrd also provides calendar entries for sessions at events. The date and time of sessions at IWMW 2012 has been provided together with links to the main page on the IWMW 2012 web site have been added, together with slideshows and links to reports on the sessions which we are aware of. It should be noted that, as illustrated, a Lanyrd has a Wiki-style environment for uploading resources which avoids the single-curator bottleneck. As the person who set up the IWMW 2012 Laynrd entry, together with the IWMW guide for all IWMW events, it should be noted that I receive an email alert when new entries are added to the coverage, such as:

<http://lanyrd.com/2012/iwmw12/?t=c955d8172reV> (In guide IWMW) [22nd Jun 2012 07:52] *
@sheilmcn added coverage “Developing Digital Literacies and the role  of institutional support services” (http://www.slideshare.net/sheilamac/developing-digital-literacies-and-the-role-of-institutional-support-services  type:slides)
to session  “B2: Developing Digital Literacies and the Role of Institutional  Support Services” http://lanyrd.com/sqwtp

This can help to spot if inappropriate entries are being added.

Vimeo

As described in a post on Streaming of IWMW 2012 Plenary Talks – But Who Pays? we used the ustream.tv service for the live video stream. The videos are currently being processed and will be made available via UKOLN’s Vimeo account shortly. This service will be used to wider access to the plenary talks so that they are available for those who were not present at the event – although, of course, they can also be viewed by people who were at the event and wish to watch the talks again. In addition to the video recordings of the talks we have also taken a number of short interviews with participants at the event which will enable their thoughts on the event to be shared with a wider audience.

Flickr

With so many delegates now having digital cameras and smartphones there are a large number of photographs which have been uploaded to Flickr with the IWMW12 tag which can help to provide a collective memory of the event.

Having a large number of photographs, rather than a small set of selected ones taken  by an official photographer, provides a much broader perspective on the event. It also means that images browsing interface services, such as Tag Galaxy, are more useful by having a more diverse range of content.

The two images show a display of a Tag Galaxy search for photographs on Flickr with the “iwmw12” tag and one of the many photographs taken by Sharon Steeples of the final conclusions session during which I showed an image of the video stream, captured earlier that morning when Dawn Ellis gave a summary of Web developments at the University of Edinburgh, subverting normal conference-style approaches to case studies by telling this as a fairy tale. The video recording of this talk will be particularly worth watching.

Twitter

As can be seen from the image shown above, the lecture theatre also has a large blackboard.  The opportunity to use a blackboard during the final session provided too much temptation to ignore –  so in the summing up a tweet posted on the backboard was displayed, as a reminder that not everyone necessarily has a mobile device they could use for tweeting. However many people did use Twitter during the event. As is widely known, content posted on the Twitter stream becomes unavailable available a short period. There is therefore a need to analyse event tweets shortly after an event – or archive the tweets to allow them to be analysed subsequently.

Topsy

As can be seen from the image of the Topsy search for #IWMW12 tweets posted over a period of the past 7 days (click for a larger display) there were 666 mentions on 18 June and 574 on 19 June.  The most highly tweeted link was to the IWMW 2012 video page, which was mentioned in 43 tweetsduring the week on 17-24 June 2012. In total Topsy reported that there were 748 tweets during the week on 17-24 June 2012, 808 in the month from 24 May-24 June and an overall total of 846 tweets to date.

Other Commercial Twitter Analytics Tools

It should be noted that a large number of Twitter analytics tools are available which be used to analyse how Twitter has been used. The Tweetreach service, for example, reports that tweets containing the #iwmw12 hashtag have reached 7,553 Twitter accounts. However, as is often the case with usage statistics, such figures need to be treated with a pinch of salt.

Beyond Commercial Twitter Analysis Tools

Topsy, Tweetreach and other Twitter analytics tools can provide a useful summary of use of Twitter hashtags. However  in the UK higher education development community we are fortunate to have the expertise of developers such as Martin Hawksey and Tony Hirst who have a well-established track record in the development of value Twitter analysis tools and who can continually develop their tools based on particular needs and interests of the community.

As Martin described in a post entitled IWMW12 Data Hacks for the IWMW 2012 event he was  “collecting an archive of tweets which already gives you the TAGSExplorer view“.

Looking at Martin’s Twitter archive of #iwmw12 tweets, provided by the TAGS v.40 service, we can see that the top five Twitterers were @iwmwlive (281 tweets), @PlanetClaire (149 tweets), @sharonsteeples (103 tweets), @mariekeguy (100 tweets) and @jessica_hobbs (81 tweets). Since the @iwmwlive Twitter account was managed by Kirsty Pitkin it seems that the top twitters at the event were all female: this seems particularly interesting in light of the fact that only about a quarter of the participants were female.

It should also be noted that this tool also provides a display of the tweets over time.  It can also be seen (right) that tweeting peaked at 2pm on Tuesday, 19 June 2012 with 229 tweets.

Finally I should mention Martin’s most recent development:  a filterable/searchable archive of IWMW12 tweets. As illustrated below, this provides a clickable word cloud of the content of the tweets, together with a search box and browse interface for the tweets.  It was while browsing the tweets that I came across a comment from @JohnGreenway who, during the conclusions, tweeted:

As someone from a commercial background, #iwmw12 has been excellent – hope everyone in HE realises how rare this is in other industries!

Such live tweeting helped in providing useful real time feedback not only to the event organisers but also the plenary speakers.  Other comments received during the event included:

  •  Excellent talk by Stephen Emmott – always a reliable IWMW speaker! #iwmw12 from @adriant
  • First time at #iwmw12 and had a brilliant time. Great ideas, great people, great weather, who could ask for more. from @millaraj
  • First time at IWMW: great speakers, interesting topics, fantastic Ceilidh. Many thanks to organisers and presenters. #IWMW12 #new #social from@seajays
  • Great summary by @sloands on how to build accessibility into project management processes using BS8878 #iwmw12 from @chistabel6

Further examples of tools which Martin Hawksey developed at the IWMW 2012 event can be accessed from his Delicious IWMW12 Hacks set of bookmarks.

The paper.li Daily newspaper

Finally I should mentioned the IWMW12 paper.li daily newspaper, which had been set up in advance of the event. This automated newspaper consisted of articles based on links which had been tweeted  containing the event hashtag.

Reflections

Conferences have never ended immediately after the final talk has been given – this is always the paperwork to be processed, the evaluation forms to be analysed and feedback given to the speakers and local event organisers. What is different nowadays is that event resources and discussions are no longer ‘trapped in space and time’.  If an event has value, it should surely have value for those who may not have been able to attend.

It was therefore appropriate that during my opening talk I was able to announce the launch of the JISC-funded Greening Events II; Event Amplification report. We hope that the report will be useful for others who are planning amplified events.  As Mark Power put it: “Conferences don’t end at the end anymore” – you need to make plans for managing the resources after the conference is over. We hope the report will be useful for those planning amplified events.


NOTE: Shortly after this post was published a post entitled “But who is going to read 12,000 tweets?!” How researchers can collect and share relevant social media content at conferences was posted on the LSE Impact of Social Sciences blog which echoed the approaches described in this post.

Posted in Events, Evidence, preservation, Twitter, Web2.0 | 3 Comments »

Twitter Analysis: Can #bathopenday Learn from #IWMW12?

Posted by Brian Kelly on 21 Jun 2012

In the final session at UKOLN’s IWMW 2012 event which finished yesterday I suggested that the community analysis techniques which Tony Hirst and Martin Hawksey were applying to the #IWMW12 tweets might be useful in institutional contexts. “Suppose your University is having an Open Day” I suggested “and you promoted a Twitter hashtag which could be used by visitors to your institution who, it seems, are now making greater use of Twitter. You might be able to apply the tools developed by Tony and Martin to help develop a better understanding of that important community – 17 year old students who may choose your University next year“.

After the IWMW 2012 had finished, whilst unwinding in a pub opposite the Appleton Tower in Edinburgh I checked my email and spotted an email which announced “Over 5,000 visitors expected on campus tomorrow!” and went on to add:

As with any Open Day the campus will be busy, especially the car parks. As usual we have plans in place for overflow parking but if you can car share to help ease the pressure then please do so. Buses are also likely to be very busy, so please take this into account when making your travel arrangements.

On arrival at the University I spotted posters around the campus signposting the various departments – all of which contained the Twitter hashtag for today’s Open Day: #bathopenday So whilst tweets from staff at the University could well be full of complaints about travelling up the hill to the University, it does seem that there may be an opportunity to analyse the #bathopenday tweets.

Yesterday Tony Hirst (@psychemedia) tweeted “Visualising folk commonly followed by recent users of the #iwmw12 hashtag flic.kr/p/chxGAu” which is illustrated.

In addition Martin Hawksey (@mkawskey) has provided a timeline view of #IWMW12 tweets.

Might it be possible to apply these approaches to Bath’s #bathopenday tweets, I wonder? And is anybody else taking similar approach to their Open Days?

Posted in Events, Evidence, Twitter | Tagged: | 1 Comment »

The Blog Post as a Magnetic For Impact Findings

Posted by Brian Kelly on 11 Jun 2012

We have recently been asked to provide evidence of the usage and impact of the diverse services we provide. Such a request is perfectly understandable – commercial companies with be able to point to their profit margins as evidence of the effectiveness of their activities and whilst ways of doing this for those working in higher education will be more complex, I appreciate the need to do this.

Usage statistics can be easy to gather, especially for use of social media service. As an example on Saturday @dajbelshaw tweeted:

Whoah. Just noticed my ‘TELIC: The Essential Elements of Digital Literacies’ presentation has had 9,539 views since uploading *yesterday!*

and the following day informed us that:

20,000 views now. Uploaded Friday. Insane. slidesha.re/KobQZV

This example made me realise that the velocity as well as the overall usage statistics – coincidentally my most viewed slides on Slideshare, Introduction To Facebook, have also been viewed over 20,000 times – but this has been over a period of four years.

Whilst such usage statistics can be relatively easy to gather (and I will leave it to others to interpret the metrics), it can be more time-consuming to gather qualitative evidence of the take-up of services.

On Friday, however, I noticed an incoming link which was sending traffic to this blog. The link was from the eGovernment Resource Service for the Victoria Government, Australia and related to a post on Aversive Disablism, Web Accessibility and the Web Developer which I posted on 1 May 2012, the Global Accessibility Awareness Day. It then occurred to me that having a blog post embedded in a government’s web site might be a useful indicator of the value of my work in the area of web accessibility. Further investigation I found addition pages on the web sites about an article I had written on Web Accessibility: Putting People and Processes First and a paper on Forcing Standardization or Accommodating Diversity? A Framework for Applying the WCAG in the Real World.

As described on the Web sitethe eGovernment Resource Centre provides access to the Victorian Government body of knowledge on eGovernment, government 2.0, government use of social media and information and communications technology (ICT) and government website best practices, with Australian and international examples“. It does seem to me that I will be able to use this as an example of the impact at an international level of my work.  I also realised that I would not have been aware of this if I had not seen the incoming link to the blog post.  Blog posts, it would seem, can act as a magnet for attracting evidence of impact which would be difficult to detect otherwise.

I then went on to wonder why the Victorian Government in Australia was aware of my work.  I then remembered that in January 2009 I gave the opening plenary talk on “From Web Accessibility 2.0 to Web Adaptability (1.0)” at the OzeWAI conference in Melbourne and in November 2009 gave a plenary keynote talk, provided as a pre-recorded slidecast on “From Web Accessibility To Web Adaptability” at the OzeWAI at OZCHI 2009 conference”. Perhaps the connections I made in the first trip and the followup talk I gave ten months later made an impact which I have only become aware of recently?

Meanwhile, back to the gathering of further evidence ….

Posted in Accessibility, Evidence | Leave a Comment »

Trends in Slideshare Views for IWMW Events

Posted by Brian Kelly on 31 May 2012

“Why does everybody ask for slides during/after a presentation?”

Why does everybody ask for slides during/after a presentation? What do you do with them? I’m genuinely curious.asked @MattMay last night. I use Slideshare for a number of reasons:

  • To enable a remote audience to view slides for a presentation they may be watching on a live video stream, on an audio stream or even simply listening to the tweets (and a provide a slide number on the slides to make it easier for people tweeting to identify the slide being used.
  • To enable the slides to be viewed in conjunction with a video recording of the presentation.
  • To enable my slides to be embedded elsewhere, so that the content can be reused in a blog post or on a web page.
  • To enable the content of the slides to be reused, if it is felt to be useful to others. Note that I provide a Creative Commons licence for the text of my slide, try to provide links to screenshots and give the origin of images which I may have obtained from others.
  • To enable my slides to be viewed easily on a mobile device.
  • To provide a commentable facility for the slides.
  • To enable my slides to be related, via tags, to related slideshows.

It seems that I am not alone in wishing to share my slides in this way. Slideshare, the market leader in this area, was recently acquired by LinkedIn. As described in a TechCrunch article published on 3 May 2012: “LinkedIn has just acquired professional content sharing platform SlideShare for $119 million in cash and stock“.  The article went on to state that: “SlideShare users have uploaded more than nine million presentations, and according to comScore, in March SlideShare had nearly 29 million unique visitors”.

Slideshare is also widely used in higher education. But how is it being used, especially in the context of annual events for those involved in web management and web development activities?

Use of Slideshare at IWMW Events

A year ago today, on 31 May 2011, in a post entitled Evidence of Slideshare’s Impact I reported on the number of views on slides of talks which had been given at UKOLN’s IWMW event since 2006.  hosted on Slideshare. It is timely to update that survey.

The slideshows for each year are available in the following Slideshow event groups: IWMW-2006IWMW-2007IWMW2008IWMW2009 and IWMW2010 (note we changed the naming convention in 2008 once Twitter started to gain in popularity).  Note that since not all of the slideshows have been added to the event groups the analysis also made use of the Slideshare tags: IWMW2006,IWMW2007IWMW2008IWMW2009, IWMW10 and IWMW11. It should also be noted that on 20 May Slideshare discontinued event groups so we will not be able to use this approach for grouping slides used at IWMW 2012.

The numbers of views for each slide are available on Slideshare.  A Google Spreadsheet has been created which summarises the figures. The overall totals are given below.

Year Nos. of views
(May 2011)
Nos. of views
(May 2012)
Total nos.
of slides
Nos. of
plenary slides
Nos. of slides from
parallel sessions
Comments
2006 48,360  51,535 11 11  0 Slides added retrospectively.
In May 2012 most popular plenary: 12,216 views.
In May 2011 most popular plenary: 10,190 views.
2007 44,495  61,739 7 5  2 Slides from 2 w/shop sessions included.
In May 2012 most popular plenary: 27,814 views; w/shop: 12,267 views.
In May 2011 most popular plenary: 21,679 views; w/shop: 9,838 views
2008 94,629 109,055 17 8  9 W/shop facilitators encouraged to use Slideshare.
In May 2012 most popular plenary: 33,656 views; w/shop: 18,369 views.
In May 2011 most popular plenary: 26,005 views; w/shop: 22,525 views.
2009 38,877  46,238 29 10 19 In May 2012 most popular plenary: 2,489 views; barcamp: 2,839 views.
In May 2011 most popular plenary: 3,313 views; barcamp: 4,023 views.
2010 11,833 18,758 18 10  8 In May 2012 most popular plenary: 1,896 views; w/shop: 1,601 views.
In May 2011 most popular plenary: 2,816 views; w/shop: 2,599 views.
2011   6,393  11  5  6 In May 2012 most popular plenary: 1,119 views; w/shop: 944 views.
TOTAL 238,259 297,741  88  44  44 Growth: 2011 to 2012 = 25%

Note that these figures were mostly collected on 25 May 2012, but a small number of changes were made on 30 May. Also note that two different slideshows used in workshop session at IWMW 2012 had the largest numbers of views in May 21011 and 2012.

Discussion

A paper on “Who are we talking about?: the validity of online metrics for commenting on science [v0]” presented at the Altmetrics11 Tracking scholarly impact on the social Web workshop described how:

… we are not searching in online bibliographic databases for evidence of publications but that we are isolating the existence of online activity on the social web including: blogs; micro-blogging (Twitter); activity on social platforms – LinkedIn, and Mendeley; and sharing of presentations through Slideshare. 

The potential importance of Slideshare metrics was also highlighted yesterday in an article entitled Scientists: your number is up published in  Nature:

Herbert Van de Sompel at the Research Library of the Los Alamos National Laboratory in New Mexico, who is a long-standing proponent of author identifiers, hopes that the [ORCID] system might be used to generate alternative metrics by linking authors to their outputs in “less traditional venues of scholarly communication, such as tweets, blog posts, presentations on Slideshare and videos on SciTV”.

To illustrate the possible benefits of using Slideshare to host a slideshow consider Kristen Fisher Ratan’s slides on “Metrics: The New Black?“. From this I can view Kristen’s other slideshows and discover that she is the Product Director at PloS (Public Library of Science) and that her Twitter ID is @kristenratan. I can also find related slides hosted on Slideshare with the tags almsmetricspublishing and altmetrics.  This can be useful and I haven’t even looked at the slides yet! Slide 18 (illustrated) states that “Powerpoint download feature inadvertently tracked sub-article usage” which suggests that links to a PowerPoint presentation from a paper might provide usage information about the paper which might be difficult to find in other ways. I’m please that this slideshow has been uploaded to Slideshare!

But if Slideshare have a role to play in a portfolio of online metrics which may help to provide a better understanding of the impact of scientific research, what can be learnt from these metrics taken over a period of six years? Although the IWMW event is aimed at practitioners rather than researchers, it did occur to me that the experiences gained in collating these statistics might be of interest to those who are considering use of Slideshare statistics in an alt.metrics context.  Some thoughts that occurred to me:

  • Fragmented statistics: A number of speakers uploaded slides to their own Slideshare account. In cases where this was done after the slides had been uploaded to our main IWMW Slideshare account, we did not always know about the alternative location, which could result in difficulties in aggregating the usage statistics.
  • Reuse of slides at other events: On a couple of occasions, slides used for presentations at IWMW event were also subsequently used at another event.

However there are clearly more significant things to consider when looking at Slideshare metrics: namely, what is it that is being measured?  In this post I will not attempt to answer that question.  Instead I will simply conclude by providing a simple answer to Matt May’s question: “Why does everybody ask for slides during/after a presentation? What do you do with them? I’m genuinely curious.” by pointing out what the evidence tells us “They ask for them because they wish to view them. Why, therefore, would you not provide access to the slides?“. Even if the slides don’t provide significant textual content, they may be useful by letting others see how you have designed your slides and structured your ideas.

As I concluded in last year’s post:

Martin Weller made [the] point in his post on The Slideshare Lessons when he said: “by sharing good Slideshare presentations you are sharing ideas, and people will react to these. It can be in the form of comments on your blog post which features the presentation, on the Slideshare site itself, or through other social media such as twitter“.  Why, I wonder, are people still hosting their slides in the silo of an institutional Web site when the slides can easily be made available as a social object?

Or to put it another way, why would you not publish your slides on Slideshare?

Posted in Events, Evidence, Web2.0 | Tagged: | 2 Comments »

Survey of Institutional Use of Facebook

Posted by Brian Kelly on 22 May 2012

Total nos. of Facebook Likes for Russell Group universities.

A recent post entitled  What Next, As Facebook Use in UK Universities Continues to Grow? summarised growth in institutional use of Facebook in the 20 Russell Group universities in the UK, based on the number of ‘likes’ for the official institutional Facebook page. As can be seen in the accompanying histogram, there has been significant growth since the surveys in January and September 2011.  However as Tom Wright, the Digital Engagement Manager at the University of Nottingham commentedto gauge how successful universities are with Facebook you really need to look at other metrics around engagement, reach, influence, etc.

This is certainly true, but such metrics are not always publicly available and so in order to be able to answer the question “Are universities successful in their use of Facebook?” it will clearly be advantageous to be able to see a greater range of metrics. But in addition, the metrics themselves need to relate to the intended purpose(s) of the services and institutions may be using Facebook for a range of different purposes.

In order to help gain a better understand of how Facebook is being used across the sector, Tom and I have set up a SurveyMonkey form on institutional use of Facebook which invites respondents to summarise the purposes of institutional Facebook pages and the metrics they use to monitor the effectiveness of Facebook to achieve these purposes.  As Tom describes:

Understanding the roles which social networks such as Facebook can have in supporting business requirements is important for universities such as Nottingham with campuses in China and Malaysia and students from around the world. Facebook, with its international audience, has huge potential for today’s higher education institutions with their increasingly global reach, in the areas of student recruitment, marketing, internal communications and alumni support.

The survey is intended primarily for those working in institutional Web management or marketing teams in UK universities or FE colleges.  However we appreciate that universities around the world will have similar interests in the role of Facebook, together with concerns regarding the sustainability of the service, privacy issues and its relevance in supporting educational needs.

Such issues have been described in a paper on “Social Networking and Education: Using Facebook As An Edusocial Space” published in the Proceedings of Society for Information Technology & Teacher Education International Conference 2011 (pp. 3330-3338). This paper is also available on Scribd. The abstract for the paper states that:

The acceptance of Facebook by school-aged users is evident, but the potential of using social networking sites for educational purposes is still being debated. This paper explores the use of Facebook within a high school science-mentoring program. Results indicate that the use of Facebook positively affected the relationships between mentors and mentees. In addition, students believed that they learned more by using Facebook and would like to use Facebook for other educational purposes.

and concludes:

Social networking is already one of the most common ways that communication occurs virtually. While the majority of users spend time communicating with those who they have already built relationships with in reality, it may also have the potential to build relationships virtually.

Participation of a mentor and mentee on the Facebook group page was seen to positively affect their relationship both online and offline. Students and mentors that interacted regularly, posting questions and receiving feedback through the page, were observed as having a stronger relationship than other mentor-mentee pairs.

Might this suggest that there is a role to play in the development of Facebook apps which can support such collaborative activities? Back in March 2010 in a post entitled OU Facebook Apps, Reprise Tony Hirst mentioned work at the Open University which was “looking at rebooting the OU’s Facebook strategy. With a bit of luck, this means that we’ll be doing another push on the OU Facebook apps that were developed several years ago now and which I still believe provide a sound basis for a range of community building and social learning support services“.

But although the Open University might be working in this area, what is happening in the wider sector?  The concluding section on “Recommendations for future research” in the paper mentioned above described how:

Additional research is needed to explore the most beneficial design for an edusocial space. Though Facebook has been used for some educational purposes, research could explore the specific kinds of activities that are most beneficial to learners. Using social networking sites, however, is still a controversial issue with most schools blocking the site from students and faculty. Thus, it must also be understood if students can view sites like Facebook as educational spaces and be able to engage in learning activities at appropriate times.

The survey on institutional use of Facebook aims to gather information on such development activities.  We intend to present the findings at UKOLN’s Institutional Web Management Workshop, IWMW 2012, in Edinburgh on 18-20 June.  We hope that people within the sector will respond to this survey in order that we can gain a comprehensive picture of use of Facebook across the higher and further educational sectors.

Posted in Evidence, Facebook | 4 Comments »

What Next, As Facebook Use in UK Universities Continues to Grow?

Posted by Brian Kelly on 18 May 2012

Facebook IPO

On Tuesday a headline in the Guardian announced Facebook raises price range ahead of IPO with the article describing how “Facebook has increased the price range of its stock in what will be Silicon Valley’s biggest-ever initial public offering to raise more than $12bn (£7.4bn), giving the social network a valuation that could exceed $100bn“.

What will the reaction be after today’s IPO launch? I suspect that my Twitter network will be talking about a bubble which is about to burst (if the shares go up in price) or will gloat if the price goes down. I don’t expect people to say “the financial injection can support developments which will be beneficial to use of Facebook within higher education“!

But how widely used is Facebook within higher education? And are the trends suggesting that usage has peaked, with users becoming disillusioned with social networks such as Facebook or, perhaps, moving to other services, such as Twitter – as the recent announcement in the Guardian that “Twitter now has 10m users in UK” with the “UK [being] the fourth-largest country for Twitter users in the world, with 80% accessing it with mobile phones” may suggest?

Facebook Usage for Russell Group Universities

In order to gather evidence to support discussions on the relevance of use of Facebook in the higher education sector a survey of Facebook usage, determined by links for institutional pages, has been carried out for the 20 Russell Group universities. This survey follows on from previous surveys carried in in January and September 2011 which will enable trends to be detected. Note that the data provided in the following table is also available as a Google Spreadsheet.

Institution and Web site link
Facebook name and link
Nos. of Likes
(Jan 2011)
Nos. of Likes
(Sep 2011)
Nos. of Likes
(May 2012)
% increase
since Jan 2011
% increase
since Sep 2011
 1 InstitutionUniversity of Birmingham
Fb nameunibirmingham
8,558  14,182  18,611 117%    31%
 2 InstitutionUniversity of Bristol
Fb nameUniversity-of-Bristol/108242009204639
2,186   7,913  11,480  425%    45%
 3 InstitutionUniversity of Cambridge
Fb namecambridge.university
58,392 105,645 153,000 162%    45%
 4 InstitutionCardiff University
Fb namecardiffuni
20,035  25,945   30,648  53%    18%
 5 InstitutionUniversity of Edinburgh
Fb nameUniversityOfEdinburgh
(Page URL changed since previous survey)
 12,053   24,507   103%
 6 InstitutionUniversity of Glasgow
Fb Name: glasgowuniversity
  1,860   27,149  1,346%
 7 InstitutionImperial College
Fb nameimperialcollegelondon
5,490  10,257  16,444 200%    60%
 8 InstitutionKing’s College London
Fb nameKings-College-London/54237866946
2,047   3,587   5,384 163%    50%
 9 InstitutionUniversity of Leeds
Fb nameuniversityofleeds
   899   2,143    138%
10 InstitutionUniversity of Liverpool
Fb nameUniversity-of-Liverpool/293602011521
2,811  3,742   4,410  57%    18%
11 InstitutionLSE
Fb nameLSE/6127898346
22,798  32,290 43,716  92%    35%
12 InstitutionUniversity of Manchester
Fb nameUniversity-Of-Manchester/365078871967
1,978   4,734   9,356  373%    98%
13 InstitutionNewcastle University
Fb namenewcastleuniversity
    115      693  503%
14 InstitutionUniversity of Nottingham
Fb nameTheUniofNottingham
3,588    9,991  14,692  309%   47%
15 InstitutionUniversity of Oxford
Fb namethe.university.of.oxford
137,395 293,010 541,000   294%   85%
16 InstitutionQueen’s University Belfast
Fb nameQueens-University-Belfast/108518389172588
5,211   10,063   93%
17 InstitutionUniversity of Sheffield
Fb nametheuniversityofsheffield
6,646 12,412  19,308  199%   56%
18 InstitutionUniversity of Southampton
Fb nameunisouthampton
3,328 6,387  18,062  443%  183%
19 InstitutionUniversity College London
Fb nameUCLOfficial
977 4,346  33,853 3,365%  679%
20 InstitutionUniversity of Warwick
Fb namewarwickuniversity
8,535 12,112 14,472    70%   19%
TOTAL 287,767 566,691 998,991  241%    76%

Note

  • The data for the surveys was collected on 11 January 201125 September  2011 (estimate) and 16 May 2012.
  • The Facebook page for the University of Edinburgh has changed since the last survey.

Summary

Figure 1: Growth in total nos. of Facebook ‘Likes’ for Russell Group universities.

In brief in a period of eight months we have seen an increase in the number of ‘likes’ for the twenty UK Russell Group Universities of over 432,300 users with the largest increase, of almost 248,000 occurring at the University of Oxford. The largest percentage increase in that time has taken place at University of Glasgow, which has seen a growth of 1,346% from 1,860 to 27,149 and UCL which has seen a growth of 679% from 4,346 to 33,493.

The overall trends are illustrated in the accompanying histogram. As can be seen this shows a significant growth in the overall number of Facebook likes across the Russell Group universities.

It should also be noted that according to Russell Group University Web sitehalf a million students are enrolled at Russell Group universities – one in five of all higher education students in the UK“. Although the numbers of Facebook likes will include members of staff and other interested parties, the data does seem to suggest that a significant proportion of students are using Facebook.

Discussion

I suspect that social media consultants who advise the higher education sector will find the evidence presented in this post useful in demonstrating the importance of Facebook. However some caveats need to be pointed out:

  • There may be significant growth when six formers are deciding which universities to apply to. The ‘liking’ of a university may provide a bookmark which is not an indication of engagement with the institution.
  • New students may like their new institution’s Facebook page when they arrive, but may not use the service during their time at the institution.
  • Students may not unlike their institution’s Facebook page when they graduate, meaning that the number of Facebook likes will include people who have left the institution and may no longer use the service or have an interest in the information provided.

In addition to the need to the interpretation of the data there will also be a need to make policy decisions which should be informed by such evidence, but may not need to be determined by the evidence. It may be that Facebook can be regarded in a similar way to mailing lists: people use them and gain some value from them but development work is likely to take place using other technologies. Alternatively the popularity of Facebook may mean that that it has a role to play as a platform for development of new services. As described in a post on Facebook and Twitter as Infrastructure for Dissemination of Research Papers (and More) publishers such as Spring are providing mechanisms for researchers to share peer-reviewed papers using Facebook and Twitter, so perhaps Facebook could have a role to play as a sharing tool which is embedded within institutional tools.

Alternatively might Facebook have a role to play in more significant development work. The initial popularity of the Guardian’s Facebook app suggested that Facebook could have a role to play in sharing one’s reading activities across one’s networks, although more recent evidence, as described in a post on “Facebook Social Readers Are All Collapsing” suggests that Facebook apps which provide ‘frictionless sharing’ are declining in popularity. A more recent post TechCrunch post which described how Decline Of Reader Apps Likely Due To News Feed Changes, Shows Facebook Controls The Traffic Faucet provided a more thoughtful analysis of the reasons for the decline in usage, but also highlighted the dependencies which organisations will have in reliance on commercial companies whose business decisions may adversely effect organisations which rely on their services.

Figure 2: Facebook ‘Likes’ for Russell Group universities
(see Table for institution names)

The question “What next for Facebook use in UK Universities?” will be an interesting one. And with over half a million ‘likes’ will Oxford University be thinking about benefits which can be gained from such a large network? Alternatively will institutions such as Newcastle University with small Facebook networks shrug their metaphorical shoulders at such suggestions and argue that Facebook has no value to their teaching and learning and research activities? Or might the popularity of Facebook at Oxford and Cambridge Universities, which, as can be seen from the histogram, has a significant effect on the overall totals for Russell Group universities, simply reflect the brand awareness for these two institutions?

What are your thoughts? And what evidence will you need to gather if you feel that alternatives to Facebook will have a significant role to play?

Footnote: A follow-up post about a Survey of Institutional Use of Facebook has been published. This contains information about a survey in which we invite those involved in using Facebook to support institutional activities to provide details of their work. We invite people to complete this survey in order to provide a better understanding of Facebook use within the sector.

Posted in Evidence, Facebook | 6 Comments »

Is Blekko’s Traffic Really Going Through The Roof? Will It Challenge Google?

Posted by Brian Kelly on 18 Apr 2012

A tweet from @philbradley alerted me to an article on Searchengineland.com which announced Blekko’s Traffic Is Up Almost 400 Percent; Here Are The CEO’s Five Reasons Why. Phil was enthusiastic in his tweet: #Blekko traffic goes through the roof – for good reason. Try it out! 

The reason for the Searchengineland.com’s headline seems self-evidence from an image showing the growth in traffic for Blekko.com since January which, to provide a comparison, is contrasted with traffic for the duckduckgo.com search engine. As described in the article:

According to comScore’s numbers, Blekko is now getting about triple the traffic of fellow underdog search engine DuckDuckGo

Blekko’s CEO seems to have provided a significant contribution to the article, and is quoted as including the following reasons for Blekko’s popularity:

  1. Improved index quality.
  2. Dissatisfaction with Google.

Are we seeing an example of weak signals of a significant change in the search engine marketplace? And if this is the case, should institutions be making plans for changes in working practices?

Using Alexa to compare the daily traffic for Blekko, Duckduckgo and Google we see a different picture: or perhaps it is difficult to see the story, because the traffic for Blekko and Duckduckgo fails to move above the x-axis, with a percentage traffic close to zero. It order to see a comparison of the traffic rank, there is a need to display this information on a logarithmic scale, as shown below.

Although there is a need to monitor indications of new developments, there is also a need to avoid over-hyping something new. I think there was a similar over-reaction when Yahoo sold the Delicious.com social bookmarking service, with some of the teething problems encountered in the migration of the service to new ownership leading to people migrating, perhaps prematurely, to new services. Perhaps a more appropriate headline for the Searchengineland.com article (which appears to have been based on a press release) would be “One Little Used Search Engine Used More Than A Rival“.

However one interesting aspect of the story was the suggestion of user dissatisfaction with Google. Yesterday the BBC featured an article which described how Google tackles temporary Gmail access failure which began “Google says it is looking into why thousands of users have been unable to access their Gmail accounts“. The thousands of Gmail users were apparently less than 2% of Gmail’s user base. But closer to home, yesterday Tony Hirst tweeted about how his OUseful.info blog had seemingly disappeared from Google, and he was no longer receiving the large amount of traffic which Google sends to him blog. As a prolific blogger (who has an entry in Wikipedia) Tony described his experiences in a post entitled So Google is No Longer OUseful.info’s Friend…? Use Bing.com instead… But today a Google search for “Tony Hirst blog” now seems to be working. Another minor glitch, it seems, which is quickly fixed.

I can’t help but feel that the more significant issues surrounding Google aren’t to do with performance and reliability issues: after all we have no evidence that Bing, Blekko or Duckduckgo will provide a reliable service if they had the volume of traffic which Google has and, as described last year in a post which asked Time to Move to GMail? local email service can also be unreliable. For me the more significant stories which we have seen in the past few days which may have an impact on Google’ longer-term relevance are to do with legal disputes with the BBC News describing:

and Google’s battles with Facebook and Apple being highlighted in the Guardian:

Posted in Evidence, jiscobs | 4 Comments »

Personal Perspectives on How Metrics Can Influence Practice

Posted by Brian Kelly on 9 Apr 2012

A few days ago I favourited the following tweets from @lesleywprice:

RT @LnDDave: Too many people focus on the metric instead of the impact < and its impact that matters otherwise what is the point#trainchat [source]

RT @LnDDave: Too much of ROI is sterile data; use the data to help tell a better story < I like story telling makes it real #trainchat  [source]

Interestingly, Lesley noticed that I had favourited her tweets and provided some additional contextual information:

@briankelly…tks for faving tweets. I have taken part in a couple of really good tweetchats this week worth a look #trainchat swchat

This made me realise how the simple act of favouring a tweet is an action which can provide an identification of interest and lead to a subsequent dialogue – a good example, I feel, of frictionless sharing in action. Having been given the context to those two tweets I was able to find a post on “#TrainChat: Recap of our Twitter Chat with David Kelly“. This summary, incidentally described how “For the sake of clarity, we’ve condensed Kelly’s various tweets into a single response, and in places cleaned up a little bit of Twitter grammar” – an interesting example of an emerging practice for the curation of tweets.

Although this online Twitter chat covered “Three Essential Tips for New Online Trainers” the discussions about metrics for learning resonated with me in another context – our rapper sword dancing team’s performance in the annual DERT 2012 competition. We had a great time dancing in pubs in Soho and felt that our dances reflected the dedication we had shown in practices coming up to the competition, although we were conscious of the mistake we made in one of the competition spots. However it wasn’t until we saw how the other teams did that we realised how disappointed we felt – we were in bottom place :-( Even worse, not in the bottom place in the Premier Class, but the Olympic Class, the second division. I’ll not comment on the fact that in this traditional male dance we were beaten by three women’s team and two mixed teams, but being beaten by a morris team was embarrassing!

In the debriefing which took place on the first practice evening after the competition we agreed that our standard of dance had failed to keep up with our peers in recent years. We acknowledged that the judges’ comments were fair and that even though the marking system had its flaws, the scores were a valid reflection of the standard of our dance compared with the other teams (we should add that with scores for the comic characters from one judge of 10 and 12 out of a maximum of 10, the aberrations in the marking sometimes worked to our advantage!)

At the debriefing we agree that we should set a goal of being in the top three at next year’s event, for the dance team, musicians and characters. We also agreed that we needed to develop a new dance, in keeping with the current expectations and standards which have been raised over recent years. We also agreed that to achieve these new team goals we needed to have a more coherent approach to our weekly dance practices and regular dances in pubs around the area and at folk festivals.

How does this relate to the comments about ROI and learning analytics? For me it is clear that in this context:

  • It is easy to be self-deluded about the quality of one’s performance.
  • It is also easy to be self-deluded when reading scores and interpretting the feedback.
  • Being able to have evidence on one’s ranking with one’s peers can provide a better understanding of the perceived value of the performance.
  • Such evidence can inform subsequent goals for improvement.

But the interesting aspect was that the stories, the comments from the judges which said, for example “Good strong beat and nice changes” could be misleading. In our case the value was to be found in the numeric scores, but the scores in comparison with others and not in isolation.

Of course, whilst such considerations may be important in the context of rapper sword dancing, it would be inappropriate to apply such views to, say, learning analytics. After all, rapper dancing is important, whereas learning is supposed to be about fun, isn’t it.

Finally, we were pleased with the performance after our first dance. The judges gave us the following scores:

Stepping  Sword
handling
Dance
technique
Presentation Buzz
factor
Tommy Betty
Scores out of 15 Scores out of 10
Judge 1  11 10 11  6  6
Judge 2  9  9  10  7  4  4

What do you think – perhaps of our dance, but also on the value of metrics for such cultural activities? A video of the dance is available on YouTube and embedded below.

Posted in Evidence | Tagged: | 1 Comment »

Institutional Use of Social Media in China

Posted by Brian Kelly on 23 Mar 2012

Reviewing Recent Surveys of Institutional Use of Social Media

A number of recent posts have described institutional use of social media by UK universities, including surveys of use of Twitter, Facebook and YouTube and links to social media services. These surveys were followed by a post on Institutional Use of Social Media In Europe. Such evidence-gathering can be helpful in identifying patterns of usage and informing policy-making.

But what about institutional use of social media by UK universities hosted in other countries?

Following my post on Links to Social Media Sites on Russell Group University Home Pages André Shappo, who teaches IT Internationalisation at Loughborough University and who has written about his interests in Chinese social networking services on his blog, sent me an email in which he informed me that:

There are also a small number of UK universities using China’s Sina Wēibó social media. So far I have found 20 UK universities using Sina Wēibó, two of which are Russell Group universities. My list is at http://blog.sina.com.cn/s/blog_6fab54120100vhyh.html

André went on to add that:

Much more impressive is the number of western companies/brands using Sina Wēibó. So far I have found over 300 of them. My list is at http://blog.sina.com.cn/s/blog_6fab54120100wohn.html There are also other regional social media systems that could be used. It seems to me that UK universities are, in general, slow to take advantage of regional social media.

Are the twenty UK universities which André has identified leading the way? Are they established practices which could be adopted by others? A brief survey is described below which aims to provide evidence of the ways in which UK universities are using the Weibo social networking service. Note that as described in Wikipedia:

Sina Weibo (Chinese: 新浪微博; pinyin: Xīnlàng Wēibó; literally “Sina Microblog”) is a Chinese microblogging (weibo) website. Akin to a hybrid of Twitter and Facebook, it is one of the most popular sites in China, in use by well over 30% of Internet users, with a similar market penetration that Twitter has established in the USA. It was launched by SINA Corporation on 14 August 2009 and has more than 250 million registered users as of October 2011.

The Survey

The survey was carried out on Wednesday 29 February 2012. The findings are given in the following table, which includes links to the institutional entry on the Weibo service. Note that the survey is based on the twenty verified UK University accounts mentioned in André Shappo ‘s list. André has informed that there this is at least one additional institutional account, but this has not yet been verified.

Ref. No. Institution Sina Page Nos. of Fans
 1 Birmingham City University http://www.weibo.com/bcuchina  1,603
 2 Coventry University http://www.weibo.com/coventryuniversity  2,858
 3 Kingston University London http://www.weibo.com/kingstonlondon 13,836
 4 Lancaster University http://www.weibo.com/lancasteruni   3,190
 5 Leeds Metropolitan University http://www.weibo.com/leedsmetropolitan   1,330
 6 London Metropolitan University http://www.weibo.com/londonmet   1,455
 7 Northumbria University http://www.weibo.com/unnchina   2,605
 8 Sheffield Hallam University http://www.weibo.com/shuuk     800
 9 University of Bristol http://www.weibo.com/bristol   4,612
10 University of Derby http://www.weibo.com/derbychina     315
11 University of Essex http://www.weibo.com/universityofessex   4,408
12 University of Huddersfield http://www.weibo.com/hudchina 30,340
13 University of Leicester http://www.weibo.com/leicesteruniversity   3,648
14 University of Manchester Business School http://www.weibo.com/mbschinacentre   3,180
15 University of Northampton http://www.weibo.com/northampton   1,486
16 University of Sheffield http://www.weibo.com/sheffielduni   3,312
17 University of Sunderland http://www.weibo.com/sunderlandchina   2,941
18 University of Ulster http://www.weibo.com/u/2184786424   1,827
19 University of Wales, Newport http://www.weibo.com/newport     829
20 University of Westminster http://www.weibo.com/westminsterchina   4,919
TOTAL 89,494

Discussion

We can see that nineteen of the twenty organisations have a branded URL for their presence on the Weibo service. However it was interesting to note that whilst some institutions make use of the institutional name (coventryuniversityleedsmetropolitan and universityofessex) others provide a Chinese context (derbychinahudchina and westminsterchina).

The University of Huddersfield has the largest number of followers by a significant amount, with over twice as many followers as the next largest (Kingston University).

The University of Huddersfield’s site is illustrated. Since the site was viewed using the Chrome browser it was possible to use Google’s translate feature to read some of the posts. It was interesting to spot one post which illustrated how the University is making use of the service:

Revisiting Andre Shappo’s suggestion that:

It seems to me that UK universities are, in general, slow to take advantage of regional social media.

it does seem to me that, in light of changes to UK University funding models, we will see a greater emphasis on ways of marketing to and engaging with potential students from overseas. Clearly use of social media should provide a more cost-effective mechanisms for such engagement than physically transporting people and publications to countries such as China. However in order to maximise the benefits to the UK higher educational institutional as a whole we should be looking to identify and share best practices across the sector. This initial survey aims to provide an initial summary of use of Weibo by the early institutional adopters.

Paradata: The data published in the table was collected on Wednesday 29 February 2012.


Twitter conversation from Topsy: [View]

Posted in Evidence, Social Networking | 5 Comments »

Profiling Staff and Researcher Use of Cloud Services Across Russell Group Universities

Posted by Brian Kelly on 5 Mar 2012

Personal Benefits of Maximising Inbound Links to Research Papers

A recent post on this blog which described How Researchers Can Use Inbound Linking Strategies to Enhance Access to Their Papers reviewed personal experiences of the benefits of making use of third party services to provide inbound links to research publications.

In the post I suggested that the large numbers of downloads of my papers from the University of Bath institutional repository may be due to the enhanced Google juice provided by having links to my papers from such services. The purpose of the post was to suggest that researchers may benefit from increased access to their research publications if they are pro-active in using such services. My speculations may, of course, be incorrect; the downloads may be due to the quality of the papers rather than the numbers of in-bound links, for example :-). In addition, downloads themselves are not, of course an indication of quality. However since the papers, which have been through some form of peer-reviewing, will not have any influence if they are never read, I am happy to regard such approaches as helping to enhance the numbers of people reading the papers which may, or may not, lead to some form of subsequent ‘impact’.  Note that the slideshow on “Metrics: The New Black?” by Kristen Fisher Ratan which are available on Slideshare explores such considerations in more detail.

Profiling Institutional Use of Such Services

I recently came across the Libresearch blog which is provided by Jenny DelaSalle who, on her @JennyDelasalle Twitter profile describes herself as a “Research support Librarian: interested in bibliometrics, copyright, scholarly communications, and all sorts!”  I read her posts on topics including Webometrics and altmetrics: digital world measurementsWarwick people on external profile sites and 1,670 Warwick people on Academia.edu?. In the latter two posts she documented evidence of take-up of a number of third party services by researchers at the University of Warwick. Her post included a reference to one of my posts which profiled Russell Group university use of Google Scholar Citations. I am now able to build on Jenny’s work by using some of the survey methodology techniques she has helpfully documented in her blog to document evidence of take-up across the twenty Russell group university of popular third party service which provide links to research publications.

Having read the post on Warwick people on external profile sites it occurred to me that such institutional profiling work would benefit from being seen in a wider context. I therefore used the methodologies documented by Jenny in her blog post to gather similar information across the twenty Russell Group universities.

The findings are given in the following table. Note that the data for the Academia, LinkedIn and ResearcherID was collected on 1 March 2012 and the data for Google Scholar Citations on 3 March 2012.

Ref. No. Institution  Academia LinkedIn LinkedIn ResearcherID Google Scholar
Citations 
(Followers) (Current)
1 University of Birmingham   1,473   4,161   2,855      77 77
2 University of Bristol   1,603   3,687   3,167     231 55
3 University of Cambridge   5,287   7,371   6,919     400 83
4 Cardiff University   1,456   3,558   3,087     442 38
5 University of Edinburgh   3,341   5,947   5,536     241 75
6 University of Glasgow   1,572   3,147   3,646       27  70
7 Imperial College   1,383   7,615   6,306     399  78
8 King’s College London   2,182   5,078       25      64  35
9 University of Leeds   2,706    5,251   5,954    198  39
10 University of Liverpool   1,292   3,325   4,330    148  26
11 London School of Economics   1,909    6,907   1,914      36  37
12 University of Manchester   3,603    6,517   7,425     278  74
13 Newcastle University   1,509    3,583   3,001     173   94
14 University of Nottingham   2,022    5,107   6,010     315   52
15 University of Oxford   6,723    7,771   8,751     346 128
16 Queen’s University Belfast   1,100    1,978   5   1,989       88   24
17 University of Sheffield   1,701    4,171   5,269      255   36
18 University of Southampton   1,738    4,176   4,642      255   52
19 University College London   4,587    9,034   6,334      673  160
20 University of Warwick   1,770    3,667   2,855     199    34
TOTAL 48,957 102,051  88,03190,015  5,599 1,267

Discussion

As described in an article on Using LinkedIn For SEO:

Your profile can be an excellent source of SEO friendly links because:

    • LinkedIn has great authority in Google
    • Your website links can be given unique anchor text with the dofollow attribute
    • Your LinkedIn profile can have highly relevant content relative to the websites you own

It might be reasonable to assuming the use of the LinkedIn service comes mainly from staff and research students. In light of the popularity of the service might be find that encouraging researchers to provide links to copies of their papers hosted in their institutional repository will provide benefits not only for the individual researcher, but for the repository service itself, though the increased numbers of inbound links?

The DirectionsSEO site provides information on 5 Inbound Link Analysis Tools which may help to provide evidence of the value of inbound links. Initial experimentation with the Linkdiagnosis.com service suggests, however, that WordPress.com has the highest SEO ranking of domains linking to the University of Bath Opus repository service. But before concluding that researchers should be blogging about their research publications on the WordPress.com platform  I’d welcome feedback on the suggestion that the next stage for maximising access to research publication should be based on inbound linking strategies rather than further developments to institutional services.

Paradata:   As described in  a post on Paradata for Online Surveys blog posts which contain live links to data will include a summary of the survey environment in order to help ensure that survey findings are reproducible, with information on potentially misleading information being highlighted.

The data for the AcademiaLinkedIn and ResearcherID was collected on 1 March 2012 and the data for Google Scholar Citations on 3 March 2012.

The values for Google Scholar Citation for the universities of Birmingham and Newcastle include ‘UK’ in the search field in order to avoid including information from US and Australian universities with the same name.

It should also be noted that I was logged into the services when I gathered the information.

It should also be noted that the low values for LinkedIn followers for King’s College London and Queen’s University Belfast are felt to be due to the apostrophe used in the institution’s names. For example of search (carried out on 6 March 2012) on LinkedIn for King’s College London gives 3,418 hits but a search for Kings College London gives 294 hits.

Posted in Evidence, Repositories | 6 Comments »

How Researchers Can Use Inbound Linking Strategies to Enhance Access to Their Papers

Posted by Brian Kelly on 2 Mar 2012

The Value of Inbound Links to Resources

Via Smartr, the iPod Touch app I use to read articles which have been posted by Twitter followers, this morning I came across a link provided by a tweet which described an Inbound linking strategy to get to the top listing on google fast. The post described how the author, a web manager at Florida International University:

… developed  a strategy I would make inbound links to the FIU President’s Council site from places I can control a few of these places include FIU News,Alumni AssociationFIU A to Z index, blogs that have comments open, etc.  and on all those I make links using the words FIU President’s Council that link directly to the sites homepage.

The importance of providing links to a resource in order to maximise access to the resource is well understood – particularly, it seems, by spammers.  But how could such well-established techniques be used in an ethical way by researchers?

The answer, it seems to me, is quite simple. Researchers do have access to a wide range of web services which can legitimately provide links to their research publications.   This is an approach I have been using for several years. A summary of the numbers of publications which are listed in the services I use is given in the following Table.

Service My Account Summary
Microsoft Academic Search My details 39*
Google Scholar Citations My details  82
Researcher ID My details 10
Scopus My details  23
Academia.edu My details  50
Researchgate My details 110
Mendeley My details  23

*  The Microsoft Academic Search automatically includes papers from people with the same name.  These need to be manually excluded and there is a delay before updates are validated.  The service currently lists 286 papers, including many from medical researchers of the same name.  However only 39 papers have been claimed as authored by me.

It should also be noted that a number of the services provide links to the research papers (which in my case and normally hosted on the University of Bath institutional repository) although other services only provide the metadata.

Evidence of Enhanced Access

There is a cost to registering for such services and uploading details of one’s papers. However in practice I have found that it does not take a significant amount of time to upload relevant information and the services can provide useful information, such as helping to visualise one’s professional network and, as illustrated (taken from Mendeley) growth in  the number of citations, downloads, followers, etc.

But although individual  may or may not find such information of interest or value, there remains a question as to whether there is any tangible evidence of growth in downloads due to a policy of enhancing the numbers of links to such resources.

A possible answer to that question may be found form an analysis of the download statistics for items stored on Opus, the University of Bath institutional repository.

In order to make comparisons an image is shown of the top 20 most downloaded items provided by staff at UKOLN.

From this list we can see that I am a co-author of 15 of the top 20 items.

There may be several explanations for this:

Quality of the papers: Although two of my papers are the highest ranked papers which have been published at the W4A conference series I am quite happy to say that I am convinced that my colleagues have produced papers of much greater research value.

Social media optimisation: The paper on  Library 2.0: balancing the risks and benefits to maximise the dividends is the second most downloaded single paper from the University of Bath repository. The popularity of this paper was due to the large numbers of downloads shortly after the availability of the paper had been announced on this blog.  Although I am convinced that use of social media can also enhance access to peer-reviewed papers, several of the other popular papers in the above list were published between 2004 and 2007, before Twitter and before I was making significant use of the blog.

To conclude, I believe that adding information about one’s research publications to services such as Academia.edu, ResearchGate, Microsoft Academic Search and Google Scholar citations can increase the visibility of the papers to Google, as well as to users of the services, which may then lead to increased numbers of downloads, citations and take-up of the ideas described in the papers.

Do you agree?

Posted in Evidence, Papers, Repositories | 8 Comments »

An SEO Analysis of UK University Web Sites

Posted by Brian Kelly on 8 Feb 2012

Why Benchmark SEO For UK University Web Sites?

The recent JISC Grant funding 18/11: OER rapid innovation describes (in the PDF document) how this call is based on a conceptualisation of “open educational resources as a component of a wider field of ‘open academic practice’, encompassing the many ways in which higher education is engaging and sharing with wider online culture“. The paper goes on to remind bidders that “Effective Search Engine Optimisation is key to open educational resources providing benefits of discoverability, reach reputation and marketing“.

The JISC Call will be funding further developments of OER resources. But how easy will it be to find such resources, in light of the popularity of Google for finding resources? Or to put it another way, how Google-friendly are UK University Web sites? Are there examples of best practices which could be applied elsewhere in order to provide benefits across the UK higher education sector? And are there weaknesses which, if known about, could be addressed?

Recently an SEO Analysis of UK Russell Group University Home Pages Using Blekko was published on this blog which was followed by an Analysis of Incoming Links to Russell Group University Home Pages which also made use of Blekko. These surveys were carried out across the 20 Russell Group universities which describe themselves as being the “20 leading UK universities which are committed to maintaining the very best research, an outstanding teaching and learning experience and unrivalled links with business and the public sector“.

Having evaluated use of the tool across this sample (and spotting possible problem areas where university web sites may have multiple domain name and entry point variants) the next step was to make use of the tool across all UK university web sites.

The Survey

The survey began on 27 January 2012 using the Blekko search engine. A list of UK university web sites has been created within Blekko which automatically lists Blekko’s SEO rankings for the web sites. This data was added to a Google spreadsheet which was used to create the accompanying histogram.

It should be noted that the list of UK universities should not be regarded as a definitive list. There may be institutions included which should not be regarded as a UK university. In addition there may be a small number of institutions which may have been omitted from the analysis. The accompanying spreadsheet may be updated in light of feedback received.

Discussion

What can we learn from this data? The rankings for the top five institutions are given in Table 1. From these five institutions it might be useful to explore the reasons why these web sites are so highly ranked. [Note the Russell Group universities were inadvertently omitted from the list when this post was published. Details have been added and Table 1 has been updated].

Table 1: Top Five SEO Rankings according to Blekko
Ref. No. Institution Rank
(on 27 Jan 2012) 
 Current Blekko
Ranking
1 UCL 1,433.67 View
2 University of Liverpool 1,286.85 View
3 University of Leeds 1,284.97 View
4 Durham University 1,277.32 View
5 University of York 1,246.03 View

The embarrassing aspect of such comparisons lies in describing the web sites which are poorly ranked. Table 2 lists the SEO ranking figures for the lowest ranked institutional web sites. In addition to the table a screenshot of the table is also included, which was taken at 6 pm on Monday 6 February 2012 (note that the data and time shown in the image was the date the entry was added to the table).

Table 2: Bottom Five SEO Rankings according to Blekko
Ref. No. Institution Rank
(on 27 Jan 2012)
Current Blekko
Ranking
1 Trinity Saint David View
2 Cardiff Metropolitan University  – View
3 UWL (University of West London)  – View
4 UCP Marjon  28.91 View
5 De Montfort University  31.58 View

It should be noted that two of the three web sites for which no rank value was available were in Wales. This may suggest that technical decisions taken to provide bilingual web sites might adversely affect search engine rankings. Since such factors only affect Welsh institutions it would seem that these institutions will have a vested interested in identifying and implementing best practices for such web sites.

I must admit that I was surprised when I noticed a large institution such as De Montfort University listed in Table 2, with a Rank of 31.58. Viewing the detailed entry I found that a host rank value of 507.9 was given – very different from the rank of 31.58 which is listed in the table of all institutions.

Can We Trust the Findings?

Further investigation revealed further discrepancies between the entries in the overall list of UK universities and the detailed entries. In the process of creating listing for use with the Blekko service, listings for UK Russell Group universities (as well as the 1994 Group universities) were created.

Table 3 gives the Blekko Rank value together with the Host Rank value which is provided in the detailed entry for the web site. In addition the accompanying screenshot provides additional evidence of the findings as captured at 7 pm on 6 February 2012.

Table 3: Top Five SEO Rankings
for Russell Group Universities
Ref. No. Institution Rank
(on 6 Feb 2012)
Host Rank
(on 6 Feb 2012)
1 UCL 1,433.67 1,607.6
2 University of Liverpool 1,286.80 1,260.3
3 University of Leeds 1,284.97 1,141.8
4 LSE 1,224.59 1,201.1
5 University of Nottingham 1,138.99 1,382.9

Table 4 gives this information for the five Russell Group universities with the lowest SEO ranking values.

Table 4: Bottom Five SEO Rankings
for Russell Group Universities
Ref. No. Institution Rank
(on 6 Feb 2012)
Host Rank
(on 6 Feb 2012)
1 University of Birmingham   80.60 205.4
2 University of Sheffield 395.04 529.7
3 Imperial College 514.22 476.8
4 University of Manchester 610.86 694.2
5 Cardiff University 692.08 752.9

From these two tables we can see that there is are some disparities in the ranking order depending on which Rank value is used, but the numbers do not seem to be significantly different.

The Limitations of Closed Research

Initially I had envisaged that this post would help to identify examples of good and bad practices, which could be shared across the sector since, as described in the JISC called described above “Effective Search Engine Optimisation is key to open educational resources providing benefits of discoverability, reach reputation and marketing“. However it seems that gathering evidence of best practices is not necessarily easy, with the tools and techniques used for gathering evidence appearing to provide ambiguous or misleading findings.

This post illustrates the dangers of research which makes use of closed systems: we do not know the assumptions which the analytic tools are making, whether there are limitations in these assumptions or if there bugs in the implementation of the underlying algorithms.

These are reasons why open research approaches should be used, where possible. As described in WikipediaOpen research is research conducted in the spirit of free and open source software” which provides “clear accounts of the methodology freely available via the internet, along with any data or results extracted or derived from them“. The Blekko service initially appeared to support such open research practices since the web site states that “blekko doesn’t believe in keeping secrets“. However it subsequently became apparent that although Blekko may publish information about the SEO ranking for web sites, it does not describe how these rankings are determined.

It seems, as illustrated by a post which recently asked “How visible are UK universities in social media terms? A comparison of 20 Russell Group universities suggests that many large universities are just getting started“, that open research is not yet the norm in the analysis of web sites. The post describes:

Recent research by Horst Joepen from Searchmetrics [which] derives a ‘social media visibility’ score for 20 Russell Group universities, looking across their presence on Facebook, Twitter, Linked-in, Google+ and other media.

The econsultancy blog describes how:

The visibility score we use here is based on the total number of links a web domain has scored on the six social sites, Facebook, Twitter, LinkedIn, Google+, Delicious and StumbleUpon, while accounting for different weightings we give to links on individual social sites.

Image from eConsultancy blog

But what are these different weightings? And how valid is it to simply take this score and divide it by the size of the institutions (based on the number of staff and students) in order to provide the chart which, as illustrates, puts LSE as the clear leader?

It should be noted that this work is based on the analysis of:

roughly 207,900 links every week related to content on the websites of the Russell Group universities posted on Twitter, Facebook (likes, comments and shares), Linkedin, Google+ and social bookmarking sites StumbleUpon and Delicious. 

and is therefore not directly related to the SEO analysis addressed in this blog. This work is being referenced in order to reiterate the point of the dangers of closed research.

However the LSE Impact of Social Sciences blog, which hosted the post about this study, made the point that:

The LSE Impacts blog approach is that some data (no doubt with limitations) are better than none at all. 

I would agree with this view – it can be useful in gathering, analysing and visualising such data and in order to provide stories which interpret such findings. The Blekko analysis, for example, seems to be suggesting that the Universities of UCL, Liverpool, Leeds, Durham and York have implemented strategies which make their web site highly visible to search engines, but Trinity Saint David, Cardiff Metropolitan University and the University of West London seem to have implemented technical decisions which may act as barriers to search engines. The eConsultancy analysis, meanwhile, suggests that LSE’s approaches to use of social media services is particularly successful. But are such interpretations valid?

Unanswered Questions

Questions which need to be answered are:

  • How valid are the assumptions which are made which underpin the analysis?
  • How robust are the data collection and analysis services?
  • Are the findings corroborated by related surveys? (such as the survey of Facebook ‘likes’ for Russell Group universities described in a post which asked Is It Time To Ditch Facebook, When There’s Half a Million Fans Across Russell Group Universities?)
  • What relevance do the findings have to the related business purposes of the institutions?
  • What actions should institutions be taking in light of the findings and the answers to the first three questions?

What do you think?


Paradata: As described in a post on Paradata for Online Surveys blog posts which contain live links to data will include a summary of the survey environment in order to help ensure that survey findings are reproducible, with information on potentially misleading information being highlighted.

This survey was initially carried out over a period of a few weeks in January and February 2012 using Chrome on a Windows 7 PC and Safari on an Apple Macintosh. The survey was carried out using the Blekko web-based tool. A request was made to Blekko for more detailed information about their ranking scores and their harvesting strategies but the reply simply provided the limited information which is provided on the Blekko web site. Inconsistencies in the findings were noted and this information was submitted to the Blekko support email (and also via an online support form on the web site). However no response has been received.

The information on the survey of visibility of Russell Group universities on social media sites was based on posts published on the LSE Impact of Social Sciences and eConsultancy blogs.

Footnote: The findings for the Russell Group universities were omitted from the list of all UK universities when this post was initially published. The data has now been added and Table 1 and the associated histogram have been updated.


Twitter conversation from Topsy: [View]

Posted in Evidence, search | 9 Comments »

Analysis of Incoming Links to Russell Group University Home Pages

Posted by Brian Kelly on 1 Feb 2012

 

“Google is the Home Page!”

Back in April 2011 the UK government’s Alpha.gov.uk team discussed the implications for web developers of Google being the homepage. This was followed up last week with a post which asked “What does “Google is the homepage” mean IRL?“. The background to this discussion was a realisation that:

Since for the vast majority of people their web journeys … start with a search engine rather than a direct visit we should think of Google as the homepage and we should also feed Google, Bing and other search engines nice friendly urls.

These discussions are, of course, equally valid for university web sites.

SEO and the Importance of Links

Since the importance of ensuring that content can be easily found using search engines is true for content hosted on University web site, last year the JISC funded the Linked You project. The project provided a series of recommendations to enhance access to resources based on the design of URIs for resources hosted on institutional web site. But such advice covers only a part of the picture. In addition to what the report describes as implementing an “improved underlying URI structure” links to institutional web sites also play an important role.

2011 Overall Ranking AlgorithmThe 2011 Overall Ranking Algorithm chart provided by SEOMoz (Source: SEOmoz Ranking Factors) highlights the importance of links to a web site for search engine optimisation. As described on their page on How Search Engines Work the top factor which affects the visibility of web resources to search engine suggested in the Yahoo guidelines is “The number of other sites linking to it“. The SEOMoz web site goes on to describe how “Through links, engines analyze the popularity of a site & page based on the number and popularity of pages linking to them, as well as metrics like trust, spam, and authority“.

An analysis of links from authoritative sites to institutional web sites will therefore help us to gain a better understanding of the extent and profile of links from key resources and may help to inform policies and practices for helping to maximise the benefits which may be gained by implementing appropriate linking strategies.

Top Incoming Links According to Blekko

As described in a recent post which provided an SEO Analysis of UK Russell Group University Home Pages Using Blekko registered users of the Blekko search engine can access a range of data relevant to SEO for web sites. The following table provides a summary of one set of data: the top three highest ranked web sites which have links to the twenty Russell Group university web sites.

Table 1: Top Incoming Links According to Blekko
Ref. No. Institution Blekko Analysis
(free subscription
required)
Highest Ranked
Incoming Link
Second Highest
Incoming Link
Third Highest
Incoming Link
1 University of Birmingham [Analyse] twitter.com has
6 incoming links
http://www.youtube.com has
1 incoming links
en.wikipedia.org has
7 incoming links
2 University of Bristol [Analyse] twitter.com has
25 incoming links
http://www.google.com has
1 incoming links
en.wikipedia.org has
313 incoming links
3 University of Cambridge [Analyse] twitter.com has
10 incoming links
http://www.youtube.com has
2 incoming links
en.wikipedia.org has
328 incoming links
4 Cardiff University [Analyse] twitter.com has
11 incoming links
en.wikipedia.org has
227 incoming links
http://www.linkedin.com has
9 incoming links
5 University of Edinburgh [Analyse] twitter.com has
15 incoming links
http://www.youtube.com has
3 incoming links
en.wikipedia.org has
95 incoming links
6 University of Glasgow [Analyse] twitter.com has
8 incoming links
en.wikipedia.org has
315 incoming links
http://www.flickr.com has
7 incoming links
7 Imperial College [Analyse] twitter.com has
6 incoming links
http://www.youtube.com has
3 incoming links
en.wikipedia.org has
115 incoming links
8 King’s College London [Analyse] twitter.com has
20 incoming links
en.wikipedia.org has
762 incoming links
http://www.linkedin.com has
17 incoming links
9 University of Leeds [Analyse] twitter.com has
12 incoming links
http://www.microsoft.com has
3 incoming links
http://www.youtube.com has
2 incoming links
10 University of Liverpool [Analyse] twitter.com has
10 incoming links
en.wikipedia.org has
278 incoming links
http://www.linkedin.com has
8 incoming links
11 LSE [Analyse] twitter.com has
15 incoming links
http://www.youtube.com has
1 incoming links
en.wikipedia.org has
257 incoming links
12 University of Manchester [Analyse] twitter.com has
43 incoming links
http://www.youtube.com has
2 incoming links
en.wikipedia.org has
179 incoming links
13 Newcastle University [Analyse] twitter.com has
5 incoming links
en.wikipedia.org has
216 incoming links
http://www.microsoft.com has
1 incoming links
14 University of Nottingham [Analyse] twitter.com has
24 incoming links
http://www.youtube.com has
2 incoming links
en.wikipedia.org has
525 incoming links
15 University of Oxford [Analyse] twitter.com has
11 incoming links
en.wikipedia.org has
462 incoming links
http://www.google.com has
3 incoming links
16 Queen’s University Belfast [Analyse] twitter.com has
5 incoming links
en.wikipedia.org has
277 incoming links
http://www.youtube.com has
1 incoming links
17 University of Sheffield [Analyse] twitter.com has
13 incoming links
http://www.youtube.com has
4 incoming links
en.wikipedia.org has
68 incoming links
18 University of Southampton [Analyse] twitter.com has
4 incoming links
en.wikipedia.org has
258 incoming links
http://www.flickr.com has
1 incoming links
19 UCL [Analyse] twitter.com has
67 incoming links
http://www.youtube.com has
16 incoming links
en.wikipedia.org has
1,175 incoming links
20 University of Warwick [Analyse] twitter.com has
6 incoming links
en.wikipedia.org has
220 incoming links
http://www.linkedin.com has
4 incoming links

The following table summarise the services which are listed in the above table. Note that the data is collected used the Domain SEO Inbound Links search option which has the format “http://foo.ac.uk/ /domainlinks“.

Table 2: Summary of Highly Ranked Web Sites
Ranking
(based on nos. of
incoming links)
  Service       Page Rank  Occurrences     Nos. of Links  
1 [Wikipedia]   11,561.1 19  6,077
2 [Twitter]   18,323.8 20    316
3 [LinkedIn]     7,766.7   4      38
4 [YouTube]   13,530.4 11      37
5 [Flickr]     7,168.3   2        8
6 [Microsoft]     1,865.5   2        4
6 [Google]    10,929.5   2        4
Total  60   6,484

From this table we have evidence of the most highly ranked web sites which provide links to a selection of university web sites. As described above such authoritative web sites can have a significant role to play in enhancing the visibility of web sites in search engine rankings. The question then is how can this evidence help to inform policy and practice?

SEO and Links for Authoratitive Web Sites

Since all 20 Russell Group Universities have inbound links from Twitter in the list of top three highly ranked web sites it might appear sensible to explore best practices for use of Twitter to maximise search engine visibility. However Twitter’s robots.txt file blocks conforming indexing robots from accessing Twitter user’s profile and tweets:

Allow: /*?*_escaped_fragment_
Disallow: /search
Disallow: /*?
Disallow: /*/with_friends

Twitter would therefore appear to provide no SEO benefits. The case for Wikipedia is, however, very different, as indexing robots are permitted to index the content of Wikipedia pages, including links contained in Wikipedia articles. As can be seen from the above table all Russell group universities apart the University of Leeds have links from Wikipedia amongst the top three highest ranked web sites which provide inbound links (and Wikipedia is the fourth highest ranked web site which provides 371 inbound links to the University of Leeds). There are therefore over 6,000 links from Wikipedia to the twenty Russell Group universities with UCL having over 1,000 inbound links.

One strategy to enhance an institution’s ‘Google juice‘ would therefore appear to be to embed links to the institutional web site from Wikipedia articles. However contributions to Wikipedia articles should be provided from a Neutral Point of View and should be factual and verifiable. In order to avoid adding material with a vested interest, there would be advantages in encouraging members of the university to contribute to articles even if, and perhaps especially if, links are made to other institutions.

Top ranked sites linking to http://www.ox.ac.uk

It should be noted that the importance of incoming links from social media site sites has been highlighted recently in a post on the eConsultancy blog which asked “How visible are universities on social networks? The post pointed out that:

16 -24 year olds are heavy users of social networks, so you’d expect universities to have been ‘socially active’ by generating and posting interesting, engaging content (articles, images, video and audio) on their own websites, and then sharing it on social networks.

The survey, which was based on data harvested by the company and not made publicly available, suggested that Facebook generated over 80% of links from social media sites to Russell Group university web sites. It was interesting to note that the data provided in the Blekko analysis did not include links from Facebook despite that fact that, as illustrated in the accompanying image for the top incoming links for the http://www.ox.ac.uk domain, the highest ranked domains tend to be social media sites.

The Blekko findings for facebook.com show that the sites has a ranking of 23,420.3 and has 317,167,273 pages. It is therefore unclear as to why according to Blekko Facebook is not highlighted as a significant site with links to Russell Group universities.

Domain Naming Issues

Note that as described in a recent post on SEO Analysis of UK Russell Group University Home Pages Using Blekko, significantly different results can be obtained for analyses using variants of the institutional domain name, as illustrated below.

Table 3: Domain Name Variants
Ref. No. Institution /
Domain
Nos. of links
from Twitter
Nos. of links
from 
Wikipedia
1 University of Birmingham /
www.birmingham.ac.uk
  6  7
University of Birmingham /
www.bham.ac.uk
  0  23
2 University of Cardiff /
www.cardiff.ac.uk
 11 227
University of Cardiff /
www.cf.ac.uk
  1  69
3 Imperial College /
www.imperial.ac.uk
  6 115
Imperial College /
www.ic.ac.uk
  0  14
4 University of Liverpool /
www.liverpool.ac.uk
  1    1
University of Liverpool /
www.liv.ac.uk
10 278
5 University of Newcastle /
www.newcastle.ac.uk
  0    1
University of Newcastle /
www.ncl.ac.uk
  25 216
6 University of Southampton /
www.southampton.ac.uk
  5  53
University of Southampton /
www.soton.ac.uk
  4 258
7 University of Sheffield /
www.sheffield.ac.uk
 25  68
University of Sheffield /
www.shef.ac.uk
 13 339

As can be seen the official version of the domain name can have significantly fewer links than the variant (1 versus 216 links from Wikipedia in the case of http://www.liverpool.ac.uk and http://www.liv.ac.uk). Another suggestion, therefore, which may help maximise SEO benefits would be to ensure that links from highly ranked web sites, such as Wikipedia, link to a definitive version of the domain name.

Conclusions

This article began by describing how “Through links, engines analyze the popularity of a site & page based on the number and popularity of pages linking to them, as well as metrics like trust, spam, and authority“. The survey has identified the most highly ranked web sites which contain links to university web sites. It is suggested that in addition to implementing the recommendations provided by the Linked You project, related to the URI design of an institutional web site, an institution’s social media strategy should address ways in which links to the institutional web site can be provided in an ethical way. In addition, institutions should seek to ensure that links to their web sites are not fragmented across variants of the web site’s domain name.

Paradata: As described in a post on Paradata for Online Surveys blog posts which contain live links to data will include a summary of the survey environment in order to help ensure that survey findings are reproducible, with information on potentially misleading information being highlighted. This survey was initially carried out on 8 January 2012 initially using Chrome on a Windows 7 PC and was completed using Safari on an Apple Macintosh. In addition the figures in Tables 1, 2 and 3 were rechecked on 1 February 2012 and any changed values were updated.

It was noted that differing results were obtained if variants of the domain name (e.g. http://www.cardiff.ac.uk and http://www.cf.uk and www3.warwick.ac.uk and http://www.warwick.uk) and entry point URLs were used (e.g. http://www2.lse.ac.uk/ and http://www2.lse.ac.uk/home.aspx). This has been documented above.

Posted in Evidence | 2 Comments »

SEO Analysis of UK Russell Group University Home Pages Using Blekko

Posted by Brian Kelly on 25 Jan 2012

The JISC-funded Linking You Project

The Linking You project was provided by the University of Lincoln and funded by the JISC under the Infrastructure for Education and Research Programme. Its aim was to look at and make recommendations for improving the way that identifiers for .ac.uk domains are planned and managed in higher education institutions. The background to this work was described on the project web site:

The web is now fundamental to the activity and idea of the university. This Toolkit provides a standard way of thinking about your institutional URI structure, making it easier for people (and their browsers) to both remember your web addresses and locate where they are in your web site. It also helps prepare your institution for the world of linked data by proposing a clear and concise model for your data, making smooth integration with other systems easier and faster. A good URI structure can be easily understood by both humans and machines.

Although one of the benefits which implementation of the report’s recommendation was to “Improve discoverability of resources (and SEO)”, the Linking You project focussed primarily on identifiers for resources hosted within an institutional domain. This post aims to complement the Linking You work by gathering evidence on additional aspects: gaining an understanding of the size of institutional web sites, measuring the numbers of links to the institutional home page and other resources hosted within the domains and variants of the URI for the most important page on a web site – the institutional home page.

About Blekko

A few weeks ago James Burke (@deburca) introduced me to Blekko: a “search engine that slashes out spam, content farms, and malware. We do this by having a smaller crawl of 3 billion pages that focuses on quality websites. We also have a tool called a slashtag that organizes websites around specific topics and improves search results for those topics.” In response to the question “What information is available on the SEO pages?” the site describes how:

blekko doesn’t believe in keeping secrets. As part of our effort to make search more transparent, we provide a view of the data that our crawler gathers as it crawls the web.”

Every blekko search result has data associated with it that you can see. You can access it by either clicking on the SEO link tool in the second line of each result or else searching with the /seo slashtag. For example, apple.com /seo

Further information about Blekko (although it spells its name as ‘blekko’ on its web site I’ll use ‘Blekko’ in this post) is available on Wikipedia.

Using Blekko to Analyse Russell Group University Web Sites

What might Blekko tell us about UK university web sites? Blekko’s SEO pages provide details of the following information: geographic link distribution by state and country; inbound links; duplicated content; page source; sections and site pages. Blekko was used to survey the 20 Russell Group university home pages. The survey was carried out on 2 January 2012. However on 24 January it was noticed that the host rank, numbers of site pages and numbers of inbound links had changed significantly from 702.3 to 205.4, 945 to 8,406 and 24,442 to 627 respectively. The findings were rechecked but no other significant changes were noted.

The results are given in the following table. Note that you need to be logged in to the service in order to view the results.

Ref. No. Institution Blekko Analysis  Host
Rank
 Site   Pages   Inbound Links
(domain) 
Inbound Links
(URL) 
Outbound Links Notes
1 University of Birmingham [Analysis]  205.4   8,406 36,082 from
3,608 domains
627  {0 links} There are 13 outbound links (11 unique) from http://www.birmingham.ac.uk/index.aspx
2 University of Bristol [Analysis]  812.1  21,018 73,016 from
5,705 domains
40,101  5 links
3 University of Cambridge [Analysis]   1,042.7   16,041 309,734 from 10,145 domains 337,091  8 links
(7 unique)
4 Cardiff University [Analysis]         816.5   17,213 75,635 from
5,638 domains
59,590 5 links There are 29 links (26 unique) from http://www.cf.ac.uk/
5 University of Edinburgh [Analysis]      991.5   11,544 160,422 from 6,885 domains 168,545 {0 links} There is 1 outbound link from http://www.ed.ac.uk/home
6 University of Glasgow [Analysis] 1,090.5   12,243 100,271 from 9,454 domains 40,101 5 links
7 Imperial College [Analysis]  476.8   12,984 87,086 from
2,920 domains
34,566  {0 links} There are 3 outbound links from http://www3.imperial.ac.uk/
8 King’s College London [Analysis] 1,105.4   14,263 97,943 from
9,566 domains
26,986  {0 links} The are 11 outbound links (9 unique) from http://www.kcl.ac.uk/index.aspx
9 University of Leeds [Analysis]   1,141.8   16,617 134,501 from 10,886 domains 88,520 7 links
(5 unique)
10 University of Liverpool [Analysis]   1,260.3     4,727 59,797 from
9,794 domains
19,082  0 links
11 London School of Economics & Political Science [Analysis] 1,201.1   12,243 122,437 from 10,886 domains 29,795 {0 links} There are 3 outbound links from http://www2.lse.ac.uk/home.aspx
12 University of Manchester [Analysis]   694    13,292 186,893 from 5,193 domains 215,887  8 links
(7 unique)
13 Newcastle University [Analysis] 1,125    16,041 75,635 from
9,127 domains
  40,101 4 links
(3 unique)
14 University of Nottingham [Analysis] 1,380.8    16,041 94,551 from 10,759 domains   34,566  16 links
(12 unique)
15 University of Oxford [Analysis] 1,092.4    11,959 309,734 from 12,388 domains 290,563 1 link
16 Queen’s University Belfast [Analysis]   928.4   12,534 59,099 from
6,492 domain
  21,068 4 links
17 University of Sheffield [Analysis]   529.7    13,449 44,578 from  3,524 domains 20,050 13 links 8 outbound links from http://www.sheffield.ac.uk
are to http://www.shef.ac.uk/
18 University of Southampton [Analysis]   1,018.1    12,338 129,845 from 9,127 domains 69,132  9 links 5 outbound links from http://www.soton.ac.uk
are to http://www.southampton.ac.uk/
19 UCL [Analysis] 1,607.6  507,319 783,542 from 23,638 domains 476,718 9 links
20 University of Warwick [Analysis] 820     9,679 45,638 from
4,106 domains
  16,448 {0 links} 14 links (12 unique) for
http://www2.warwick.ac.uk/

Note that in the above table explanatory notes are given for figures displayed in braces e.g. {0}. Also note that the Universities of Newcastle and Nottingham both seem to have 16,041 pages.

A Tale of Two Domains

Whilst carrying out this survey it was noticed when checking inconsistencies that different results were obtained when using variants of the domain name and the institutional entry point. The following table lists known domain name variants. Note that the main domain was taken from the address given on the Russell Group web site.

Institution Main Domain Variant
University of Birmingham www.birmingham.ac.uk www.bham.ac.uk (Automatic redirect)
University of Bristol www.bristol.ac.uk www.bris.ac.uk
University of Cambridge www.cam.ac.uk Page at www.cambridge.ac.uk provides notice giving official domain name
University of Edinburgh www.ed.ac.uk www.edinburgh.ac.uk
University of Glasgow www.gla.ac.uk www.glasgow.ac,uk (Automatic redirect)
Imperial College www.imperial.ac.uk www.ic.ac.uk
University of Liverpool www.liv.ac.uk www.liverpool.ac.uk
University of Manchester www.manchester.ac.uk www.man.ac.uk (Automatic redirect)
Newcastle University www.ncl.ac.uk www.newcastle.ac.uk
University of Oxford www.ox.ac.uk www.oxford.ac.uk (Automatic redirect)
University of Southampton www.soton.ac.uk www.southampton.ac.uk

It should be noted that although the table describes the institutional part of the domain which is taken from the Russell Group web site, the analysis is carried out using www.official_domain.ac.uk  In two cases the well-established www. prefix was not used. These were www3.imperial.ac.uk and www2.warwick.ac.uk. However for the analyses the www.  prefix was used as it was felt that this would be the form used by the majority of users.

Discussion

The Blekko web site contains a page which summarises its core principles, which include:

Quality vs Quantity: blekko biases towards quality sites. We do not attempt to gather all of the world’s information. We purposefully bias our index away from sites with low quality content.

Source based, not link based: blekko does NOT rely solely on link based authority.Too many people engage in efforts to game search by linking for purposes other than navigation. blekko relies on human beings and their judgement of the authority of sources to dictate search results.

Open and Transparent: blekko makes freely available to its users all of the data that provide the underpinning of our search results. This includes web data, ranking information and the curation efforts of our users.

Blekko would appear to have a role to play in providing universities (which are unlikely to use ‘black hat’ SEO techniques such as use of link farms) with a better understanding of their visibility to search engines. However, despite the commitment to openness and transparency, the Blekko web site does not appear to provide details of their ranking algorithms.

Despite the current difficulties in interpretting the host rank in the above table, the information is provided as a snapshot, which may prove useful if Blekko subsequently do publish details. Of perhaps greater interest is the site pages column which, it would seem, contains the number of pages which have been indexed.

There does appear to be a significant diversity in the size of the Web sites, ranging from 4,727 (for the University of Liverpool) to  507,319 (for UCL), although apart from these two outliers the size of other Russell Group university web sites ranges from  9,679 and then 9,679 to  20,053.  These figures seem to suggest that there may be differing patterns of uses for institutional Web sites, ranging from the small and managed provision of focussed resources through to a more devolved approach. However although the managed approach would appear to have benefits, it does lead to the question as to where resources and services which are felt to be useful to the individual researcher or academic or their department should be hosted, and whether policies which acts as barriers to the creation of resources on an institutional service will result in content being hosted on cloud services.

Further interpretation of these findings will probably require an understanding of the institutional web environment.  However one aspect of the survey which does not require an understanding of the local context is the numbers of links from external services to the institutional web site. Links from authoritative web sites to a web site can influence the discoverability of the resources. A more detailed survey of such links will be published shortly.


Paradata: As described in a post on Paradata for Online Surveys it is important to document the tools and methodologies used when gathering evidence based on use of online tools in order that findings can are reproducible. In addition possible limitations in the tools or the way in which the tools are used should be documented.

This survey was carried out using the Blekko web-based service over the first two weeks in January 2012 and the findings rechecked on 25 January 2012 and changes recorded. Links are provided to the  results provided by the service. However in order to view the findings you will need to be signed in to the service (the service is free to join).

The findings for the University of Birmingham had changed significantly over a period of three weeks. It is not clear whether the variation was due to changes in the University of Birmingham web site, an artefact of the multiple domain names and entry point URLs for the University of Birmingham home page (http://www.birmingham.ac.uk/http://www.birmingham.ac.uk/index.aspxhttp://www.bham.ac.uk/ and http://www.bham.ac.uk/index.aspx all resolve to the same page) or changes at Blekko (e.g. reindexing the web site).

Note that the form of the domain name given on the Russell Group University Web site has been used. This is normally based on the full name, with the exceptions of Cambridge (which uses “cam.ac.uk”), Edinburgh (“ed.ac.uk”) and Glasgow (“gla.ac.uk”).

The results for the host rank are based on an undocumented algorithm. The information on the size of the site is dependent on the number of pages which are harvested. prefix was used as it was felt that this would be the form used by the majority of users.

Posted in Evidence | 3 Comments »

Links to Social Media Sites on Russell Group University Home Pages

Posted by Brian Kelly on 18 Jan 2012

Providing a Benchmark of University Use Of Social Web Services

In a recent post in which I gave My Predictions for 2012 I predicted that “Social networking services will continue to grow in importance across the higher education sector“. But how will we be able to assess the accuracy of that prediction? One approach is to see if there are significant changes in the number of links to social media services from institutional home pages.

The following survey provides a summary of links to social media services which are hosted on the institutional entry point for the 20 Russell Group universities.

Update: The information published about Imperial College was incorrect. This has been updated.

Ref No. Institution Services Type of Link Screenshot Icons for KCL
1 Birmingham None
2 Bristol None
3 Cambridge [iPhone] – [iTunesU] – [YouTube]
– [Facebook] – [Twitter] – [Flickr]
Direct link to institutional presence on social media service
4 Cardiff None
5 Edinburgh None
6 Glasgow [Generic bookmarks] – [WordPress] – [Facebook] – [Twitter] – [email] Link to visitor’s own presence on social media service.
7 Imperial College None [Delicious] – [Twitter] – [Digg] – [Stumble] – [Facebook] Link to visitor’s own presence on social media service.
8 King’s College London [Facebook] – [Twitter] – [YouTube] – [Favourites ] – [Digg] -[Delicious] – [RSS] See sidebar
9 Leeds [Facebook] – [Twitter]
10 Liverpool None
11 LSE [iTunesU] – [YouTube] – [Twitter] – [Facebook] – [Delicious] – [RSS] – [Flickr] Link to page on institutional web site providing information about institutional use of social media services.
12 Manchester [Facebook] – [Twitter] – [Google Maps] Direct link to institutional presence on social media service.
13 Newcastle [Facebook] – [Twitter]
– [YouTube] – [iTunesU]
Link to page on institutional web site providing information about institutional use of social media services.
14 Nottingham [Facebook] – [Twitter] – [YouTube] – [Flickr] – [LinkedIn] – [FourSquare] Direct link to institutional presence on social media service.
15 Oxford None
16 Queen’s University Belfast [Facebook] – [Twitter] Direct link to institutional presence on social media service.
17 Sheffield [Facebook] – [Twitter] – [YouTube] Direct link to institutional presence on social media service.
18 Southampton [Facebook] – [Twitter] – [YouTube] – [iTunesU] Direct link to institutional presence on social media service.
19 UCL [Twitter] – [YouTube] – [Facebook] – [Soundcloud] – [Flickr] – [iTunesU] Direct link to institutional presence on social media service.
20 Warwick [Facebook] – [YouTube] – [Twitter] – [iTunesU] Direct link to institutional presence on social media service.
Total 59 64

A summary of the number of occurrences of the services is given below.

Service Occurrences Note
Facebook 14 15 Links to institutional Facebook page.
Twitter 14 15 Links to institutional Twitter page.
YouTube   9 Links to institutional YouTube page.
iTunesU   6 Links to institutional iTunes page.
Flickr   4 Links to institutional Flickr page.
Delicious   2 3 (1) Provides access to links provided by the Careers Service and (2) allows page to be bookmarked.
Soundcloud   1 Links to institutional SoundCloud page.
LinkedIn   1 Links to institutional LinkedIn page.
FourSquare   1 Links to institutional FourSquare geo-location service.
Digg   2 Allows site to be bookmarked.
WordPress   1 Enables WordPress users to create post with link to University home page.
RSS   1 Purpose of this icon is not defined.
Stumble   1 Allows site to be bookmarked.
iPhone   1 Link to iPhone app about University
Google Maps   1 Link to map of University.
Generic Bookmarks   1 Link to bookmarks providing access to several social media services.
Email   1 Provides an email facility.
Total 5964

Discussion

If either all of the Russell Group University home pages had links to the same social media services or none did, this survey would be uninteresting. However since about 30% of the institutions do not have such links this seems to be suggesting that the value of having such links on a high profile page is not universally agreed.

For those institutions which do provide such links we can see that Facebook and Twitter are the most popular services, followed by social media sharing services. A number of services, including LinkedIn and FourSquare, have links from only a single institution.

It was also interesting to observe that although most institutions provided links to their institutional presence on social media services, a number of institutions used such links to allow visitors to provide links to the institution from the visitor’s own account, so that the institutional home page could be bookmarked or commented on.

Finally we can also observe how institutions label access to these services. This includes use of terms such as “Join us“, “Follow us“,”Find us on …“, and “xxx in the Social Media“.

From a user perspective we should also note that the different purposes provided by these links may be confusing. The norm is for links to provide read access to an institutional presence on a social media service. However in a number of cases the links are intended to allow users with accounts on particular services to bookmark or cite the institutional page on the service. Although this usage may be appropriate across a group of pages with the same purposes (for example, blog posts) this approach may cause confusion for a visitor who is either unfamiliar with the service or who expects the links to provide read access to the service.

Looking to the Future

This post has sought to identify patterns of usage of links to social media services on Russell Group university home pages and highlighted areas in which it may be beneficial for institutions to reappraise their uses of such services. However the main purpose of this survey was to provide a benchmark to help identify future trends in institutional use of social media.

Use of institutional home pages for such benchmarking can be beneficial since changes to institutional home pages will probably require approval at a senior level, and will therefore be less likely to reflect short term technological trends.

It will therefore be interesting at the end of the year to observe whether:

  • The current popular social networking services continue to remain popular.
  • New social media services are provided on social media services.
  • The ways in which the links to social media services are labelled and the functionality they provide changes.

I’d welcome comments on patterns across the wider University sector.

Posted in Evidence, Social Web | 9 Comments »

Final Reports from UKOLN’s Evidence, Impact, Metrics Work

Posted by Brian Kelly on 21 Dec 2011

During 2010-11 I led UKOLN’s Evidence, Impact, Metrics work. The aim of this work was to identify best practices for gathering quantitive evidence and supporting metrics on the use of networked services to support institutional and project activities.

An Evidence, Impact, Metrics blog was set up on the UKOLN Web site, but the usage statistics for the first few blog posts provided evidence of a lack of use of the blog. This evidence led to a decision to set up an Evidence category on the UK Web Focus blog which was used with related posts which were published on this blog. The aim of the blog posts was to raise awareness of the importance of metrics, explore ways of gathering and interpretting such metrics and encourage discussion on the advantages and disadvantages of using metrics, leading to recommendations on how metrics can be used.

As described in a report on Blogs Posts about Evidence of Impact by 13 December 2011 there had been 28,5907 views of the 35 relevant posts published in this category. In addition there had been 275 comments, although the numbers for the comments include trackbacks and may also contain automatically-generated links from other WordPress blogs which may subsequently be deleted.

This example provides an illustration of how metrics can be used. It should be noted that this does not say anything about the quality or relevance of the posts. It also summarises ways in which the metrics may be misleading (and note it was only when updating the figures on the numbers of comments posted on the blog that I became aware that automatically generated links to posts on this blog may subsequently be deleted.

The final report on this work has been published on the Evidence, Impact, Metrics blog. The report has been produced as a series of self-contained documents which are suitable for printing as well as being published in HTML format.

The following sections of the report are available:

  • Why the Need for this Work?:  This document provides the background to the work.
    [HTML] – [MS Word]
  • Summary of Events:  This document provides a summary of the three one-day workshops and talks given at other events.
    [HTML] – [MS Word]
  • Summary of Blog Posts:  This document provides a summary of the blog pots published related to this work.
    [HTML] – [MS Word]
  • Feedback from the Second Workshop: This document provides a summary of the feedback received at the second one-day workshop.
    [HTML] – [MS Word]
  • Summary of the Final Workshop:  This document provides a report on the third and final one-day workshop.
    [HTML] – [MS Word]
  • A Framework For Metrics:  This document provides a summary of the lightweight framework developed for gathering quantitative evidence.
    [HTML] – [MS Word]
  • Metrics FAQ:  This document provides an FAQ about the metrics work.
    [HTML] – [MS Word]

Note that the MS Word files are intended for printing in A5 format on a printer which supports double-side printing. For a number of the reports the content is duplicated to enable A5 summaries to be printed. The HTML format contain the same information in a more universal format.

As can be seen from the altmetrics manifesto the research community has strong interests in developing metrics which can help to identify evidence of value related to various aspects of research activities. The manifesto highlights the changes in ways in which research activities is being carried out and points out that “as many as a third of scholars are on Twitter, and a growing number tend scholarly blogs“.

The Evidence, Impact, Metrics work has sought to engage in a related area of work for those involved in both project and service who wish to make use of new approaches for which metrics can help to identify the value (or not) or new ways of working and share examples of appropriate best practices. Feedback on this work is welcomed.


Twitter conversation from Topsy:  [View]

Posted in Evidence | 2 Comments »

Beyond Blogging as an Open Practice, What About Associated Open Usage Data?

Posted by Brian Kelly on 14 Dec 2011

 

Should Projects Be Required To Blog? They Should Now!

A recent post on Blogging Practices Session at the JISC MRD Launch Event (#jiscmrd) contains access to the slides hosted on Slideshare used at the JISC MRD Programme Launch Meeting. In the talk I reflected on the discussion on Should Projects Be Required To Have Blogs? which took place initially on Twitter and then on this blog in February 2009.

The context to the discussion was described by Amber Thomas: “I should clarify that my colleagues and I were thinking of mandating blogs for a specific set of projects not across all our programmes“. During the discussion the consensus seemed to be that we should encourage a culture of openness rather than mandate a particular technology such as blogs. One dissenting voice was Owen Stephens who commented “I note that Brian omitted one of my later tweets – not sure if this was by mistake or deliberately because he recognised it for a slightly more light-hearted comment “i say mandate – let them write blogs!” – but I wasn’t entirely joking.

Owen’s view is now becoming more widely accepted across the JISC development environment with a number of programmes, including the recently established JISC Managing Research Data and the open JISC OER Rapid Innovation call both requiring funded projects to provide blogs. This current call (available in MS Word and PDF formats) states that:

In keeping with the size of the grants and short duration of the projects, the bidding process is lightweight (see the Bid Form) and the reporting process will be blog-based

and goes on to state that:

We would also expect to see projects making use of various media for dissemination and engagement with subject and OER communities, including via project blogs and twitter (tag: ukoer)

I’m pleased that JISC have formalised this requirement as I feel that blogs can help to embed open working practices in development activities as well as providing access to information which is more easily integrated into other systems and viewed on variety of devices than formats normally used for reporting purposes.

But how should projects go about measuring the effectiveness of their blogging processes and should should the findings we made openly available, as part of the open practices which projects may be being encouraged to adopt, and as data which is available under an appropriate open data – as we might expect data associated with these two programmes in particular – which is unencumbered by licencing restrictions which may be imposed by publishers or other content owners?

Openness for Blog Usage Data

In addition to providing project blogs there may be a need to be able to demonstrate the value of project blogs. And as well as the individual blogs, programme managers may wish to be able to demonstrate the value of the aggregation of blogs. But how might this be done?

A simple approach would be to publish a public usage icon on the blog. As well as providing usage statistics such tools should also be able to provide answers to questions such as “Has IE6 gone yet?” and “What proportion of visitors use a mobile device?“. But beyond the tools which we will be familiar with in the context of traditional Web sites there may be a need to be able to measures aspects which are of particular relevance to blogs, such as comments posted on blogs and links to blogs posted from elsewhere.

A post on Blog Analytic Services for JISC MRD Project Blogs explored this issue and described how tools such as Technorati and eBuzzing may provide lightweight solutions which may help to provide a better understanding of a blog’s engagement across the blogosphere. It should be acknowledged that such tools do have limitations and can be ‘gamed’. However in some circumstances they may help to identify examples of good practice. In addition gaining an understanding of the strengths and weaknesses of such analytic tools may be helpful if the altmetrics initiative which, in its manifesto, describes how “the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem” and goes on to “call for more tools and research based on altmetrics“.

In a post The OER Turn (which is, according to the author, ” the most read post of 2011 on [the JISC Digital Infrastructure] team blog“) Amber Thomas reflects on developments in the Open Educational Resources environment and describes how she now “find[s] [her]self asking what the “Open” in Open Content means” and concludes by asking “What questions should be asking about open content?“.

My contribution to the discussion is that I propose that when adopting open practices, one should be willing to provide open accesses to usage data associated with the practices.

This was an idea I explored in a post on Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers in which I highlighted the comment published in JISC-funded report on Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources which said that:

Being able to demonstrate your impact numerically can be a means of convincing others to visit your resource, and thus increase the resource’s future impact. For instance, the amount of traffic and size of iTunesU featured prominently in early press reports.

which suggests how quantitative data can be used to support marketing activities. But beyond such marketing considerations, shouldn’t those who believe in the provision of open content and who, in addition, wish to minimise limitations on how the content can be reused (by removing non-commercial and share-alike restrictions from Creative Commons licences, for example) also be willing to make usage statistics similarly freely available? And to argue that “my use case is unique and usage statistics won’t provide the nuanced understanding which is needed” is little different from those who wish to keep strict control on their data?

In other words, what is the limit to the mantra “set your data free“? Does this include setting your usage data free?


Twitter conversation from Topsy: [View]

Posted in Blog, Evidence | 7 Comments »

Paper on Metrics Accepted

Posted by Brian Kelly on 5 Dec 2011

“How can metrics be developed that fulfill requirements such as validity, reliability, and suitability?”

The Call for Papers was unambiguous about the important of metrics:

The goal of this symposium is to bring researchers and practitioners together to scope the extent and magnitude of existing …. metrics, and to develop a roadmap for future research and development in the field.

although there was an acknowledgement of the challenges in developing appropriate metrics:

Using numerical metrics potentially allows a more continuous scale for [measurements] and, to the extent that the metrics are reliable, could be used for comparisons. However, it is unclear how metrics can be developed that fulfill requirements such as validity, reliability, and suitability.

I’m pleased to say that I’ve had a paper accepted for the online symposium which will take place on 5 December 2011.  But what is the subject of the symposium?  I have recently published posts about the complexity of metrics for research papers, including issues such as download statistics for papers which are distributed across multiple services and metrics for providing answers to the question of “what makes a good repository?”.  Or perhaps the paper concerned metrics associated with use of Social Web services, another area I have addressed in several posts over the past year.

Both areas are very complex, with people questioning the validity of current approaches which are being taken to developing metrics which can be used to make comparisons – clearly areas worthy of research into how metrics can be developed and to have a questioning and critical appraisal of approaches which are being proposed. But this wasn’t the area addressed in the paper and in the symposium.

Online Symposium on Website Accessibility Metrics

The paper was, in fact, accepted for the Online Symposium on Website Accessibility Metrics (and is available in MS Word, PDF and HTML formats)

As the call for papers points out “conformance to the Web Content Accessibility Guidelines (WCAG) is based on 4 ordinal levels of conformance (none, A, AA, and AAA) but these levels are too far apart to allow granular comparison and progress monitoring; if a websites satisfied many success criteria in addition to all Level A success criteria, the website would only conform to level A of WCAG 2.0 but the additional effort would not be visible.”  It seems that rather than having simple four conformance levels, WAI are looking for more sophisticated algorithms which will be able to differentiate cases in which, for example, a Web page contains hundreds of images, none of which contain the alt attributes which are needed to enhance access to assistive technologies and a Web page which also contains hundreds of images, only one of which fails to have a meaningful alt attribute.  Currently both pages with fail WCAG conformance, since this requires all images to contain alt attributes.

It seem that the goal is a Klout score for Web accessibility, but with the difference that the underlying algorithms will be made public. But just as with Klout there is, I feel, a need to question the underlying assumptions which underpin the belief that accessibility can be determined by conformance with a set of rules, developed as part of the WAI’s model based on conformance with guidelines for content (WCAG), authoring tools (ATAG) and browsers and other user agents (UAAG). It is worth, therefore, making some comparisons between metrics-based tools such as Klout for measuring and the range of web accessibility measurement tools of which the now defunct Bobby tool was an early example.

Metrics for Online Reputation (Twitter)  Metrics for Online Web Accessibility Impact of Scholarly Research
Example of Tools Klout, Peerindex, … A-Checker, Bobby (defunct) and others listed in the Complete list of accessibility evaluation tools (last updated in 2006 with several broken links) Publish or Perish, Microsoft Academic Search, Google Scholar Citations, …
Purpose Measurement of online influence Measurement of accessibility of Web resources Measurement of productivity and impact of published scientific works
Underlying model Undocumented algorithms based on analysis of Twitter communities, posts, retweets, etc. Based on conformance with WAI model, based on three sets of guidelines, for content, authoring tools and user agents. Conformance, however, focuses only on WCAG guidelines. h-index, g-index, ….
Legal status No legal status. Conformance required in several countries. No legal status but may be used to determine research funding.
Limitations The system can be easily ‘gamed’. Tools such as Klout provide use of themselves in order to increase scores. The tools fail to take into account differences across different communities (e.g. use same approaches for comparing reputation of celebrities, brands and public sector organisations). The system can be easily ‘gamed’. The WGAC 1.0 guidelines promoted use of technologies developed within the host consortium, even when such technologies were little used. The tools fail to take into account the different ways in which the Web can be used (e.g. to provide access to information, to support teaching and learning, to provide access to cultural resources, for games, …). May be skewed by numbers of authors, self-citations, context of citations, …

Using Metrics In Context

However I do feel that there is value in metrics, whether this is for helping to identify the quality of research publications, online reputation or  accessibility of online resources.  The difficulty arises when the metric is regarded as the truth, and becomes a goal in itself.  So whilst I feel there is validity in publishing details of Klout, PeerIndex and Tweetstat statistics across a selection of institutional Twitter accounts in order to help understand patterns of usage and, I should add, to understand the limitations of such metrics-based tools, I also feel that institutions would be foolhardy to regard such statistics as concrete evidence of value.  Rather such statistics can be useful when used in conjunction with other evidence-based parameters.

The danger with Web accessibility metrics is that they have been used as a goal in their own right. In addition, sadly, the previous government has mandated conformance with these metrics across Government Web sites.  And back in 2004 WAI gave their views on Why Standards Harmonization is Essential to Web Accessibility, which seems to be leading to WCAG conformance being mandated across EU countries. If a proposal on “Why Online Reputation Standards Harmonisation is Essential” was published, especially by the body responsible for the online reputation standard which was proposed as the only standard which should be used,  there would be uproar, with, I would hope, the research community seeking to explore limitations in the proposed standard.

Fortunately the organisers of the WAI symposium do seem to be aware or criticisms of their approaches to Web accessibility as providing the only legitimate approach.  The Call for Papers invited contribution which “may include approaches for measuring ‘accessibility in terms of conformance‘ (metrics that reflect violations of conformance of web content with accessibility guidelines such as WCAG or derivatives such as Section 508) and ‘accessibility in use‘ (metrics that reflect the impact that accessibility issues have on real users, regardless of guidelines)” (my emphasis).

The fundamental objection myself and fellow author of our series of paper on this subject, is that accessibility is not an innate characteristic  of a digital object, but of the user’s difficulty in engaging with an object to fulfil a desired purpose. The view that all Web resources must be universally accessible to everyone, which underlies pressures for organisations to conform with WCAG guidelines, is a flawed approach.

So if I’m critical of metrics related to conformance with guidelines, what do I feel is needed?  Our papers argues for making use of metrics related to guidelines related to the processes surround the development of online resources.  In the UK the BS 8878 guidelines provide the relevant Code of Practice.  As Jonathon Hassell pointed out in a recent post on For World Usability Day: The state of accessibility on the HassellInclusion blog:

[BS8878’s] goals were to share best practice in the first Standard about the process of accessibility rather than it’s technical aspects. It’s succeeded in helping harmonise the separate worlds of inclusive design, personalisation and WCAG approaches to accessibility.

Jonathon went on to add:

Uptake is always difficult to measure, and it’s still early days for organisations to go public and say they have changed the way they work to follow BS8878. However, some organisations already have including: Royal Mail, beta.gov.uk and Southampton University. And many others are working on it. BS8878 is one of the best-selling standards BSI have ever created – so it’s met their goals. I’ve trained many organisations globally and my BS8878 presentations on slideshare have been viewed by over 6000 people from over 25 countries.

There is a need to encourage greater take-up of BS 8878, and I hope our paper will help in describing ways in which such take-up can be measured.

But what of the development of new ways of measuring WCAG conformance? As described in a paper on Involving Users in the Development of a Web Accessibility Tool at a cost of over 2M Euros the EU-funded European Internet Accessibility Observatory Project developed a robot for measuring conformance with WCAG guidelines across a range of government Web sites in the EU. As described on the eGovernment Monitor Web site has released  the eAccessibility Checker which builds on the EU-funded project and can be found at http://accessibility.egovmon.no/.  However looking at the results of a survey carried out last month across a number of Norwegian Web sites it seems that  there of a number of problems which are experienced by over 80% of the Web sites! If such tools report a low-level of conformance can’t we then use this as evidence of the failures of the WAI model rather than, as has been the case in the past, a failure in organisations to be willing to enhance the accessibility of their services?

Posted in Accessibility, Evidence | Leave a Comment »

Paradata for Online Surveys

Posted by Brian Kelly on 29 Nov 2011

In a recent post on “Surveying Russell Group University Use of Google Scholar Citations” I used Google Scholar Citation’s search facility to audit the numbers of researchers in the twenty Russell Group Universities who have claimed profiles on the service.

Looking at my own host institution, which is a member of the 1994 Group, at the time of writing there are 33 profiles for a search for the University of Bath but only 23 profiles for a search for the “University of Bath”.  We can see that the findings differ depending on the search syntax, such as whether the search term in enclosed in quotes or not.

There is therefore a need to be explicit about the way in which the searches are constructed in order to ensure that findings are reproducible.  In previous surveys I have tried to document the survey methodology in the text of the blog posts but it has occurs to me that the specific details may be overlooked.  I therefore feel that further surveys should include explicit details of the survey paradata, a term which is defined in Wikipedia as “data about the process by which the survey data were collected“.

The blog posts I have published have, wherever possible, provided live links to the services used to gather the data. Such links may provide parameters which may differ depending on factors such as the browser environment you are using,  The hyperlink used for the search described above, for example, is:

http://scholar.google.com/citations?view_op=search_authors&hl=en&mauthors=%22University+of+Bath%22&after_author=sDUAAPv___8J&astart=20

As described in the Google documentation:

The hl parameter specifies the interface language (host language) of your user interface. To improve the performance and the quality of your search results, you are strongly encouraged to set this parameter explicitly.

This is a simple query. However the Google search box in my browser produces the following URL as a result of a search for google scholar citations:

http://www.google.co.uk/search?client=safari&rls=en&q=google+scholar+citations&ie=UTF-8&oe=UTF-8&redir_esc=&ei=6lXSTrP8FoOh8gOkoJ0I

In order to ensure that a rich description of the survey environment is available, my intention is that surveys published in future will contain survey paradata details along the lines illustrated in the following table, which describes the survey published in the recent post.

Details Description Data Note
Search term The official name of the host institution. Column 1 Name is not included in quotes.
Date The date of the survey. 24 November 2011 If the survey is carried out across several days, this should be documented.
Search service Google Scholar Citations service. http://scholar.google.com/citations If, for example, a UK version of the service is released, this should be documented.
Browser environment Name & version of browser and platform. Safari v 5.5.1 running on an Apple Macintosh Include details of browser plugins if this is felt to be relevant.
Language The default language (English) is used. EN
Search options Search options selected. Used the “Search Authors” option. If additional search options are available they should be documented.
Location Details of where the survey was carried out. Search carried out in Bath, UK.
User account Information on whether surveyer was logged in. Search carried out whilst logged in to Google.
Possible problem areas
  • There may be name clashes (e.g. University of Newcastle and University of Newcastle, New South Wales).
  • Searches may include email address fields as well as name of institution

Any suggestions on things I may have missed?

Posted in Evidence | Tagged: | 7 Comments »

Google Scholar Citations and Metadata Quality

Posted by Brian Kelly on 28 Nov 2011

Back in 2005 Debra Hiom, Amanda Closier and myself wrote a paper entitled “Gateway Standardization: A Quality Assurance Framework For Metadata” which was published in the Library Trends journal. The paper (which is available in MS Word and PDF format from the University of Bath repository) described the systematic approaches to ‘spring-cleaning’ metadata which the SOSIG subject gateway which, at the time, was a subject gateway in the Resource Discovery Network.  The approaches which were taken at SOSIG reflected a quality assurance framework which was being developed by the JISC-funded QA Focus project which was described in a paper on “Developing a quality culture for digital library programmes“.

The quality assurance approaches or metadata we described in the papers was focussed primarily on the service providers. However, six years later, the importance of the quality of metadata for resource discovery is no longer just of relevance to service providers. In a Web 2.0 environment in which content providers can make their teaching and learning and research outputs available on a wide range of services without the mediation of information professionals there is a need to ensure that a wider range of content providers are aware of risks that poor quality metadata can lead to valuable content being difficult to find.

I became aware of such risks while Surveying Russell Group University Use of Google Scholar Citations which I described in a recent blog post.  As mentioned in the post I became aware of the dangers of over-counting the numbers of researchers who have claimed a profile by aggregating researchers from the University of Birmingham with those from the University of Birmingham at Alabama or those from Newcastle University with Newcastle University,  New South Wales.

 Of further investigation I discovered entries from researchers who had misspelt the name of their university by using “univeristy” – a common typo which I myself have made. Currently it seems there are only 33 such misspellings.
In our paper we described how:

We have recommended to the JISC that those JISC-funded projects making significant use of metadata should address these issues as part of the project’s reporting procedures.

Whilst the issues referred to are still valid for projects which have significant metadata requirements, we now have the question of approaches which researchers can use when they are uploading information about their papers which may be harvested by a range of services, who aren’t in a position to implement metadata quality checking tools in services which may be used by full-time information management staff.

So what can individual researchers do to ensure that their papers don’t become difficult to find in tools such as Google Scholar Citations?

I have experimented with tools such as Collabgraph, a finalist in the Mendeley/PLoS API Binary Battle. This helped me to spot that a number of my papers listed in my Mendeley library had listed two sets of co-authors in a single string.  This brought home to me the potential benefits of visualisations for spotting errors in textual data.

In addition to use of such tools a recommendation I am making to colleagues is to create a profile and check you pages while the service is still new and there are only small numbers of users.  This means, for  example, that I can search for authors called “Kelly” and discover that there are currently only 26 entries and that there are no duplicate entries for me.

I can also search for my department, UKOLN, and check that the entries are correct.In this case we are fortunate in having a unique name for our department.  However in many other cases there may be legitimate variants: for example I currently find seven entries for Computer Science, Southampton and 43 entries for ECS, Southampton with the discrepancy due, in part, to many researchers having a foo@ecs.southampton.ac.uk email address.

As I started to reflect on ways in which errors could be introduced into such services and ways in which end user might search for resources I realised that although early adopters can gain benefits in adopting profiles in such services (by gaining additional exposure to one’s research and being able to more easily spot errors when there is only are small numbers of  profiles available) at some point the bottom-up approach will suffer from limitations. What we really need will be the centralised provision of quality assured metadata about research publications.  But services such as Google Citations Scholar won’t disappear in the short term (although, as with a range of other Google services, they could disappear in the future if they turn out not to be aligned with Google’s business interests).  My conclusions: be an early adopter in order to provide another mechanism for making ones research papers more visible but be prepared to accept the risk that the benefits may not last forever.

Posted in Evidence, Repositories | 1 Comment »

Surveying Russell Group University Use of Google Scholar Citations

Posted by Brian Kelly on 24 Nov 2011

Measuring Take-up of Google Scholar Citations

A recent post gave some “Thoughts on Google Scholar Citations“. I concluded by suggesting that researchers could find it useful to claim their account on Google Scholar Citations and  ensure that the details of their papers are accurate but speculated on whether there would be barriers to researchers doing this. In order to investigate the level of usage of Google Scholar Citations in the UK higher education sector a survey of its usage across the twenty Russell Group Universities has been carried out and the findings published in this post. The institution’s name, as listed in the first column, was used as a search term.  The number of entries gives the current number of researchers found, with a link provided to the current final page of results.  In addition in order to investigate whether the service is being used by new researchers, who are likely to have a low number of citations or well-established researchers with large numbers of citations, a summary of the top three researchers having the largest numbers of citations is give, with links to the researchers profile together with details of the numbers of citations for the three researchers having the lowest numbers of citations. The results are given in the following table.  The survey was carried out on Tuesday 22 November 2011

Institution Nos. of entries Highest Citations Lowest Citations
University of Birmingham    33 *  (18,989)* – 5,817 –  5,7704,243  13 – 15 – 16
University of Bristol 40   21,761 –  9,223  –   8,271   0  –  0  –  6
University of Cambridge 73   46,12118,272 –  17,806   0  –  0  –  0
Cardiff University 20    6,665 –   6,142  –   3,823   0  –  0  –  1
University of Edinburgh 68   13,844 – 12,158  –   9,082   0  –  0  –  0
University of Glasgow 64   13,22811,718  –   5,773   0  –  0  –  1
Imperial College 71   31,261 –   9,630  –   9,303   0  –  4  –  4
Kings College London 23     6,052  – 6,030  –    4,513   0  –  0  –  0
University of Leeds 30   12,686 –  6,780  –    6,732   0  –  1  –  4
University of Liverpool 15   34,49920,014  –  14,717   1  –  1  –  8
London School of Economics 17   14,191 –  9,222  –    6,303   0  –  0  –  0
University of Manchester 73   19,57218,155 –  13,708   1  –  1  –  2
Newcastle University    44 *   11,18510,679  –   3,111   0  –  1  –  4
University of Nottingham  40   11,506 –   9,084  –   5,661   0  –  0  –  0
University of Oxford 109   25,36324,311 –  16,639   0  –  0  –  0
Queen’s University Belfast  15    2,357  –  1,913  –   1,667   1 – 24 – 37
University of Sheffield  32    5,735  –  3,318  –   2,980   0  –  1 –   1
University of Southampton  39   42,197  –  9,009  –  4,708   0  –  0  –  4
University College London 145   31,44030,842 –  20,058   0  –  0  –  0
University of Warwick  23     3,194 –  2,923  –   1,850   0  –  0  –  0
Total      974 * **

* It was noted that the first entry for a search for the University of Birmingham referred to Mary Vignolo Wheatley from the University of Alabama at Birmingham. The numbers of Google Scholar Citation entries is therefore overstated for the University of Birmingham and potentially for the other institutions which are listed. ** I was informed after publication of this post that of the 44 citations quoted for Newcastle, 11 are actually for the University of Newcastle, NSW, Australia. Such errors could creep in for other institutions for which there are name clashes (e.g. York University and New York University). This highlights the need for globally unique institutional identifiers – but such discussions are outs the scope of this post. It was also noticed that the third entry for the University of Cambridge referred to Alan Turing, the English mathematician, logician, cryptanalyst, and computer scientist who, as described in Wikipedia, lived from 1912-1954.  Unsurprisingly his Google Scholar Citation entry states that his email address has not been verified!

Discussion

In a recent discussion about Google Scholar Citations I have been told about the difficulties in claiming authorship of papers after one has left one’s host institution and no longer has an institutional email address.  A second discussion I heard from one person who claimed his Google Scholar  account shortly before leaving his host institution who provided an alternative email account which could be used one his institutional email account had been deleted. The first example highlights a potential difficulty in asserting authorship of papers after one has left the host institution and the second example describes one way in which such potential problems can be addressed.  It would therefore appear sensible for researchers to claim a Google Scholar account while they are in a position to associate it with papers published in their host institution. An interesting issue, therefore, will be who should take responsibility for advising researchers on best practices for using services such as Google Scholar Citations.  Should the library include such advice in its training courses for new researchers?

Conclusions

A recent post by Wouter Gerritsma, subject librarian and bibliometrician at Wageningen UR Library described “How Google Scholar Citations passes the competition left and right“. Wouter’s post concluded:

Google Scholar is only about five years old. Give them another five years and they will have changed the market for abstracting and indexing database totally. If only 20 percent of all scientists make their publication lists correct (also editing of the references which can be done to improve the mistakes Google has made) even without making them publically available, Google sits on a treasure trove of high quality metadata. Really interesting to see how this story will develop.

It will be interesting to see how this story develops.  And as the launch of Google Scholar Citations was only announced a week ago today, we do have an opportunity to observe its take-up within our institutions from its early days.  Monitoring the take-up of the service, the approaches taken in managing the information and understanding difficulties in such management activities will be valuable not only in developing plans for use with other services in this space. Hmm, I wonder if Google Scholar Citations has APIs which will enable such monitoring approaches to be implemented in a scalable way?

Posted in Evidence, Repositories | 15 Comments »

Thoughts on Google Scholar Citations

Posted by Brian Kelly on 22 Nov 2011

Citation Analysis Services

I recently wrote a post entitled “Will the Real Scott Wilson Please Stand Up, Please Stand Up” in which I described my initial experiences with the Microsoft Academic Search service.  I have to admit that I was impressed by the user interface and how, for example, it depicted links with my co-authors.

Revisiting Microsoft Academic Search

The main limitation with the Microsoft Academic Search service was, I felt, the accuracy of the data and the need to get author buy-in in order that authors could claim their own papers and remove papers incorrectly attributed to them.  The information it has about me, for example, suggests that I have published 56 papers, including one dating back to 1979. In fact it should know about 30 of my papers, the earliest of which was published in 1994.

Several weeks ago I edited my publications list to remove papers written by other Brian Kellys.  These edits have been accepted and when I sign in I get confirmation of the 38 papers I have confirmed authorship of and the 18 which have been removed from the list. However the wiki-style approach to editing the content means that edits have to be confirmed and this does not appear to have happened.  I therefore appear to be claiming more publications that is the case and, possibly, the citation statistics (G-Index=11 and H-Index=6) for my papers may be inaccurately calculated.

Google Scholar Citations

Whenever I come across a new service which appears to provide value I am also interested in seeing if there are alternative offerings. In part this is to ensure that I don’t find myself being locked into a single vendor. But in addition it can also help to see how other providers address the same area. As the Microsoft Academic Search service is based on harvesting metadata about papers hosted on institutional repositories, publishers Web sites and similar resources we should expect to see similar competing services.  I was therefore pleased when I received an email last week which announced that the Google Scholar Citations service, which I had signed up to during the beta testing, had been opened as a public service.

A post was published on the Google Scholar blog on Wednesday 16 November 2011 entitled “Google Scholar Citations Open To All‘ which described how:

You can quickly identify which articles are yours, by selecting one or more groups of articles that are computed statistically. Then, we collect citations to your articles, graph them over time, and compute your citation metrics – the widely used h-index; the i-10 index, which is simply the number of articles with at least ten citations; and, of course, the total number of citations to your articles. Each metric is computed over all citations and also over citations in articles published in the last five years.

My Google Scholar Citations page is illustrated below. In comparison with my Microsoft Academic Search page this page appears somewhat limited in its functionality. It also has much less social connectivity, with links to only six of my co-authors who have registered for the service.

In addition to differences in the user interface and the social connections, Google Scholar Citations also has differences in the papers it has analysed and the corresponding citation indices, giving a H-index of 11 (in comparison with Microsoft Academic Search’s H-index of 6). Google Citations also provides a I10-Index score of 12 whereas Microsoft Academic Search provides G-Index score of 11.

Google Scholar Citations’ analysis of the papers indexed by Google Scholar seems to be based on a more accurate representation of my papers, possibly because I verified my papers some time ago.  Google Scholar also includes a number of popular articles I wrote which haven’t been deposited in the University of Bath repository and therefore don’t seem to have been indexed by Microsoft Academic Search, such as the Ariadne article on “An accessibility analysis of UK university entry points” for which there have been 28 citations. But in addition a paper on “Using networked technologies to support conferences”  delivered at the EUNIS 2005 conference which has been deposited in the in the University of Bath repository has been indexed by Google Scholar but not by  Microsoft Academic Search.

Whilst investigating Google Citations I came across a tweet from Les Carr who provided a link to his Google Citations page, which is illustrated below (which brought to my attention the paper on “Earlier web usage statistics as predictors of later citation impact” from 2006 which will be worth reading in light of Social Web developments since the paper was published in 2006).

Carr

In order to make some further comparisons between the coverage and citation analyses of Google Citations and Microsoft Academic Search I’ve summarised details for Les Carr together with the co-authors of my papers who have registered with Google Scholar Citations in the following table.

Name Microsoft
Academic

Search (MAS)
Google
Citations
registered on(GC)
Nos. of
publications (MAS)
Nos. of
publications (GC)
Nos. of
citations (MAS)
Nos. of
citations (GC)
G-Index (MAS) I10-Index (GC) H-Index (MAS) H-Index (GC)
Brian Kelly Link  Link  56  83  153 498 11 12  6 11
David Sloan Link Link  42  67  204 615 13 12  7 12
Jane Seale Link Link   6  85    49  714   6 14  4 12
Helen Petrie Link Link 106 172  569 1,397  22 34 15 18
Lorcan Dempsey Link Link  10 110    29 1,139   5 30  1 19
Alastair Dunning Link Link   3  13    8   29   2   1  2   3
Les Carr Link Link 169 206 1,158 1,558  28 42 17  21

It should be noted that:

  • The Microsoft Academic Search entry for Jane Seale has her affiliation listed as the University of Southampton. She is now based at the University of Plymouth so her citation statistics may be split across two entries.
  • There are two Microsoft Academic Search entries for Lorcan Dempsey: entry 1  and entry 2.
  • here are two Microsoft Academic Search entries for Alastair Dunning: entry 1  and entry 2.

Discussion

I’m pleased that Google have provided an alternative to Microsoft for providing details of citations for research publications (there are similar services, of course, but I thought it would be worth focusing this post on a newly released service and provide comparisons with a service I described recently).

Microsoft Academic Search seems to have taken an approach of indexing as many research papers as it can find, associating the papers with author and institutions. The Microsoft Academic Search  entry point currently states that it provides access to “6,684,802 publications and 18,831,151 authors, 5,472 updated last week“.  Papers are automatically assigned to organisations, with the details for the University of Bath providing the following information: Publications: 29,331; Citation Count: 131,732; H-Index: 96 and 1,638 authors. In addition papers may also be assigned to departments with the details for Bath/UKOLN providing the following information: Publications: 262; Citation Count: 932; H-Index: 15 and 245 authors.

The problem with such automated processing is that the data can be flawed with.  In contract the Google Scholar Citations requires users to opt-in before their papers are assigned to their Google account.  This means, for example, that Google Scholar Citations currently has details for only 18 authors from the University of Bath.

It seems to me that rather than the functionality of the services I’ve described, the main challenges will be getting buy-in from the authors’ whose papers have been indexed.  They will be both a significant user community for such services as well as possibly having responsibility for cleaning up the data.

Some questions which came to mind when I was looking at these services:

  • What is being indexed?  The Microsoft Academic Search service seems to have indexed primarily my peer-reviewed papers which I have deposited in the University institutional repository and from publishers’ databases. The Google Scholar Citation service, in contrast, seems to have also included papers from the UKOLN Web site which I wouldn’t have classed as ‘papers’.  I have removed papers which don’t fit in with my view of what should be included, but I appreciated that such definitions are likely to be very subjective.
  • Motivation to manage one’s content. What is the motivation to manage one’s content?  Since the automated harvesting and assignment of papers is liable to lead to errors, there will be a need for the data to be cleansed.  But what are the motivating factors for authors to do this?
  • Barriers to the management of one’s content.  Although authors may have motivating factors, such as ensuring that popular services provide an accurate view of their research publications, there may also be barriers to updating one’s data.  This might include the user interfaces provided by the services, the turnaround time for changes to be approved and the requirements for a Windows Live ID (in the case of Microsoft Academic Search) or a Google ID (in the case of Google Scholar Citations).

I recently came across a tweet from Guus van Brekkel (@digcmd) who described:

How Google Scholar Citations passes the competition left and right at WoW! Wouter on the Web bit.ly/uw8ppc

The tweet introduced me to the WoW!ter blog, written  by Wouter Gerritsma, subject librarian and bibliometrician at Wageningen UR Library. In the post Wouter gave his thoughts on the service:

 Google Scholar Citations really excels at finding publications you completely forgot about. 

and went on to make comparisons with other alternatives:

Google Scholar easily beats ResearcherID since it updates automatically and Scopus ID because you can make your list with citations publically available. To make your publication list openly available is really recommended to all scientists, it helps your personal branding.

although he admitted that:

there are disadvantages to Google Scholar as well. The most serious at this moment all kind of ghost citations.

Wouter concluded:

Google Scholar is only about five years old. Give them another five years and they will have changed the market for abstracting and indexing database totally. If only 20 percent of all scientists make their publication lists correct (also editing of the references which can be done to improve the mistakes Google has made) even without making them publically available, Google sits on a treasure trove of high quality metadata. Really interesting to see how this story will develop.

Perhaps the risk of failing to engage with the service and update the information which Google has will turn out to be the motivating factor for updating the content.  I’ve updated my content and started to email my co-authors so that they are listed. Have you updated your papers?  And if not, I’d be interested to know the reasons why not.


Twitter conversation from Topsy: [View]

Posted in Evidence | 14 Comments »

To What Extent Do Multiple Copies of Papers Affect Download Statistics?

Posted by Brian Kelly on 14 Nov 2011

Are Multiple Copies of Papers Bad For The Researcher?

If authors are encouraged to provide pre-prints of their papers in addition to the paper which is hosted at the publisher’s Web site, how might that affect the associated usage statistics?  If usage statistics are fragmented, how easy might it be to aggregate the statistics? And if doing this is difficult, does it matter?

This was a question I was asked recently.  In order to try and gain a better understanding of what the issues were I have analysed the usage statistics for the five most downloaded papers which I have uploaded to Opus, the University of Bath institutional repository.   This exercise helped me to understand that the issues is more complicated than I initially appreciated.  The data for my papers is summarised below.


Paper 1
Library 2.0: balancing the risks and benefits to maximise the dividends
Journal/Event Program Electronic Library & Information Systems, 43, 2009
Opus statistics 1,516
UKOLN Web site statistics 190 consisting of 14 (.doc files viewed in 2011) +129 (.doc files viewed in 2009) + 47 HTML file viewed in 2009)
Publisher’s information [Paper] – Usage statistics not available
Nos. of citations 8 Citations according to Google scholar
Other known copies There are 210 records listed in Google Scholar search which includes links to versions on Opus and the UKOLN Web site
Notes Two versions of paper published: Initial paper presented at Building Bridges 2009 conference. Paper subsequently republished in Program.
Paper 2
From Web Accessibility to Web Adaptability
Journal/Event Disability and Rehability: Assistive Technology, 4, 2009
Opus statistics 491
UKOLN Web site statistics 0 views
Publisher’s information [Paper] – Usage statistics not available
Nos. of citations 6 citations according to Google scholar
Other known copies David Sloan’s list of publications (PDF file available). There are 10 records listed on Google Scholar search which includes links to versions on Opus and the UKOLN Web site.
Notes This paper was embargoed and so was not released until 18 months after publication.
Paper 3
Implementing a Holistic Approach to E-Learning Accessibility
Journal/Event ALT-C, 2005
Opus statistics 409
UKOLN Web site statistics 4,021 views consisting of 295 (HTML views in 2011) + 557 (HTML views in 2010) + 592 (HTML views in 2009) + 1,009 (HTML views in 2008) +861 (HTML views in 2007) + 707 (HTML views in 2006) + 635 (HTML views in 2005)
Publisher’s information Not available on conference web site
Nos. of citations 20 citations according to Google scholar
Other known copies There are 8 records listed on Google Scholar search which includes links to versions on Opus and the UKOLN Web site together with a copy of the MS Word file hosted by MediaLT organisation in Norway.
Notes This paper was awarded the prize for Best Research Paper at the ALT-C 2005 conference.
Paper 4
Developing A Holistic Approach For E-Learning Accessibility
Journal/Event Canadian Journal of Learning and Technology, 30 (3), 2004
Opus statistics 404
UKOLN Web site statistics 498 views consisting of 188 (HTML views in 2011) + 310 (HTML views in 2010)
Publisher’s information [Publisher’s copy] – Usage statistics not available
Nos. of citations 36 citations according to Google scholar
Other known copies There are 6 records listed on a Google Scholar search which includes links to metadata records on Opus and the UKOLN Web site.
Notes This paper was available on the UKOLN Web site for a significant period of time.
Paper 5
Empowering users and their institutions: A risks and opportunities framework for exploiting the potential of the social web
Journal/Event CULTURAL HERITAGE online conference web site, 2009
Opus statistics 356
UKOLN Web site statistics 0 views
Publisher’s information [Publisher’s copy] – Usage statistics not available
Nos. of citations 1 citation according to Google scholar
Other known copies There are 3 records listed on a Google Scholar search (which has one link to a copy on the UKOLN Web site) and 12 on a second Google Scholar search which includes links copies on the conference Web site.
Notes This paper was not made available on the UKOLN Web site. The publisher’s copy consists of two large PDF file of all papers presented at the conference.  Also note that this was a recent paper, by which time it had been decided to only publicise the copy on the institutional repository.

In total there have been 3,176 views of these five papers from the institutional repository and 4,709 views from the UKOLN Web site. Reviewing this evidence it seems that copies which were provided on the UKOLN Web site in 2004 and 2005 have had significant numbers of downloads from the Web site, in excess, significantly in one case, the numbers of downloads in the Opus repository.

It should also be noted that, as described in a blog post entitled Scridb Seems to be Successful in Enhancing Access to Papers papers hosted on the Scribd document sharing service do seem to attract a very large number of downloads, as shown below.

Discussion

If download statistics are used to complement citation statistics in order to provide some indication of the value of research publications it would appear that there will be pressures to either ensure that content is hosted only in a single location of that download statistics from multiple repositories can be aggregated.

However it does not seem clear how one might aggregate usage statistics from a diversity of services. I have been able to publish the statistics for files hosted on the UKOLN repository as I have access to the usage statistics, but this is clearly not a scalable solution.  SImilarly for the papers I have described I have not been able to find any statistics for the copy hosted on the publisher’s site.

One might then conclude that the recommendation should be that research papers should only be hosted in a single location. But is this a realistic approach?  I have always been keen on maximising access to my papers. Initially this was done by hosting the papers on the UKOLN Web site, before the University of Bath provided an institutional repository.  Although the papers are now hosted on the repository, and this is now the preferred location, I am reluctant to delete the original copy since this may cause long-established links to the paper to break and thus  cause access problems for users following such links. Similarly I would be reluctant to stop co-authors hosting a copy of the paper on their own repository. Indeed, since I seek to make use of Creative Commons licences to encourage reuse where possible it would seem to go against the grain to try to control such reuse in order simply to enhance metrics.

This, it seems to me, is the crux of the matter.  If the aim of research papers is to have an impact and open access can enhance this goal, then surely we need to accept the fragmentation of resources, including research publications. Looking at the metrics for the papers listed above it does seem that where a paper is available from multiple locations this enhances the numbers of downloads and subsequent citations although I would welcome a more rigourous analysis.

However such speculations are based on a very small sample and very subjective opinions. In addition the analysis of the usage statistics for the UKOLN Web site seems surprising, with figures displayed primarily for the HTML versions of papers and not the MS Word and PDF versions. This may be due to the usage statistics package not displaying findings for resources for which here have only been a small number of downloads.   However if this is the case it seems to suggest the advantages of providing a research paper in HTML format as well as MS Word and PDF.

But how typical are these findings, I wonder?  And what do people think about the tensions between maximising access to papers by setting them free and being able to better understand their usage by providing papers in a more managed environment?

Posted in Evidence | 3 Comments »

How People Find This Blog, Five Years On

Posted by Brian Kelly on 1 Nov 2011

Summary of Blog Usage

Today sees the fifth anniversary of the launch of the UK Web Focus blog which took place on 1 November 2006. A year after the launch I provided  a review of  The First Year Of The UK Web Focus Blog and on 1 November 2009 reviewed The Third Anniversary of the UK Web Focus Blog. A year later I published a post on the Fourth Anniversary of this Blog – Feedback Invited.

In those reflective posts I asked in 2008 whether “on reflecting on the various feedback I’ve received, it seems to me that I’ll need to give some thought to perhaps creating a new blog” – in the end, although I contributed to several project and event-focussed blogs, I published posts primarily on this blog. In 2009 I commented that  “with over 600 posts published on the UK Web Focus blog, I can’t recall all of the things I have written about!“. The following year I described h0w “the blog [is] my open notebook [used] to keep a record of activities I had been involved in and my observations and thoughts on developments“.

This year I’ll again provide a snapshot of the statistics for the blog.  There have been 988 posts published and 4,610 comments (which, I should add, includes referrer links). There have been 377,300+ views, with an average of 205 views per day over the five years. The busiest day was 14 January 2011 when there were 1,420 views following the publication of a post on Institutional Use of Twitter by Russell Group Universities.

The numbers of daily views peaked in 2009 with an average of 247 views per day. Last year there were 243 views per day and so far in 2011 there have been 230 views per day.  This slight decrease reflects the number of posts published, with 263 posts published in 2009, 200 in 2010 and 137 to date in 2011. In terms of the average numbers of views per post in 2009 there were 342 views per post,  443 in 2010 and 508 to date in 2011.

Analysis of Blog Referrer Traffic

In addition to these usage statistics I’d also like to analyse the Web sites which drove traffic to this blog. As can be seen from the accompanying image showing details of the referrer traffic (captured over a week ago) the Twitter Web site was the most significant driver of traffic (having provided 8,0291 views up to today – 28 October 2011) , sending more than twice as much traffic than Google Reader, which was in second place with 3,792 views.

After the UKOLN Web site (2,672 views ) there were then some further Web-based RSS readers (Netvibes and Bloglines with 2,356 and 2,300 views) followed by the Google search page (2,058 views) and then an individual’s blog provided by Stephen Downes which delivered 1,292 views followed by Facebook with 1,112 views and another aggregated collection of visits generated by Google search which delivered 1,006 views.  These were the only services which have delivered over 1,000 views. In total the top nine referring Web sites delivered 24,617 views.

This is, however, a very small proportion of the 376,700+ total number of views (~6.5%).  How else have people arrived at the blog if not by the Twitter Web site, Google, RSS readers and other popular blogs and Web sites?  The answer could be that there is a long tail of referring Web sites.

Unfortunately WordPress does not provide a total for referrer statistics. However after copying the data into Excel I find that there are 499 referring Web sites which deliver a total of 45,804 visits, with the last seven entries each delivering five visits.  I am assuming that there WordPress either displays a maximum 0f  500 entries or has a cut-off of five visits.  But based on the statistics which are available it seems that Web sites referrers only deliver ~12.2% of the traffic.  In order to understand how the missing 78% of the traffic arrived at the Web site I’ve looked at traffic for a particular post.

Looking  at the statistics for the recent post on Are University Web Sites in Decline? it seems that there were 297 views of the blog on the day the post was published, but there were only 102 referrers from Web sites. Looking at the bit.ly statistics for the post it seems that there were 36 clicks on this link on the day of publication and 32 on the following day, with only 11% of those views coming from the Twitter Web site. However over 50% of the views are still unaccounted for. Some of these will probably be from email subscribers of the blog; there are 95 subscribers who use the Feedburner email service with 32 subscribers viewing the post on 20 October. And the remainder?  I suspect they’ll be other Twitter users who have followed a URL provided by a link shortening services besides bit.ly.

Using another example, as described above the busiest day for this was 14 January 2011 when there were 1,420 views following the publication of a post on Institutional Use of Twitter by Russell Group Universities. Looking at the bit.ly statistics for this post we can see that there were 798 of the shortened link on the day the short link was published on Twitter.  The WordPress statistics for the post show that there were 1,088 views of the post on the blog on the day of publication.

Conclusions

The conclusion I have reached: most people now view posts on this blog following alerts they have come across on Twitter rather than via a Google search or by subscribing to the blog’s RSS feed.  Or, to put it more succinctly, social search is beating Google and RSS.

Is this really the case or have I misinterpreted the data? And if the data is accurate for my blog is this trend being replicated across other blogs?

I’d be very interested to hear from other blog authors on how traffic is arriving at their blog. Tony Hirst has kindly provided a screenshot of referring traffic to his OUseful blog for the past year which shows that Twitter Web site is also, by a significant margin, the most popular referrer site. Any other bloggers have findings they are willing to share?

Posted in Blog, Evidence | 14 Comments »

Are University Web Sites in Decline?

Posted by Brian Kelly on 20 Oct 2011

Are Web Sites In Decline?

Are organisational Web sites in decline? Earlier this year an article suggested that this was the case for an number of well-known companies, such as Coca Cola (“Coca Cola’s website traffic is down more than 40% in just 12 months“). The article cited a study by Webtrends published in March 2011 which revealed that static or declining website traffic is affecting the majority of Fortune 100 web sites, with 68% experiencing negative growth over the past 12 months with a 24% average decrease in unique visitors.

Are we seeing similar trends across University Web sites?

Analysis of Usage Trends for Russell Group Universities

A recent tweet from Martin Hawskey suggested that Google’s Double Click Ad Planner service could be useful in providing usage statistics for University Web sites. This tool has been used to provide a graph of estimated usage of the twenty Russell Group Universities for a period of slightly over a year, from March 2010 to August 2011. The findings are displayed in the following table.

Institution /
Double Click Stats
Graph  Additional Statistics
1 University of Birmingham
Stats for Birmingham
Unique visitors (estimated cookies)
390K
Unique visitors (users)
260K
Reach
0.0%
Page views
9.1M
Total visits
920K
Avg visits per cookie
2.4
Avg time on site
13:00
2 University of Bristol
Stats for Bristol
 
Unique visitors (estimated cookies)
160K
Unique visitors (users)
110K
Page views
1.1M
Total visits
270K
Avg visits per cookie
1.6
Avg time on site
6:00
3 University of CambridgeStats for Cambridge  
Unique visitors (estimated cookies)
1.3M
Unique visitors (users)
1M
Reach
0.1%
Page views
19M
Total visits
2.4M
Avg visits per cookie
1.8
Avg time on site
10:50
4 Cardiff University
Stats for Cardiff
 
Unique visitors (estimated cookies)
220K
Unique visitors (users)
150K
Page views
3.9M
Total visits
570K
Avg visits per cookie
2.6
Avg time on site
7:50
5 University of Edinburgh
Stats for Edinburgh
 
Unique visitors (estimated cookies)
680K
Unique visitors (users)
520K
Page views
13M
Total visits
1.4M
Avg visits per cookie
2.1
Avg time on site
10:50
6 University of Glasgow
Stats for Glasgow
 
Unique visitors (estimated cookies)
420K
Unique visitors (users)
300K
Page views
6.8M
Total visits
900K
Avg visits per cookie
2.1
Avg time on site
10:40
7 Imperial College
Stats for Imperial College
 
Unique visitors (estimated cookies)
200K
Unique visitors (users)>
140K
Page views
2.4M
Total visits
310K
Avg visits per cookie
1.6
Avg time on site
8:00
8 King’s College London
Stats for King’s College London
 
Unique visitors (estimated cookies)
1.3M
Unique visitors (users)
1M
Page views
19M
Total visits
2.4M
Avg visits per cookie
1.8
Avg time on site
10:50
9 University of Leeds
Stats for Leeds
 
Unique visitors (estimated cookies)
580K
Unique visitors (users)
390K
Page views
15M
Total visits
1.5M
Avg visits per cookie
2.5
Avg time on site
11:40
10 University of Liverpool
Stats for Liverpool
 
Unique visitors (estimated cookies)
350K
Unique visitors (users)
240K
Reach
0.0%
Page views
8.9M
Total visits
1.1M
Avg visits per cookie
3
Avg time on site
11:50
11 LSE
Stats for LSE
 
Unique visitors (estimated cookies)
470K
Unique visitors (users)
350K
Page views
5.6M
Total visits
860K
Avg visits per cookie
1.8
Avg time on site
9:40
12 University of Manchester
Stats for Manchester
 
Unique visitors (estimated cookies)
610K
Unique visitors (users)
430K
Page views
14M
Total visits
1.6M
Avg visits per cookie
2.6
Avg time on site
11:40
13 Newcastle University
Stats for Newcastle
 
Unique visitors (estimated cookies)
390K
Unique visitors (users)
260K
Page views
6.2M
Total visits
1M
Avg visits per cookie
2.6
Avg time on site
9:50
14 University of Nottingham
Stats for Nottingham
 
Unique visitors (estimated cookies)
470K
Unique visitors (users)
320K
Page views
13M
Total visits
1.4M
Avg visits per cookie
3.1
Avg time on site
13:00
15 University of Oxford
Stats for Oxford
 
Unique visitors (estimated cookies)
1.5M
Unique visitors (users)
1.1M
Page views
21M
Total visits
2.7M
Avg visits per cookie
1.9
Avg time on site
9:50
16 Queen’s University Belfast
Stats for Queen’s University Belfast
 
Unique visitors (estimated cookies)
220K
Unique visitors (users)
140K
Page views
6.7M
Total visits
770K
Avg visits per cookie
3.5
Avg time on site
13:00
17 University of Sheffield
Stats for Sheffield
 
Unique visitors (estimated cookies)
470K
Unique visitors (users)
320K
Reach
0.0%
Page views
7.4M
Total visits
1.1M
Avg visits per cookie
2.3
Avg time on site
8:00
18 University of Southampton
Stats for Southampton
 
Unique visitors (estimated cookies)
430K
Unique visitors (users)
290K
Reach
0.0%
Page views
6.6M
Total visits
910K
Avg visits per cookie
2.1
Avg time on site
8:50
19 University College London
Stats for University College London
 
Unique visitors (estimated cookies)
830K
Unique visitors (users)
560K
Page views
9.9M
Total visits
1.6M
Avg visits per cookie
1.9
Avg time on site
8:40
20 University of Warwick
Stats for Warwick
 
Unique visitors (estimated cookies)
430K
Unique visitors (users)
320K
Page views
6.8M
Total visits
980K
Avg visits per cookie
2.3
Avg time on site
7:50

It should be noted that, as described on an Ad Planner help pageinformation from a variety of sources including anonymized, aggregated Google Toolbar data from users who have opted in to enhanced features, publisher opt-in anonymous Google Analytics data, opt-in external consumer panel data, and other third-party market research“.

Using Google Trends To Make Comparisons

In order to see if the findings were reproducible using other tools the Google Trends service was also used. The findings are depicted below, with trends since late 2008 being shown in groups of five institutions.

Trends across Oxford, Cambridge, UCL, Edinburgh and Southampton

Trends across Birmingham, Bristol, Cardiff, Glasgow and Imperial College

Trends across KCL, Leeds, Liverpool, LSE and Manchester

Trends across Newcastle, Nottingham, Queen’s University Belfast, Sheffield and Warwick

It can be seen from these comparisons that similar trends are taking place across all twenty Russell Group Universities, with the possible exception of Warwick University, which did not see a drop in usage in 2009, although after this its usage patterns followed that of the other institutions.

It should be noted that the Google Trends site does give the warning that  “several approximations are used when computing these results” and gives the warning that “All traffic statistics are estimates“. The site goes on to add that “The data Trends produces may contain inaccuracies for a number of reasons, including data-sampling issues and a variety of approximations that are used to compute results” and gives the warning that “you probably wouldn’t want to write your Ph.D. dissertation based on the information provided by Trends“!  So perhaps it would be inappropriate to make policy decisions based on this data. But if no addition data is available, how else can be make evidence-based policy decisions?  And as described in a post on “University Web Sites Cost Money!” we know that the Daily Telegraph has a record of publishing an article entitled “Universities spending millions on websites which students rate as inadequate“ based on flawed interpretation of statistics gathered using Freedom of Information requests.  Unless and until universities are willing to openly publish Web site usage statistics we need to be prepared to accept that alternative metrics may well be used.

University adoption of social mediaSummary

Whilst the evidence is suggesting that we are seeing a slight decrease in the amount of traffic to institutional Web sites for Russell Group Universities, there is additional evidence which suggests that the same group of twenty UK Universities are seeing increased activity across the institutions’ Facebook sites.

As summarised in a recent post entitled Is It Time To Ditch Facebook, When There’s Half a Million Fans Across Russell Group Universities?   “in a period of nine months we have seen an increase in the number of ‘likes’ for the twenty UK Russell Group Universities of over 274,000 users or almost 100% with the largest increase, of over 155,000 occurring at the University of Oxford“. The post goes on to describe how are “seeing a huge increase in the number of Facebook ‘likes’ with all of the institutions seeing a growth of between 33% and 345%“.

The findings from the declining usage of institutional Web sites could be used to question the importance of those working in institutional Web teams. However the evidence from Facebook suggests that certain services initially provided on institutional Web sites seem to have migrated to popular social web services – and clearly there will be a need to manage the content and interactions with potential students wherever such interactions take place. For example a couple of day ago a post on Mashable described 7 Ways Universities Are Using Facebook as a Marketing Tool which included providing virtual tours; demonstrating pride in the institution; marketing ‘shwag‘; supporting alumni activities; sharing departmental; content; reaching out to potential students and exploiting geo-location services – all activities which will require institutional support.

The importance of social web across higher education has also been identified in an infograph which was launched in August 2011 in a post entitled “How colleges and universities have embraced social media” on the US-based Schools.com service (and embedded in this post).

This article suggests that the US higher education system seemed initially reluctant to embrace social media:

Universities are often at the forefront of intellectual thought, but they have been known to lag behind the rest of society when it comes to learning and adopting new technologies. Such has certainly been the case with social media technologies. In fact, so reluctant were universities to adopt social media on campus that in 2007, only about half of colleges reported social media usage.

but have recently recognised the benefits which can be gained:

According to a recent report from the University of Massachusetts, however, colleges have finally caught on; in 2011, 100% of universities are using at least one form of social media–and they are reporting that it’s now an important and successful piece of their outreach efforts. Check out the below infographic to learn more about how colleges have been slowly going social.

The Mashable blog is in agreement with these views of the current importance of social media to US Universities. A post entitled 6 Best Practices for Universities Embracing Social Media suggests that:

For universities, deciding to use social media is a no-brainer. The 18- to 24-year-old college student demographic is all over the social web, and its younger counterpart (the high school crowd) is equally immersed.

and goes on to describe how:

Already, many schools have leveraged social media in a big way. In fact, a recent study showed that an astounding 100% of universities have a social media presence. From luring in potential new students with admissions blogs and creative use of location-based services like SCVNGR, to keeping alumni engaged via dynamic, content-rich Facebook and Ning communities, to informing students about campus offerings through Twitter feeds and YouTube videos, it’s clear that universities recognize the importance of social media.

But in addition to the popularity of Social Web sites, another possible reason for the lack of growth in usage of institutional Web site may be a consequence of the difficulties in navigating such sites on mobile devices. In the US a Read/Write Web article informs use that “7% of U.S. [is] Web Traffic From Handheld Devices“. How many institutional Web sites provide easy-to-use interfaces on mobile devices, I wonder?

Implications

There is a danger that the evidence of decline in traffic to institutional Web sites could be used to justify cuts in levels of funding for institutional Web teams.  However additional evidence suggests that users may be simply making use of alternative sources of information and interactions or may be using mobile devices which may provide cumbersome experiences when accessing sites which have not been configured to provide optimal interfaces when using small screens, no mouse interface and other characteristics of mobile devices.

I think it would therefore be a mistake to argue that there is a decrease in interest in or relevance of online services which may initially have been provided on institutional Web sites. Rather I feel we are seeing a move towards a variety of cloud-based services.  The high-profile services may include Facebook together with social media sharing services such as YouTube and iTunes (for which usage across Russell Group universities has been documented in posts on How is the UK HE Sector Using YouTube? and What are UK Universities doing with iTunes U?). But in addition we are also seeing policy and funding decisions being made by funding bodies such as HEFCE which will see a move towards cloud-based services which will be more closely-aligned with the requirements of the UK’s higher education sector, with the migration of the Jorum service from a project to a service role providing a good example of how key online services traditional hosted within the institutional may be more cost-effective if hosted externally but developed with the needs on institutions in mind.

How should the evidence, such as the examples I’ve listed in this post, be used to inform institutional policies, I wonder? And might there be a need to make changes to existing Web team structures, if responsibilities for managing institutional Web sites are separate from managing content and interactions hosted outside the institution?

Posted in Evidence, IWMC | 15 Comments »

When Trends Can Mislead: The Rise, Fall and Rise of Apache

Posted by Brian Kelly on 11 Oct 2011

Back in April 2008 in a post entitled “The Rise and Fall of Apache?” I contrasted the fall in the numbers of Web servers running on the Apache Web server software with the corresponding rise in use of the Microsoft Web server software, as illustrated below.

But how have things changed over the past three years? A recent email alert from Netcraft has provided an answer. As can be seen from the October 2011 Web Server Survey (illustrated below) since 2009 there has been a steady decline in usage of Microsoft server software and a corresponding increase in use of Apache.

 

One lesson from this is that trends won’t always be an accurate predictor of future developments. But in addition, when I published the initial post Mike Nolan, Richard Cunningham and others suggested that the overall figures of Web server usage were not necessarily accurate. Rather it would be more appropriate to show the trends for active Web server usage.

Those comments, all of which were made on the day the post was published, were valuable in informing me of flaws in my interpretation of the data. The timeliness of the responses was also helpful in minimising dangers that others may have read the post and be unaware of the flaws in the interpretation of the data.  I think that illustrates the value of providing commentable articles and in minimising barriers for commenting (note there are no approval processes in place which could delay publication of comments).

So now we should be able to say with some confidence that the Apache server is well-established as the leading tools for providing Web sites around the world.  I suspect that this will also be true across the UK higher education sector. And although we sometime talk of the value of platform-independent solutions, there are times when it may be legitimate to develop solutions for particular platforms.  I am particularly interested in ways in which institutions may be able to implement recommendations provided by the Linked You Toolkit developed at the University of Lincoln.

One the the recommendations was that:

attention needs to be given to the way institutions transition to a shared ontology for the sector. Research needs to be done that examines and recommends strategies for migrating from existing and legacy URI structures to a model of best practice. HTTP 3xx status codes are at the heart of this.

Might appropriate strategies for development of shared approaches be based on developments for the Apache server, which it seems, is likely to be widely deployed across the sector?

Posted in Evidence | 1 Comment »

Is It Time To Ditch Facebook, When There’s Half a Million Fans Across Russell Group Universities?

Posted by Brian Kelly on 26 Sep 2011

Implication of Changes To Facebook

The changes to Facebook announced at Facebook’s F8 Developers conference last week haven’t gone down well in some circles with a number of the people I follow on Twitter expressing their concerns at the privacy implications of recent changes and one or two having gone as far as to delete their Facebook accounts.

Might those technically-savvy people be setting a trend which will become more widespread as the privacy concerns become more widely known beyond those who read blog posts which describe in detail how Facebook can monitor your interactions, even when you are logged out of the service? Or are these people in a minority and will we see that once the changes have been fully deployed and problems fixed in light of user feedback could be see an increase in Facebook usage?

Gathering Evidence of Institutional Use of Facebook

In order to be able to gather evidence of possible changes in usage patterns within the UK HE sector I have updated a survey of Use of Facebook by Russell Group Universities which was carried out in January 2011. A summary of the numbers of people who have ‘liked’ the pages, together with details of the changes from the previous survey are given in the following table.

Institution and Web site link
Facebook name and link
Nos. of Likes
(Jan 2011)
Nos. of Likes
(Sep 2011)
Percentage
increase
1 InstitutionUniversity of Birmingham
Fb nameunibirmingham
8,558 14,182 66%
2 InstitutionUniversity of Bristol
Fb nameUniversity-of-Bristol/108242009204639
2,186 7,913  262%
3 InstitutionUniversity of Cambridge
Fb namecambridge.university
58,392 105,645 81%
4 InstitutionCardiff University
Fb namecardiffuni
20,035 25,945 29%
5 InstitutionUniversity of Edinburgh
Fb name: University of Edinburgh/108598582497363
(None found in first survey)
12,053
6 InstitutionUniversity of Glasgow
Fb Name: glasgowuniversity
(None found in first survey)
1,860
7 InstitutionImperial College
Fb nameimperialcollegelondon
5,490 10,257  87%
8 InstitutionKing’s College London
Fb nameKings-College-London/54237866946
2,047 3,587 75%
9 InstitutionUniversity of Leeds
Fb name: universityofleeds
(None found in first survey)
899
10 InstitutionUniversity of Liverpool
Fb nameUniversity-of-Liverpool/293602011521
2,811 3,742 33%
11 InstitutionLSE
Fb nameLSE/6127898346
22,798 32,290 42%
12 InstitutionUniversity of Manchester
Fb nameUniversity-Of-Manchester/365078871967
1,978 4,734 139%
13 InstitutionNewcastle University
Fb name: newcastleuniversity
115
14 InstitutionUniversity of Nottingham
Fb nameThe-University-of-Nottingham/130981200144
TheUniofNottingham
3,588 3,854 9,991 7% 178%
15 InstitutionUniversity of Oxford
Fb namethe.university.of.oxford
137,395 293,010  113%
16 InstitutionQueen’s University Belfast
Fb nameQueens-University-Belfast/108518389172588
5,211
17 InstitutionUniversity of Sheffield
Fb nametheuniversityofsheffield
6,646 12,412  87%
18 InstitutionUniversity of Southampton
Fb nameunisouthampton
3,328 6,387  92%
19 InstitutionUniversity College London
Fb nameUCLOfficial
977 4,346 345%
20 InstitutionUniversity of Warwick
Fb namewarwickuniversity
8,535 12,112 42%
TOTAL 286,169
287,767 
560,554
566,691 
96%
97%

Summary

In brief in a period of nine months we have seen an increase in the number of ‘likes’ for the twenty UK Russell Group Universities of over 274,000 users or almost 100% with the largest increase, of over 155,000 occurring at the University of Oxford.

Discussion

The previous survey highlighted emerging patterns of institutional use of Facebook and provided some suggestions on best practices (such as providing a Facebook page rather than a group and having a short and branded URL).  It seems that institutions are implementing such best practices more widely.  We are also seeing a huge increase in the number of Facebook ‘likes’ with apart from Nottingham’s 7% increase, all of the other institutions seeing a growth of between 33% and 345%.

But might this represent a peak for institutional use of Facebook?   Since we have over half a million users, many of whom will be staff or students at Russell Group Universities we might expect this particular demographic to have a better understanding of the dangers of misuse of Facebook than the general public.  It will be interesting to see how these figures change over the next academic year.

Beyond the Evidence of Usage – Is Facebook a Walled Garden?

This post has focussed on institutional use of Facebook to provide services to end users (a business-to-consumer relationship).  Of course there are privacy implications associated with use of Facebook and it might be argued that Universities shouldn’t be using unethical network providers – just as there were pressures on universities not to support businesses which had links with South Africa during the apartheid era.

I’ve not heard people seriously suggesting that Universities should stop their institutional use of Facebook, but there is a need to have a better understanding of the concerns people have regarding Facebook, in part so that we can ensure that possible alternatives to Facebook don’t repeat such concerns. The one particular areas of concerns I’d like to address in this post is that Facebook is a ‘walled garden.’

This morning I was involved in a brief Twitter discussion in which Twitter was dismissed as a ‘walled garden’. It was suggested that, just like AOL, you need to sign up to access content hosted on Facebook. Surely not? So I logged out of Facebook and visited the University of Warwick page and, as can be seen, I can view the page.

But rather than restrictions on accessing public information, perhaps Facebook is described as a walled garden because you can put information in, but not get it out again?

This was the case at one point, but know there is a Facebook Export service which “uses the Facebook Open Graph protocol to export your Facebook data to an xml file. Facebook Export does not store any data about you. You can then use this xml file to import your data to other services and websites that support the Facebook Export (FBE) format.

Or perhaps the concern is that use of Facebook apps locks information into a particular application? I feel there may be an element of truth to this concern – you can develop Facebook apps which do trap the data into the app.  But the Russell Group University Facebook pages seem to be using the default Facebook features, so this isn’t really a current concern. And even apps such as the Guardian Facebook app shouldn’t be regarded as acting as a walled garden since the same data can be accessed in several other ways, such as via RSS feeds, Android and iPhone apps and on the Web itself.

I, therefore, am unconvinced that current institutional use of Facebook can be regarded as using a Walled Garden and that Universities are promoting a propriety service.  Of much greater relevance will be how people react to the recent changes in Facebook. If people start to leave, there will be a need to reconsider Universities’ uses of Facebook as a marketing and engagement service.

Posted in Evidence, Facebook | 32 Comments »

Is It Now Time to Embed Use of Google+?

Posted by Brian Kelly on 21 Sep 2011

Is Google+ Dead?

Is Google+ dead?  Dan Reimold certainly thinks so. In a post entitled “Google+: Social Media Upstart ‘Worse Than a Ghost Town‘” he suggest that Google+ may “simply [be ] a social media step too far” and is now “worse than a ghost town“.  In his conclusions he reflects on his personal experiences as a Google+ user:

As it stands, my Circles are sparse. The stream of updates has basically run dry — reduced to one buddy who regularly writes. My initial excitement about signing on and inviting people to join me has waned. Nowadays, I apparently get tired just thinking about it. 

A similar discussion about the relevance – and perhaps sustainability – took place amongst some of my Twitter followers recently. It seems that some feel Google+ is irrelevant and others are pleased with what they claim is a failed Google service and are waiting for the Diaspora service to be launched. However, as I said in the Twitter discussion, I am not convinced by this argument.

Why Google+ May be a Slow-Burner

Lessons from Growth of Twitter

In January 2011 in a post on Evidence of Personal Usage Of Social Web Services I described how use of the Tweetstats service provided me with evidence of growth of my Twitter usage which contradicted the understanding I had at the time. I had thought that I was an early adopter of Twitter and had used if fairly consistently since my first tweet in January 2007. But the Tweetstats graph (illustrated) shows little use in 2007. It wasn’t until early 2008 that I started to use Twitter on a regular basis.  The gaps in graph in the early part of 12008 puzzled me initially until I came across a blog post in which I described how I had made intensive use of Twitter whilst attending the Museums and the Web 2008 conference.  It seems that, perhaps due to a glitch in Twitter or Tweetstats, no usage had been detected for a period of a couple of months, which included the time when I first start to use Twitter on a regular basis.

Looking back it seems that attending a conference abroad made me aware of the benefits which Twitter can provide during a conference and that I soon became aware of the additional benefits which can be gained by developing links with one’s professional network.

A few days ago Aaron Tay pointed out that:

Some technology rewards getting in early e.g Twitter (early accs get more followers) & some don’t e.g qrcode http://j.mp/mUncdt

The post he cited (on the Seth Godin blog) made the observation that:

Worth considering: The difference between a technology where getting in early pays dividends, and those that don’t. For example, having a website or a blog or a Twitter account early can help, because each day you add new users and fans.

QR codes, on the other hand, don’t reward those that get in the ground floor. You can always start tomorrow.

Seth pointed out a important advantage that early adopters of social networks can have – the ease of gaining the critical mass which may be needed in order for the service to provide value.  There is a danger that this may be construed as a suggestion that the numbers of followers alone is a key factor in having an effective social networking service – and seeking new followers simply to enhance one’s Klout or Peerindex ranking is an example of misunderstanding of the relevance of a critical mass. Rather than simply indiscriminately seeking to grow large numbers of followers it you are looking to use a social network for professional purposes there is a need for to be reach the critical mass across one’s peers.

I recently installed the Social Bros application which provides evidence of personal use of Twitter.  I used this recently to investigate the number of followers the people I follow on Twitter have.  As can be seem most of the people I follow have 100-500 followers, with significant numbers having 1,000-5,000 and 500-1,000 followers. In order to develop a community of this size it can be useful to be an early adopter so that one can stake a claim. The following influx of users will have to search for contacts, and, having spotted and made contact with you, you will be able to reciprocate, if  you so choose.

As described in a Wikipedia entry on the Network Effectsites like Twitter and Facebook [become] more useful the more users join“. But as well as users needing a critical mass and an understanding of the benefits of the service, there will also be a need for east-to-use tools. Initially I used the Twitter Web site but as I discovered from reading my early posts about Twitter,  I was using the Twhirl client around the time my Twitter use became embedded in my daily work routine.  The Tweetstats service I mentioned earlier also provides me with statistics on the Twitter clients I have used.  As can be seen Tweetdeck is now my preferred tool, with the usage statistics of the Web client primarily either reflecting, I suspect, my early use of the Web or use in Internet cafes.

Implications for My Use of Google+

What lessons might we learn from these reflections on how Twitter developed from claiming an id but making little use to finding valuable (and unexpected)  use cases which lead to the service being embedded in my professional life which can be applied to Google+?

Like, I suspect, many others of my peers I have claimed a Google+ account and have established contacts with people I know from both real world and online interactions (there are currently 116 people in my circles and 385 people who have included me in their circle).

Yesterday I found that Google+ accounts are now freely available to everyone, so the comment I have heard that Google+ is exclusive to the early adopters is not longer the case.

I also heard yesterday that Google+ have released APIs which should help in developing a richer environment of tools and services based around Google+ (in this case, use of Huddle) which, I feel, was valuable in Twitter becoming mainstream.

The Google+ service itself is becoming richer in functionality, with recent tweets from Aaron Tay alerting me to articles which describe how “Google+ Hangouts Go Mobile & Get More Collaborative” and explain “Why Google Plus Hangouts is the Killer App: Docs“.

It seems to me that it is now timely to explore ways in which Google+ may deliver benefits and also to gain an understanding of best practices including personal work flow processes.  Earlier this year I set up a daily blog which I used to keep notes and ideas.  I spotted using it after six months, partly because I felt I was getting little new from using a second WordPress blog.  However I’ve now made a decision to use Google+ as a middle ground between the (sometimes, as in this case, long) posts I publish on this blog and the conversations and  announcements which take place on Twitter.

Anyone else planning to make greater use of Google+? Or, like Dan Reimold, do you feel it’s a ghost town and is unlike to have a significant role to play?

Posted in Evidence, Social Networking | 9 Comments »

Bath is the University of the Year! But What if Online Metrics Were Included?

Posted by Brian Kelly on 14 Sep 2011

University of the Year

For the first time in a long, long time last weekend I bought the Sunday Times.  The reason for this was to read the Sunday Time’s announcement that the University of Bath has been identified as the University of the Year.

As someone who has worked and lived in Bath for almost 15 years I was very pleased with the news – but not as pleased, I suspect, as the Vice-Chancellor and members of the University’s Press Office which, of course, published a University news item with details of the announcement which informed us that:

The University of Bath has been awarded the title of ‘University of the Year 2011/12’ by The Sunday Times, one of the most prominent and influential newspapers in the world.

The news item went on to highlight another metric:

In that league table the University of Bath has risen to 5th out of 122 UK universities and colleges – its highest ever position.

Last Friday I viewed a video clip in which the Vice-Chancellor announced the news and as I left campus on Friday evening I noticed the posters which were scattered around the University Parade informing potential students (and their parents) which would be visiting the campus on the following day for the University Open Day  of what a great University they are visiting.

Being identified as the top University by (ahem) “one of the most prominent and influential newspapers in the world” is clearly deemed important by the powers that be at the University. And, in addition, several people I follow on Twitter who don’t work in marketing positions also tweeted the news.

What If Online Metrics Also Counted?

Yesterday Sheila MacNeill, Assistant Director at JISC CETIS, alerted me to a Mashable article which asked “How Digitally Connected Are the U.S. News Top 20 Colleges?”. The article referred to a  U.S. News list of top ranking national universities and national liberal arts colleges which appears similar to the Sunday Times survey. The Mashable article described how they:

decided to add another factor for review: social media connectedness. Below you’ll find top 10 lists of universities and liberal arts colleges alongside an analysis of their social media presences

This puts Harvard in equal first place with 66,737 Twitter followers, 698,933 Facebook likes and 390 YouTube videos and 27,786 subscribers. Harvard tied with Princeton University which had 15,572 Twitter followers, 52,125 Facebook likes and 164 YouTube videos and 2,978 subscribers. The positions in this league table seem to have been based on an undocumented weighting of the social media metrics.

Lies, Damned Lies and Social Media (and Other) Analytics

It is easy to dismiss the Mashable article as trivia, statistically flawed or dangerous, depending on your particular take. But can’t the same criticisms be made of the Sunday Times league tables?

Since the Sunday Times article is hidden behind a pay wall (and I’d left my copy at home) I subscribed to the Times / Sunday Times service in order to read about the methodology they had employed (note to self, cancel the Direct Debit payment before the full payment is due!).

The methodology (which is summarised here) states:

Universities were ranked according to marks scored in nine key performance areas.

Teaching excellence (250 points): The results of questions 1 to 12 of the 2011 national student survey (NSS) are scored taking a theoretical minimum and maximum score of 50% and 90% respectively. …

Student satisfaction (+50 to =55 points): The responses given to Question 22 of the National Student Survey: “Overall, I am satisfied with the quality of the course” were compared to a benchmark for the given institution, devised according to a formula based on the social and subject mix. …

Peer assessment (100 points): Academics across all institutions included in our guide were asked to rate departments in their subject field on a five-point scale for the quality of their undergraduate provision and a figure was awarded to each institution based on coverting (em>sic</em>) the average score for each institution on to a 100-point scale. …

Research quality (200 points): We used data from the most recent research assessment exercise, published in December 2008. Five different ratings were awarded for research quality, ranging from 4* to unclassified, from which we calculated an average score per member of staff entered for assessment. This average score was converted to a percentage and double weighted to give a score out of 200.  …

A-level/Higher points (250 points): Nationally audited data for the 2009-10 academic year were used for league table calculations. All entry points gained under the Ucas tariff system were used to calculate mean scores for all universities. Grades for leading qualifications were awarded points according to the following scale: A-levels – A*: 140; A:120, B:100, C:80, D:60 and E:40; AS-levels – A:60, B:50, C:40, D:30, E:20; Advanced Highers – A:120, B:100, C:80; Highers – A:72, B:60, C:48.  …

Unemployment (200 points): The number of students assumed to be unemployed six months after graduation was calculated as a percentage of the total number of known destinations. This is shown as a percentage in each profile. For the league table calculation, the percentage was subtracted from 50. …

Firsts/2:1s awarded (100): We calculated the percentage of students who graduated with firsts or 2:1 degrees.  …

Dropout rate (+57 to -74 points): The number of students who drop out before completing their courses was compared with the number expected to do so (the benchmark figure shown in brackets in the university profiles). Benchmarks vary according to subject mix and students’ entry qualifications. The percentage difference between the projected dropout rate and the benchmark was multiplied by five and awarded as a bonus/penalty mark. Universities that lost fewer students than their benchmark gained, those losing more had points deducted. …

Hmm. Are the ways in which the individual scores are compiled and then the scores for the nine categories aggregated significantly different from the way in which social media analytic companies such as Klout and Peerindex determine their scores (and which I summarised in a post on Social Analytics for Russell Group University Twitter Accounts)?

Doesn’t it seem likely that we will see the Sunday Times survey of UK universities in future years include analyses of universities’ online presence?

And won’t this be treated as important by those involved in University marketing and student recruitment, despite the limitations such methodologies may have?

Posted in Evidence | 9 Comments »

Recognising, Appreciating, Measuring and Evaluating the Impact of Open Science

Posted by Brian Kelly on 6 Sep 2011

The #SOLO11 Conference

As I mentioned in yesterday’s post on Use of Twitter at the SOLO11 Conference on Friday and Saturday, 2-3 September 2011 I attended the Science Online London 2011 event, SOLO11.

We are now starting to see various posts on the event being published. One of the first reports on the events was written by Alexander Gerber and published on the Scienceblogs service based in Germany. Alexander began his brief post by saying:

My sobering conclusion after two days of ScienceOnline London: The technologies are ready for take-off, the early-adopter-scientists are eager to kickstart the engine, but the runway to widespread usage of interactive technologies in science is still blocked by the debris of the traditional academic system. This system needs to be adapted to the new media paradigms, before web 2.0 / 3.0 can have a significant impact on both research and outreach. 

and went on to list three central questions which he feels need to be answered:

  • How can we recognise, appreciate, measure and evaluate the impact of outreach and open science in funding and evaluation practice?
  • Which new forms of citation need to be installed for that?
  • How can we create a reward system that goes way beyond peer-reviewed citations?

I’d like to address certain aspects of the first question, in particular ways in which one might measure and evaluate the use of social media to support such outreach activities since this issue was discussed during a workshop session on Online Communication Tools which I spoke at.  However I would first like to give some thoughts on the opening plenary talk at the event.

Plenary Talk on Open Science

For me the highlight of SOLO11 was the opening plenary talk on “Open Science” which was given by Michael Nielsen, a “writer; open scientist; geek; quantum physicist; writing a book about networked science“.

A number of blog posts about the event have already been listed in the Science Online wiki. I found Ian Mulvany’s thoughts on the Science Online London Keynote talk particularly helpful in reminding me of the key aspects of the talk.

Michael told the audience that he didn’t intend to repeat the potential benefits of open science; rather he would look at some examples of failures in open science approaches and then look in other disciplines to see if there were parallels and strategies which could be used in the science domain.

The example given described use of open notebook science in which a readership of ~100 readers in a highly technical area had been established, but there was little active participation from others.  The author, Tobias J Osbourne, was putting in a significant amount of effort but was failing to gain value from this work.

Michael gave an example of how a significant change can be made in a short period of time which brought significant benefits: the change to driving on the right hand side of the road in Sweden at  5am on Sunday, 3 September 1967.

However although this example was successful and brought benefits (such as reduced costs) there are many other examples in which the potential benefits of  Collective Action fail to deliver, often due to some potential beneficiaries chosen to ‘freeload’ on the work of others.

We can learn from examples of successes in other areas, ranging from the establishment of trade unions and well-established practices for managing water supply in villages through to the growth of the ArXiv archive and of the Facebook social networking service.  Successful approaches include:

Starting small: For example the ArXiV service success was due to it focussing on a small subject area. Similarly Facebook was initially available only to students at Harvard University, before expanding to, initially, other Ivy Leagues and then other higher educational institutions before being available to everyone.

Monitoring and sanctions: Michael concluded by describing how there was a need to monitor use and, if needed, to be able to apply sanctions.

The concept is that there is some action where if everyone changed it would be better for everyone, but you need everyone to change at the same time. There are incentives for people not to participate because there is some cost involved in changing for the individual but if the individual does not change, they get the benefit anyway from everyone else changing. This is the same kind of problem that we have with the move to open data.

In brief, therefore, Michael felt that those who feel that open science can provide benefits tend to be too ambitious – there is a need to start with small achievable aims and to make use of approaches for broadening the scope using various approaches which have proven successful in other areas.

Analytics for Use of Social Media

The second day of the SOLO 11 event provided a series of workshop sessions.  I attended one which was billed as Scholarly HTML but it fact provided an introduction to blogging on WordPress :-(  However a workshop session on Online Communication Tools which provided an introduction to Twitter, Google+ , etc in the morning moved on in the afternoon sessions to:

… cover all angles from how to practically use the tools most beneficially in an institutional or academic environment, to how to measure their impact via statistics and online “kudos” tools

Alan Cann, one of the facilitators of the session, invited me to speak in this session as Alan had attended a one-day workshop on “Metrics and Social Web Services: Quantitative Evidence for their Use and Impact” which I organised recently. I used the slides from a talk on “Surveying Our Landscape From Top to Bottom” which reviewed various analyses of use of social media services by individuals and institutions, including tool such as Klout, PeerIndex and Twitalyser.

Alan Cann also spoke in the session and in his presentation pointed out the statistical limitations in using such services – similar concerns to those made by Tony Hirst in a talk on which he gave at the  “Metrics and Social Web Services: Quantitative Evidence for their Use and Impact” event.

Tony’s slides, which are available on Slideshare, illustrated dangers of misuse of statistics including the accompanying graphs  showing data which can all be, incorrectly, reduced to the same linear curve.

Tony went on to describe Goodhart’s Law which states that:

once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play such a role.

and Campbell’s Law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Lies, Damned Lies and Social Media Analytics?

Might, therefore, we conclude that social media analytics tools such as Klout, PeerIndex and Twitalyzer have no role to play in, for example, “measuring and evaluating the impact of outreach and open science“? Not only are, for example, the ways in which Peerindex aggregates its scores for authority, activity and audience to give a single value statistically flawed, but, if such services are used for decisions-making purposes we will see users gaming the system.

Whilst this is true, I also feel that there are dangers in trying to develop a perfect way of measuring such impact – and it was clear from the workshop that this is an acceptance of the need for such measurements.

There will be many other examples of approaches to measurements which we generally accept but which have underlying flaws. The university system, for example, may be regarded as evaluating its successful consumers as first, two-one, two-two or third class degree students.  But despite the limitations of assessment the importance of such assessment is accepted.

We might also wish to consider how such measuring schemes are used.  The approaches taken by Klout and Peerindex have parallels with Google’s ranking algorithms – and again can be gamed. But organisations are prepared to invest in ways of  gaining high Google rankings since this will provide business benefits, through Web sites being more easily found in Google searches.

We are starting to hear of examples of Klout and Peerindex statistics being used  in recruitment, with a recent article published in the New York Times inviting readers to:

IMAGINE a world in which we are assigned a number that indicates how influential we are. This number would help determine whether you receive a job, a hotel-room upgrade or free samples at the supermarket. If your influence score is low, you don’t get the promotion, the suite or the complimentary cookies.

I suspect that marketing departments will use such statistics and that people working in marketing and outreach activities will start to use personal social media analytic scores in their CVs. Note that as can be seen from the image which shows my Peerindex scores such tools can be used in a variety of ways – it is clear that you wouldn’t employ me based on the diagram if you were looking for someone who had demonstrable experience in outreach work using Twitter in the field of medicine (my areas tend to focus on technology, sport and politics).

I therefore feel that we should treat social media analytics with care and use them in conjunction with qualitative evidence of value. But to disregard such tools completely whilst waiting for the perfect solution to appear will fall into the trap which Michael Nielsen warned against, of seeking to gain broad acceptance of a universally applicable solution.

I’d welcome your thoughts.

Posted in Evidence, Impact | 2 Comments »