UK Web Focus

Innovation and best practices for the Web

Archive for July 18th, 2011

Event Report: Metrics and Social Web Services Workshop

Posted by Kirsty Pitkin on 18 July 2011

In this guest post, event amplifier Kirsty Pitkin reports on the key messages from the recent UKOLNeim workshop – Metrics and Social Web Services: Quantitative Evidence for their Use and Impact.


Introduction

In introducing the event, Brian Kelly emphasised that the aims were to explore ways of gathering evidence that can demonstrate the impact of services and to devise appropriate metrics to support the needs of the higher and further eduction sector.

Many people argue that you cannot reduce education to mere numbers, as it is really about the quality of the experience. However, Kelly argued that numbers do matter, citing the recent JISC-funded Impact Report, which found that the public and the media are influenced by metrics. As we have to engage with this wider community, metrics are going to become more relevant.

View the introduction in full on Vimeo

The slides to accompany this talk are available on Slideshare

Why Impact, ROI and Marketing are No Longer Dirty Words

Amber Thomas, JISC

Amber ThomasThomas mapped out the current landscape, drawing on her own experiences and those of colleagues working in other areas at JISC. She observed a dominant culture of resistance to measurement within education for a number of reasons, including the concern that caring about metrics will mean that only highly cited people or resources will be valued. She noted that the search for an effective impact model is taking place on shifting sands, as issues associated with the value, ownership and control of media channels are being contested, as is the fundamental role of the university within British society.

In discussing impact, Thomas noted that it would be tempting to use the language of markets – with education as a “product” – but stressed that this not how we see ourselves in the education sector. One of the challenges we face is how to represent the accepted narrative of the sector as a nurturer and broker of knowledge, through the use of metrics.

Thomas went on to describe some of the dirty words in this space and the measurements that are associated with them. However, she noted that these measurements can be used for good, as they can help to instigate change. To support this, she provided a model for the role of metrics in decision making, with metrics being one form of evidence, and evidence being only one form of influence on the decision maker.

She concluded by outlining our options for responding to the impact debate: we could deny the impact agenda is important, or we could deepen our understanding and improve our metrics so they work for us and are fit for purpose. The possible directions we could take include developing business intelligence approaches, improving data visualisation techniques and looking for better tools to give us deeper understanding of the metrics. She also stressed that we need to look more closely at the use and expectations of social media in the commercial sector, as we might find we are expecting too much of ourselves.

“I don’t think we can ignore the debate on impact and metrics… what we need to do is engage with the impact debate and use the sort of language that is expected of us to defend the values of the sector a we wish to defend them.”

View the presentation in full at Vimeo

The slides to accompany this talk are available on Slideshare

Surveying our Landscape from Top to Bottom

Brian Kelly, UKOLN

Brian KellyKelly provided an overview of the surveys he has been carrying out using a variety of analytics tools.

He began with a personal view: discussing the picture of his own Twitter usage provided by the Tweetstats tool, and how this differs from his own memory. He noted that the data did not always correspond with other evidence, emphasising that we cannot always trust the data associated with such tools.

“You need to be a bit skeptical when looking at this data… you can’t always trust all the data that you have.”

From an institutional perspective, he asked: “What can commercial analytics tools tell us about institutional use of Twitter?” He compared the Klout scores of Oxford and Cambridge Universities’ Twitter accounts, showing how visualisations of the numbers can give a much better understanding of what those numbers really mean than the numbers themselves do in isolation.

He continued in this vein by demonstrating Peer Index, which he used to analyse participants of the workshop. He noted that the top seven people are all people he knows and has had a drink with, so asked whether this shows that the gathering is really a self-referential circle? Kelly also noted how easy it can be to gain extra points and questioned whether it is ethical to boost your score in this way. However, he observed that research funding is determined by flawed metrics, and gaming the system is nothing new. So will universities head hunt researchers with valuable social media scores?

Next he looked at Slideshare statistics, using a presentation by Steve Wheeler as a case study. Wheeler made a presentation to 15 people, but his slides were viewed by over 15,000 people on Slideshare. Kelly asked us to consider the relationship between the number of views and the value of this resource. He also examined statistics from the collection of IWMW slides, observing that the commercial speakers had higher view rates, and that the most popular slides were not in corporate look and feel. This evidence could be used to challenge standard marketing perspectives.

Finally, Kelly compared Technorati and Wikio results to demonstrate that four people in the room were in the top 67 English language technology blogs. He pondered whether they should they share their success strategies, or how we could tell the story of this data in different ways.

To conclude, Brian emphasised that he believes this kind of analysis can inform decision making, so it is important to gather the data. However, the data can be flawed, so it is important to question it thoroughly.

View the presentation in full on Vimeo

The slides to accompany this talk are available on Slideshare

Learning From Institutional Approaches

Ranjit Sidhu, SiD

Ranjit SidhuSidhu focussed primarily on the role of pound signs in communicating particular messages and connecting social media metrics to reality in a powerful way.

He began by observing that the data is often vague. The analytics institutions receive look exactly the same as the analytics used by commercial organisations, despite the fact that their needs and objectives differ widely. He attributed this to the dominance of the technology, which has taken control over the information that gets delivered, thus ensuring everyone gets data that is easy to deliver, rather than data that is meaningful to them. Sidhu also observed that universities often fail to break down their data into relevant slices, instead viewing it at such a high level that it cannot usefully be interpreted in financial terms.

In a self-confessed rant, Sidhu emphasised that you have a chance to tell the narrative of your data. Most social media data is openly available, so if you don’t, someone else will and you will no longer have control over that narrative.

“You need to be proactive with your data. If you’re proactive, people don’t sack you.”

Sidhu went on to demonstrate the type of analytics dashboard he creates for universities, discussing the importance design as well as the analysis itself. His dashboard features nine groups of data and only three key themes, which fit onto one A4 sheet and are arranged in an attractive way. He also discussed his methodology when creating these dashboards, which involves finding out what people want to know first, then finding the data to match those requirements. This is the reverse of common practice, where people take the data that is readily available and try to fit that to their requirements.

He explained the need to match up offline experience with online experience to help to generate projections and quantify the savings produced by online tools and social media. He exemplified this by talking us through one of the most powerful statistics he creates: a calculation demonstrating the amount saved by online downloads of prospectuses compared to sending printed versions. This is usually around £500 per month. This takes the online data, combines it with existing data from the comparable offline process, and creates a tangible value.

He extended this to show other types of story we could tell with such data, including the potential value of a website visit from a specific country. Once you have this, you can more effectively demonstrate the monetary value of social media by using referrer strings to show how a visitor from that country reached your site, and therefore make better decisions about how you attract those visitors.

You have to justify your spend. Your justification has to be based on what you are trying to do at that particular time.

View the presentation in full at Vimeo

The slides to accompany this talk are available on Slideshare

Identity, Scholarship and Metrics

Martin Weller, The Open University

Martin WellerWeller posed many questions and points to ponder, focussing on how academic identity is changing now we are online.

He observed that identity is now distributed across different tools, with a greater tendency to intersect with the personal. There are more layers to consider: where once you had your discipline norms and your institutional norms, now there are more social media norms to observe to create cultural stickiness. You end up with a set of alternative representations of yourself, so your business card is now a much messier thing.

Weller went on to define impact as a change in behaviour, but emphasised that telling the story of impact online is actually very difficult. Your impact may be more about long term presence than an individual post. The metrics we currently use do not necessarily correspond to our traditional notions of academic impact: after all, what do views mean? What do links mean? What do embeds mean? How do they compare to citations?

He put forward the accepted view that blogging and tweeting provide you with an online identity, which drives attention to more traditional outputs. He placed this in the context of a digital academic footprint, which helps tell the story of the impact you are having within your community. Whilst metrics can be useful for this, he warned that they could also be dangerous, with official recognition leading to a gameable system.

He concluded by illustrating a sandwich model explaining why metrics will be increasingly important to what academics do: with top-down pressure from above to demonstrate impact when applying for funding, and bottom-up pressure from individuals asking why their impact via social media doesn’t count. Once you’ve got those two pressures, you have an inevitable situation.

View the presentation in full on Vimeo

The slides to accompany this talk are available on Slideshare

Impact of Open Media at the OU

Andrew Law, The Open University

Andrew LawLaw discussed the activities of the Open University when monitoring the various media channels used to disseminate content and how these metrics have led to real, significant funding decisions.

He observed that several of their online media channels did not necessarily have a very clear strategic remit. However, they found that the data was increasingly asking the question: “What is the purpose of all this activity?” Deeper analysis of this data led to the development of clearer stategies for these channels, based on their core institutional aims.

Law emphasised the importance of having all of the information about the different channels in one place to help dispel the myths that can grow up around particular tools. He used the example of iTunes U, which gets huge amounts of internal PR on campus, whilst channels like OpenLearn and YouTube sit very quietly in the background. However, the reality is very different and he observed that one of the challenges they face is ensuring that the broad story about the performance of all of these channels is well understood by the main stakeholders.

Law expanded on this, noting that whilst the iTunes U download statistics provide a positive story, it does not actually perform well against their KPIs compared to other channels, despite little or no investment in those other channels. He observed that their pedagogical approach to iTunes U – which includes offering multiple, small downloads, with transcripts and audio downloaded separately – can inflate the numbers. He compared this to their YouTube channel, which has received very little investment, but is performing very effectively. He also discussed the OpenLearn story, which has been quietly outstripping other channels against their KPIs – particularly in terms of conversions, because it has a lot of discoverable content. He emphasised that this is a very positive story for the university, which needs to be told and built upon.

By demonstrating these realities, the data has demanded of management a much clearer sense of purpose and strategy. This has led to real investment. The OU has massively increased the amount of money spent on YouTube and OpenLearn, representing a significant change in strategy.

In conclusion, Law did note that, so far, the data has only helped the university, not the end user, so their next steps include mapping journeys between these channels to identify the traffic blockages and better tune the service delivered across the board.

View the presentation in full on Vimeo

The Script Kiddie’s Perspective

Tony Hirst, The Open University

Tony HirstHirst provided a set of observations and reflections, which ranged from ethical issues about the use of statistics through to practical demonstrations of visualised data.

He began by observing that social media are co-opting channels that were private and making them public, so there is nothing inherently new going on. He quoted Goodhart’s Law, emphasising that, whilst measuring things can be good, once measures are adopted as targets they distort what you are measuring and create systems open to corruption.

Hirst went on to discuss the perils of summary statistics and sampling bias. He emphasised that the way you frame your expectations about the data and the information that can be lost in the processing of that data are both vital considerations if you are to accurately tell the story of that data.

Hirst discussed the role of citations as a traditional measure of scholarly impact and the ways your content can be discovered, and thereby influence through citation. He highlighted three layers of discovery: the media layer, the social layer and the search engine layer, each of which enables your material to be discovered and therefore influence behaviour. He noted that if links come through to your own domain, you can already track how they are reaching your content. What is difficult to track is when there is lots of social media activity, but none of it is coming back to your domain.

Hirst demonstrated some approaches to tracking this type of activity, including the Open University’s Course Profiles Facebook app; Google search results, which are including more personalisation; and social media statistics gleaned through APIs, many of which can be accessed via an authentication route using OAuth.

Hirst concluded by discussing some visualisations of Twitter communities to show how these can provide insight into external perspectives and how we are defined by others in our community.

View the presentation in full on Vimeo

The slides to accompany this talk are available on Slideshare

Conclusions

The workshop brought forward a number of concerns, that were often less about the tools and technologies involved, but more about the ethics and pitfalls of formalising the measurement of social media activity. The main concern seemed to be the potential for creating a gameable system, or metrics do not reflect reality in a useful way. Ensuring that the metrics we use are fit for purpose will not be an easy challenge, but the discussions held within this workshop helped to identify some potential routes to improving the value and integrity of social media data.

Posted in Evidence, Guest-post, Impact, Social Networking | Tagged: | 4 Comments »

Blog Preservation and Plugins

Posted by Brian Kelly (UK Web Focus) on 18 July 2011

Best Practices for Blog Preservation

A paper entitled “Moving From Personal to Organisational Use of the Social Web” described best practices for exploiting services such as blogs which were hosted in the Cloud. The paper further developed guidelines initially outlined in  a paper on “Approaches To Archiving Professional Blogs Hosted In The Cloud” including advice on managing the closure of a blog:

Monitoring of technologies used: Information on the technologies used to provide the blog including blog plugins, configuration options, themes, etc. can be useful if a blog environment has to be recreated.

It should be noted that advice on managing blog hosted in the Cloud might also need to be applied to blogs hosted within the institution.  As an example we implemented the above recommendation for the IWMW 2010 blog. The final post on the blog was entitled Closing the 2010 blog. In this post we documented how the blog was used (numbers of posts; numbers of contributors; etc.) and the technologies used (the them used and details of WordPress plugins which had been installed). A year later we discovered how useful it was to have provided documentation on the plugins used in the blog.

The blog environment we used to host the IWMW 2010 blog a year ago had to be upgraded. We used this as an opportunity to provide a more robust environment for additional blogs to support IWMW events, including the IWMW 2011 event.

However after the upgrade we discovered that the WordPress plugins had reverted to the defaults, with additional content which had been embedded in the blog, including the video interviews which had been published on the blog, missing from the posts. I now recall that this isn’t the first time this has happened – following a WordPress upgrade on the JISC Inform platform the plugins and the theme used on the JISC PoWR were lost and the environment had to be recreated from memory.

However the final post published last year provided the following record of the plugins which had been installed:

Details of plugins used: Akismet, Buddy Press, BP Disable Activation, Google Analyticator, Lifestream, Lux Vimeo

We subsequently re-installed the Lux Vimeo plugin – but found that the videos failed to re-appear. It seems that loss of the plugin also resulted in losing the embed code, which included the address of the videos.

Fortunately each of the posts also included a direct link to the resource on Vimeo (as illustrated in the screen shot which shows a blog post for which the video has been embedded and one for which it is still missing).  We were therefore able to re-establish the embedded video – although we decided to do this using the Embed Object plugin since this seems to provide richer functionality (and we updated the final post so that we have documented these changes).

The need to include links to remote content in addition to embedding such content was described in a post which advised Don’t Just Embed Objects; Add Links To Source Too! In this case the advice was provided in order to enhance access to content on m0bile devices, in cases in which Flash-based embedding technologies was not supported.   We have now discovered another reasons for providing such links – embedding addressing into plugins way result in the address being lost if the plugin becomes unavailable.

Best Practices for Live Blogs

The advice we had developed for those who make use of blogs stated that when archiving a blog:

Monitoring of technologies used: Information on the technologies used to provide the blog, including blog plugins, configuration options, themes, etc. can be useful if a blog environment has to be recreated.

It seems that such advice should be followed for cases when blogs which will continue to be provided are hosted on a blog platform which may be upgraded.  And since all blog platforms are liable to be upgraded the advice provided for blog preservation purposes would appear to be applicable more generally.  We are therefore applying this advice for the IWMW 2011 blog and the About page for this blog has also been updated accordingly.

Posted in preservation | Leave a Comment »