#altmetrics, My Redundancy Post and the 1-9-90 Rule

Measuring Impact in the Digital Environment

How do you assess the impact of digital content which has been published? This is a question which is very relevant in the higher education sector, where indications of success often cannot be reduced to financial indicators. It is a question which is particularly relevant to researchers who have an interest in understanding the ways in which social media can be used to maximise the impact of research papers and scholarly publications. This was a topic which was addressed recently at the UKSG 2013 conference. At the conference Mike Taylor gave a presentation on “altmetrics and the Publisher” in which he admitted the lack of consensus on the value of such approaches:

• they’re great for measuring impact in [the] diverse scholarly ecosystem
• Altmetrics are cheap gimmickry that encourage gaming the system, ie dishonesty.

A second talk entitled “altmetrics: What Are They Good For?” was given at the session by Peter Paul Groth. In his trip report Paul commented that “my main point was that altmetrics is at a stage where it can be advantageously used by scholars, projects and institutions not to rank but instead tell a story about their research“.

But there was also an awareness of the need to develop a better understanding of the strengths and weaknesses of altmetrics. We can see the importance of such metrics not only for researchers, but also for organisations which make extensive use of online technologies, through the example of W3C, the organisation responsible for the development of Web standards. In their recent weekly news digest they provided the following statistics:

Notably this week :

– over 900 stories about W3C on Twitter in 7 days.
– over 3000 mentions of W3C in 7 days.
– With 59840 Twitter followers, net increase of 521 followers in the past week.
– 19 posts that dlvr.it posted between Apr 14 – Apr 21 got 29.9K (+17.3%) clicks and reached 69.1K (+0.7%) connections. 

In light of my long-standing interest in metrics I felt it would be useful to explore metrics for blog posts and tweets.

Metrics For My Redundancy Blog Post

An Opportunity to Gather Evidence

Last Wednesday I noticed that on the day the “My Redundancy Letter Arrived Today” blog post was published my blog had received over 3,000 views (more than double the previous most popular daily visits). I realised that this provided an opportunity to explore one aspect of altmetrics: the impact of a blog post on a topic related to one’s professional activities. Since the post was published a week ago today this gives me an opportunity to collate the evidence using a variety of services and develop a better understanding of the strengths and weaknesses of such tools.

Importance of Metrics for Funders

In the past we have been asked to provide metrics related to the services we’ve provided to our funders. I recently updated the footer for blog posts, which previously included icons which facilitated ‘frictionless sharing’ to include a number of links to services which provide evidence of the extent of such sharing (although, as pointed by by Alun Hughes, who chaired the review of UKOLN and CETIS, the work of the review group was subsequently overtaken by internal changes within Jisc and the review was not concluded).

The Potential Audience for the Blog Post: TweetReach

In order to estimate the potential audience for the blog post I used the TweetReach tool to obtain estimates of the numbers of Twitter users who may have seen a tweet with a link to the blog post.

As can be seen the estimated reach at 08.30 today was 77,669, based on 50 of an estimated 145 tweets.

TweetReach also provided statistics on the size of the Twitter communities of those who have tweeted links. As can be seen had between 1,000 and 10,000 followers, followed by a significant group with between 10,000 and 100,000 followers.

TweetReach provides an indication of the total reach, with this potential reach being significant due to the numbers of Twitter users with large numbers of followers who included a link to the blog post in their tweets.

But, of course, many of the tweets will not have been seen – most experienced Twitter users will nowadays regard Twitter as a stream of information to be dipped into, and not as information which should always be processed.

The Tweeters and Retweeters: Topsy

The Topsy tool provides a greater focus on Twitter users who tweet and retweet links to the blog post (although I should add that such information is also provided by TweetReach).

From Topsy it seems that there have been 142 tweets which include links to the blog post.

As well as this headline figure, as illustrated, Topsy also provides a graph of mentions of the post over the past thirty days, as well as an archive of the tweets which contain the link.

Statistics for the Shortened URL: Bit.ly

Finally I should mentioned the statistics which are provided by the URL shortening service I use in Twitter: bit.ly.

By appending a + to a bit.ly URL you can get usage figures (by default for the past hour, but the information is also available for an extended period of time).

Looking at the statistics for https://bitly.com/17WfrgB+ (and selecting the global option) I find that there have been 1,090 clicks on the ‘bitmark’.

The bit.ly service also provides location information: over a third are from the UK; 12% from the US and since the majority (39%) are unknown this gives a long tail of other countries form which people have followed the link.

Summaries

This blog post has summarised findings from a number of Twitter analytic services which may be of interest to others who have a need to provide evidence which may help to understand the ‘impact’ of a digital resource.

However, as I have described in a post on Paradata for Online Surveys, I feel that it is important to document the survey methodology and to be open about implied assumptions as well as documenting potential pitfalls for others who may wish to replicate the findings or apply the methodology for themselves in their own context.

Blog Usage Statistics

The first potential pitfall to be aware of is that the blog usage statistics relate to the entire blog, and will include visits during the week to any of the 1,199 posts which have been published previously. The following table therefore gives the number of visits to the Redundancy blog post as well as the number of visits to the blog’s home page during the week (when the post was shown at the top of the page).

Total nos. of blog views, 24-30 April	7,442
Nos. of views of individual post, 24-30 April	5,621
Nos. of views of blog home pages, 24-30 April	765
Total nos. of views of Redundancy post, 24-30 April	6,386

It therefore appears that there have been 6,386 views of the posts during the past week, with 1,056 views of other posts on the blog.

Referrer Statistics

How did people arrive at the blog ? Looking at the referrer traffic for the past 7 days for the entire blog we can see that Twitter and Facebook were responsible for delivering most traffic, and that these two social media service were roughly comparable.

However we need to remember that referrer traffic is only provided when a Web link is followed. If visitors arrive by following a link in an email message or dedicated Twitter client, no referrer information is provided. Aggregating the referrer views it seems that 2,043 came from an identifiable Web site, with 5,399 views of all posts during the week coming either from a non-Web source or, possibly, by an anonymous Web source (e.g. a user who visits sites using an anonymising tool).

A summary of the top three ways in which people viewed content on this Web site during the past week is summarised below.

Twitter Web site	555
Facebook Web site	508
Potential Non-Web traffic	2,043

Seemingly clear indication of the social Web in delivering traffic for, admittedly, a post with human interest. Such findings will not necessarily apply in other areas, but it seems to me that such small scale indications might be useful in identifying ‘weak signals’ which would be worth investigating further in other areas.

Does the 1-9-90 Rule Apply?

As described in Wikipedia:

In Internet culture, the 1% rule or the 90–9–1 principle (sometimes also presented as 89:10:1 ratio)^[1] reflects a hypothesis that more people will lurk in a virtual community than will participate. This term is often used to refer to participation inequality in the context of the Internet.

Does this apply in the context of engagement with blog posts, I wondered? In this context I used the following definitions:

• Lurker: someone who only reads a post.
• Contributor: someone who facilitates engagement with others by lightweight ‘frictionless’ sharing, such as a tweet, a RT, a vote on the blog post, a Facebook like or a Google +1.
• Creator: someone who create new content by submitting a blog comment or commenting on Facebook.

The findings are summarised below.

Role	Activity	Numbers	Percentage
‘Lurkers’	View blog post	6,386	96%
‘Contributors’	Tweet about post	142	2.3%
	Vote on blog post	11
‘Creators’	Comment blog comments	68	1.5%
	Comment on Facebook post	32
Total		6,639

One observation I would make is that the tweets about the post are only included if they continued a link to the post. Since subsequent discussions were not included, due to the difficulties in finding such tweets, it seems that the Contributors count is understated. It therefore appears that the 1-9-90 rule may not be too far out in this case.

I’ll be the first to admit that the distinction between a contributor and a creator are somewhat arbitrary: someone who spend time in composing a relevant tweet in 140 characters (such as “A poignant, perceptive and yet defiantly uplifting post from @briankelly“) is clearly being creative. However posting a tweet will normally be a frictionless activity carried out in one’s current application environment, unlike posting a comment which is likely to involve following a link, clicking a button and filling in authentication details before creating the content. I’m therefore happy to propose this approach as a possible approach for monitoring the extent of engagement with digital content. Might this be an approach which others may be interested in helping to develop and refine?

---

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]