UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

Posts Tagged ‘Twapperkeeper’

Conventions For Metrics For Event-Related Tweets

Posted by Brian Kelly on 15 Nov 2010

According to Summarizr there have been 6,927 tweets for the #altc2010 event hashtag, which compares with 4,735 tweets for the #altc2009 event. We can therefore conclude that there has been an increase of almost 50% in Twitter usage. Or can we? If we had carried out the analysis immediately after the event the numbers would probably have been different. And use of either of these hashtags now, when talking about a past event, will have a different context to using the hashtag during the event, when such tags provided some level of engagement with the Twitter community centred around the event’s Twitter stream.

In order to make meaningful comparisons there is a need to be able to filter the tweets in a consistent fashion. Fortunately the Twapper Keeper service allows tweets to be filtered by various parameters, including a date range. And since the Summarizr service uses Twapper Keeper to provide its statistics it is possible to use Summarizr’s metrics in a consistent fashion.

But what date range should be used? An initial suggestion might be for the day(s) of the event. But this would fail to include discussions which take place immediately before and after an event. In addition this could also mean that tweets from an international audience not being included, such as tweets from an Australian audience which take place the following day. Such confusions over dates might apply particularly to events held in other countries since the times used in Twitter are based on GMT.

In order to avoid such confusions when I cite statistics from Summarizr I now include tweets posted during the week of an event, typically starting on the Sunday and finishing on the following Saturday. For an event lasting for a day I start on the day before the event and finish on the following day.

The syntax for obtaining statistics from Twapper Keeper over a date range is:


sm is the start month (from 1 to 12)
sd is the start day (from 1 to 31)
sy is the start year (e.g. 2010)

em is the end month (from 1 to 12)
ed is the end day (from 1 to 31)
ey is the end year (e.g. 2010)

For example the following URL will give statistics for the #altc2009 hashtag between 6-11 September 2009:

and the following statistics for the #altc2010 hashtag between 5-12 September 2010:

This provides the following statistics:

ALT-C 2009 ALT-C 2010
Nos. of tweets 4,010 6,238
Nos. of twitterers    650    666
Nos. of hashtags tweeted    125    277
Nos. of URLs tweeted    554    683
Nos. of geo-located tweets        0      35

This indicates that there has been of 56% in twitter usage between comparable periods in 2009 and 2010.

Note that the statistics for the numbers of geo-located tweets demonstrate that in 2009 nobody was providing geo-located tweets for the event hashtag. This data could easily be lost if Twitter users today started to refer to the 2009 event and had started to make use of geo-location.

To sum up my proposal:

  • The start date for a one-day event is the previous day and the end date is the following day. This will address internationalisation issues due to engagement for those in other time zones and cover discussions just before and just after the event.
  • The start date for an event lasting longer than a single day is the previous Sunday and the end date is the following Saturday. This will address internationalisation issues due to engagement for those in other time zones and cover discussions just before and just after the event.

Is this a convention we can agree on, to ensure that meaningful comparisons can be made?

Twitter conversation from Topsy: [View]

Posted in Twitter | Tagged: , | 5 Comments »

Twapper Keeper Goes Open Source

Posted by Brian Kelly on 9 Sep 2010

I’ve previously described how JISC have funded development work for the Twapper Keeper Twitter archiving service and that I am the project manager for this work.

The initial area of work addressed the robustness of the service by seeking to “ensure the Twapper Keeper service could continue to be a viable platform for archiving tweets“. As described on the Twapper Keeper blognumerous operational / infrastructure issues were addressed“.

The second area to be addressed was the functionality of the service, both to end users of the Web interface and to developers who may wish to make use of the Twapper Keeper APIs.  The Twapper Keeper blog was used to gather suggestions for developments to the user interface and the APIs. In May a summary of the Plans for Updates to Twapper Keeper Functionality and APIs was published and these updates have now been implemented.

The final area of work was to address the longer term sustainability of the service. The approach taken was to minimise the risks of loss of the centralised Twapper Keeper by ensuring that the software components were available as open source.

On 25 August the Twapper Keeper blog “Announced yourTwapperKeeper – archive your own tweets on your own server!“.  This provided the information that “As part of our partnership with JISC, we are now releasing an open version of Twapper Keeper that is designed to run on your own server“.

So in addition to the main Twapper Keeper service there is now an “open / downloadable version that can be run on your server!” which is available from Google Projects. In addition there is also an option whereby you can subscribe to a hosted version of the Twapper Keeper service.

And, of course, you can export your data from Twapper Keeper in a variety of formats (HTML, RSS, JSON and MS Excel).

I’m pleased that the JISC funding for this development work has provided various benefits, not only for end users but also for the developer community. But most importantly, I feel, are the ways in which the development approaches have sought to address the much more challenging issues of the longer term sustainability.

Posted in Twitter | Tagged: | 2 Comments »

Developments to Twapper Keeper

Posted by Brian Kelly on 20 Apr 2010

Many of my posts have mentioned a variety of Web services which are available in the Cloud. The importance of such services is now widely acknowledged but the issues about sustainability and engagement with the service providers are still as important as ever – though they are not the insurmountable barriers which they may have appeared to be in the past.

This is the context to JISC-funded development work to the Twapper Keeper Twitter archiving service which UKOLN is project-managing.  The Twapper Keeper service is reasonably well-known in the JISC development and e-learning communities: the service has been used, for example, to archive tweets from a number of high profile events including the ALT-C 2009 conference (see the archive of over 4,700 #altc2009 tweets), UKOLN’s Institutional Web Management Workshop (see the archive of over 1,600 #iwmw2009 tweets) and, most recently, the JISC’s 21010 conference (see the archive of over 1,900 #jisc10 tweets).

But how might the service be improved?  What enhancements could be made to the Web interface for users wishing to  archive groups of tweets or who wish to access existing archives of tweets?

As well as the user interface there are also questions about developments to the Twapper Keeper APIs so that developers can access the service and the data in order to reuse the data and avoid the data being trapped into a single application.

In addition to such development activities there will also be enhancements to the service environment to ensure that the service provides a reliable and resilient service.

It is also intended to provide an open source licence for the software components and provide documentation and additional supporting materials under an open content licence.

The development activities will be documented on the Twapper Keeper blog. An introduction to this development work was posted last week. Two subsequent posts have just been published which invite suggestions and comments on User Enhancements toTwapper Keeper and API Developments to Twapper Keeper. John O’Brien, Twapper Keeper developer and myself will use the blog to provide updates on developments.

Posted in Twitter | Tagged: | 12 Comments »