UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

  • Email Subscription (Feedburner)

  • Twitter

    Posts on this blog cover ideas often discussed on Twitter. Feel free to follow @briankelly.

    Brian Kelly on Twitter Counter

  • Syndicate This Page

    RSS Feed for this page


    Creative Commons License
    This work is licensed under a Creative Commons Attribution 2.0 UK: England & Wales License. As described in a blog post this licence applies to textual content published by the author and (unless stated otherwise) guest bloggers. Also note that on 24 October 2011 the licence was changed from CC-BY-SA to CC-BY. Comments posted on this blog will also be deemed to have been published with this licence. Please note though, that images and other resources embedded in the blog may not be covered by this licence.

    Contact Details

    Brian's email address is You can also follow him on Twitter using the ID briankelly. Also note that the @ukwebfocus Twitter ID provides automated alerts of new blog posts.

  • Contact Details

    My LinkedIn profile provides details of my professional activities.

    View Brian Kelly's profile on LinkedIn

    Also see my profile.

  • Top Posts & Pages

  • Privacy


    This blog is hosted by which uses Google Analytics (which makes use of 'cookie' technologies) to provide the blog owner with information on usage of this blog.

    Other Privacy Issues

    If you wish to make a comment on this blog you must provide an email address. This is required in order to minimise comment spamming. The email address will not be made public.

Developments to Twapper Keeper

Posted by Brian Kelly on 20 Apr 2010

Many of my posts have mentioned a variety of Web services which are available in the Cloud. The importance of such services is now widely acknowledged but the issues about sustainability and engagement with the service providers are still as important as ever – though they are not the insurmountable barriers which they may have appeared to be in the past.

This is the context to JISC-funded development work to the Twapper Keeper Twitter archiving service which UKOLN is project-managing.  The Twapper Keeper service is reasonably well-known in the JISC development and e-learning communities: the service has been used, for example, to archive tweets from a number of high profile events including the ALT-C 2009 conference (see the archive of over 4,700 #altc2009 tweets), UKOLN’s Institutional Web Management Workshop (see the archive of over 1,600 #iwmw2009 tweets) and, most recently, the JISC’s 21010 conference (see the archive of over 1,900 #jisc10 tweets).

But how might the service be improved?  What enhancements could be made to the Web interface for users wishing to  archive groups of tweets or who wish to access existing archives of tweets?

As well as the user interface there are also questions about developments to the Twapper Keeper APIs so that developers can access the service and the data in order to reuse the data and avoid the data being trapped into a single application.

In addition to such development activities there will also be enhancements to the service environment to ensure that the service provides a reliable and resilient service.

It is also intended to provide an open source licence for the software components and provide documentation and additional supporting materials under an open content licence.

The development activities will be documented on the Twapper Keeper blog. An introduction to this development work was posted last week. Two subsequent posts have just been published which invite suggestions and comments on User Enhancements toTwapper Keeper and API Developments to Twapper Keeper. John O’Brien, Twapper Keeper developer and myself will use the blog to provide updates on developments.

12 Responses to “Developments to Twapper Keeper”

  1. Tony Hirst said

    As per your tweet, and re: additional requirements

    – ability to filter a hashtagged archive via a list of users;
    – ability to archive a twitter list;
    – ability to grab snapshots of friends/followers of a set of individuals over a short period eg once a day, in order to chart how a network grows (cf ). This might require grabbing the friends list from a dynamic set of users – eg folk who are using a particular hashtag.

  2. Easily generate an aggregated set of tweets for reformatting or re-purposing as a record or display? Probably some sort of wizard or faceted browsing process that lets a user collect the set of tweets that they want and then create a URL that identifies that collection.

  3. How about the ability to separate RTs out so you can view just original content, rather than all the repeats?

  4. I’ve left a couple of suggestions on the TwapperKeeper blog – I assume that’s where you intended them to be submitted?

    More generally, I’ll raise the elephant in the room (well, it may not be in there yet but I guess it will be before long) – what impact does the news about Twitter and the Library of Congress have on this work/area?

    Related to this… (and as with anything related to archiving of one kind or another) I think sustainability of TwapperKeeper is the critical thing here (by sustainability I don’t mean JISC funding!). I would expect this work to at least be mindful of sustainability options and their acceptability to the JISC community.

    • Hi Andy

      The Twapper Keeper blog will be a better place for technical suggestions, but John O’Brien will keep an eye on suggestions made here.

      We have discussed last week’s announcement at the Chirp Twitter conference that the Library of Congress will be archiving all Twitter posts.
      This is good news, I feel, as it helps to legitimise the importance of Twitter and the value of Twitter archiving.

      Note, though, that as described in The GuardianThere will be restrictions of the use of tweets. According to the Twitter blog, there will be a six month delay before tweets are available for internal library use, for non-commercial research, public display by the library itself and preservation“.

      I feel that Twapper Keeper will be playing a role in preserving tweets for communities (such as the communities based around an event and an event hashtag) and provide rapid access to the archives in order to ensure that timely analyses can be carried out (such as, indeed, your summary of the JISC10 conference tweets).

      Finally you are correct to highlight the importance of the sustainability of the service. This is another area which will be addressed in this work. Well be writing something about the issues and proposals at a later date – for now we are focussing on the user requirements gathering.

  5. It would be great to have a feature on Twapper Keeper where individual users can opt out of their updates being archived. I’m fully aware that public tweets get archived in many places including Google, but I think there is a special responsibility on sites like this to set a good example (especially as it has UKOLN involvement). When people tweet about something they have seen on X-Factor for example (archived at they may not really want this tweet to be used for academic research in the future. They might just be having fun in the spirit of the moment. Archiving services like Twapper Keeper are a relatively new concept and I think there is a risk people might find them a bit “creepy” and not be comfortable with them. An ability to opt out puts people more in control, if they are really that unhappy there is something they can do about it. This to me also seems ethically to be a much better position.

  6. mhawksey said

    Added some thought to the Twapper Keeper blog posts. Ability to refine the search for both web and API are probably the most important to me.

    I think Liam’s suggestion might be a necessity because of IPR issues, certainly a grey area which you’ve already highlighted in earlier posts (I’m still waiting to be martyred – but have already taken steps on my own Twapper Keeper integration to try and limit liability (see disclaimer). It would be better if this could be done at source.

  7. […] on the UK Web Focus blog Liam Green-Hughes requested that it should be possible for users to delete their tweets from the Twapper Keeper archive. Liam […]

  8. […] Developments to Twapper Keeper […]

  9. […] Plans for Updates to Twapper Keeper Functionality and APIs By Brian Kelly (UK Web Focus) We recently invited suggestions for developments to the end user functionality provided by Twapper Keeper and the APIs provided by the service (with some additional suggestions being made on the UK Web Focus blog) […]

  10. […] as well as prevent users from archiving other person’s timelines, which was brought up as a concern while gathering requirements for the JISC funded enhancements to the […]

  11. […] by Brian Kelly (UK Web Focus) on 9 September 2010 I’ve previously described how JISC have funded development work for the Twapper Keeper Twitter archiving service and that I […]

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: