UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

What Are the #jiscbid Evaluators Thinking?

Posted by Brian Kelly on 4 Mar 2009

A few weeks ago Gráinne Conole, a professor of e-learning at the Open University, used Twitter to ask for suggestions on how to go about writing a bid for one of the forthcoming JISC calls. And, as I recently described, many useful suggestions were given – despite that fact that this was a competitive process and the suggestions were being provided in an open space.  A great example of a community working in an open fashion, I feel. And the benefits (better prepared submissions, clearer ideas of approaches to project management and dissemination described, etc.) will be beneficial to many stakeholders, including the JISC programme managers, evaluators of the proposals and, eventually, the users of the project deliverables.

But did this happen? Were the bids well-written and had they followed the guidelines? Or was marking the bids a time-consuming and difficult process for the many evaluators who were involved in marking the bids?

Examples of Twitter posts tagged 'jiscbids'Well we can get an insight into the evaluators though processes by looking at the Twitter stream for tweets tagged with “jiscbids”.  I think this tag was originally developed by an informal process, although at one point Amber Thomas (JISC Programme Manager) did suggest that this should be the tag adopted for sharing thoughts on the evaluation process:

#jiscbids dons [corporate hat] i am assuming all other markers are using this hashtag to offer constructive comments on anonymised bids too

Subsequently Sam Easterby-Smith (CETIS) commented that he was:

Finding the #jiscbids tweet feed rather too fascinating… @briankelly MUST do follow up to his blog post from feb 5th – woo

I’ll not, however, discuss the details of the tweets, other than to say that having the opportunity to observe evaluators’ thoughts on the marking process should provide immensely valuable feedback to those at JISC who are responsible for managing the evaluation process. There have been discussions, for example, on whether bids which were over the maximum number of pages allowed should be automatically discarded (possibly before reaching the markers) or whether such bids should be marked down, but could still be funded it the bid is strong enough.

Having been involved in bid marking in the past it has only just struck me that  in my experience there has been very little discussions on the evaluation process itself, perhaps because once the marks are returned to JISC the programme managers will be busy comparing the responses, making final decisions, suggesting changes to proposals, etc. By the time this is all over, I suspect there will be little energy left for reflecting on the evaluation process.

So I hope that someone will find the time and energy to go through the various tweets made by the evaluators (including those which did not have a #jiscbids tag). But as well as identifying aspects of the reviewing process which can be improved, there will also be a need to consider whether the openness and informality which Twitter has provided could be in conflict with a closed reviewing process. I disagree with Mike Ellis’s view that Twitter “needs an edge, a voice, a riskiness” – in some cases this may be true, but in discussing a bidding process or, as my colleague Marieke Guy has recently commented,  in the context of discussing talks at conferences, we need to establish best practices. But I hope the best practices which emerge acknowledge the benefits which can be gained from using services such as Twitter.

4 Responses to “What Are the #jiscbid Evaluators Thinking?”

  1. Lawrie Phipps said

    The Tweets were fascinating. As at 15:55 on 4 March and post deadline for last call there were 61 tweets pertaining to the recent round of marking. Of these 61;
    – 4 referenced marking briefings e.g “not being very successful in locating & identifying all the background info needed in order to mark #jiscbids & getting teensy bit annoyed” please note that no one responded with the correct information.
    – 9 mentioned the process of marking e.g. “That’s 6 minutes to transcribe 1 #jiscbids markset from template to web form with cut/paste. Could’ve been 30 secs for all 9 with upload :-(”
    – 13 were evaluative (of these 5 actually repeated what someone else had said in one shape or another)
    – The remaining tweets discussed, amongst other things, “having lunch and then back to marking”, “I’m marking” and “tips for music to mark bids by” (I’ll own up to the last one!)

    It is also important to remember that the 61 Tweets were sent by 17 people, of which 20 of the tweets were sent by 2 people.

    In your article you suggest “having the opportunity to observe evaluators’ thoughts on the marking process should provide immensely valuable feedback to those at JISC who are responsible for managing the evaluation process.”

    NOT from this twitter stream. 9 posts referred to JISC influenced processes, and only one of the suggestions is within the realms of control. The image you posted on the side of the blog is very nice, but the tweets you choose to represent are not representative of the majority of the discussion.

  2. […] note today that Brian has used a set of Twitter exchanges on another post. I would suggest that at least one of the tweets featured there was not something the author would […]

  3. […] recent 12/08 call, including discussion of how Twitter can help you prepare a bid and how it was used (and perhaps abused) during the marking process. Andy Powell has vented his frustration on some […]

  4. […] few weeks later Twitter discussions centred about the evaluations of bids. In “What Are the #jiscbid Evaluators Thinking?” in which I summarised an “insight into the evaluators though processes by looking at […]

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: