What Are the #jiscbid Evaluators Thinking?

A few weeks ago Gráinne Conole, a professor of e-learning at the Open University, used Twitter to ask for suggestions on how to go about writing a bid for one of the forthcoming JISC calls. And, as I recently described, many useful suggestions were given – despite that fact that this was a competitive process and the suggestions were being provided in an open space.  A great example of a community working in an open fashion, I feel. And the benefits (better prepared submissions, clearer ideas of approaches to project management and dissemination described, etc.) will be beneficial to many stakeholders, including the JISC programme managers, evaluators of the proposals and, eventually, the users of the project deliverables.

But did this happen? Were the bids well-written and had they followed the guidelines? Or was marking the bids a time-consuming and difficult process for the many evaluators who were involved in marking the bids?

Well we can get an insight into the evaluators though processes by looking at the Twitter stream for tweets tagged with “jiscbids”.  I think this tag was originally developed by an informal process, although at one point Amber Thomas (JISC Programme Manager) did suggest that this should be the tag adopted for sharing thoughts on the evaluation process:

#jiscbids dons [corporate hat] i am assuming all other markers are using this hashtag to offer constructive comments on anonymised bids too

Subsequently Sam Easterby-Smith (CETIS) commented that he was:

Finding the #jiscbids tweet feed rather too fascinating… @briankelly MUST do follow up to his blog post from feb 5th – woo

I’ll not, however, discuss the details of the tweets, other than to say that having the opportunity to observe evaluators’ thoughts on the marking process should provide immensely valuable feedback to those at JISC who are responsible for managing the evaluation process. There have been discussions, for example, on whether bids which were over the maximum number of pages allowed should be automatically discarded (possibly before reaching the markers) or whether such bids should be marked down, but could still be funded it the bid is strong enough.

Having been involved in bid marking in the past it has only just struck me that  in my experience there has been very little discussions on the evaluation process itself, perhaps because once the marks are returned to JISC the programme managers will be busy comparing the responses, making final decisions, suggesting changes to proposals, etc. By the time this is all over, I suspect there will be little energy left for reflecting on the evaluation process.

So I hope that someone will find the time and energy to go through the various tweets made by the evaluators (including those which did not have a #jiscbids tag). But as well as identifying aspects of the reviewing process which can be improved, there will also be a need to consider whether the openness and informality which Twitter has provided could be in conflict with a closed reviewing process. I disagree with Mike Ellis’s view that Twitter “needs an edge, a voice, a riskiness” – in some cases this may be true, but in discussing a bidding process or, as my colleague Marieke Guy has recently commented,  in the context of discussing talks at conferences, we need to establish best practices. But I hope the best practices which emerge acknowledge the benefits which can be gained from using services such as Twitter.