UK Web Focus (Brian Kelly)

Innovation and best practices for the Web

Guest Blog Post: Web 2.0 and Sustainability

Posted by ukwebfocusguest on 5 Sep 2007

Today’s guest blog post comes from Ross Gardler, manager of JISC’s OSS Watch service and a co-facilitator of a workshop session at IWMW 2007 on “Sustainable Services: Solidity based on Openness?”.


At OSS Watch we spend a considerable amount of time highlighting sustainability as one of the key benefits of open source. There is no central organisation that can simply “pull the plug” on the product and its maintenance. Open source licences ensure that the software will always be available and, while there are active users of that software, it will always be maintained.This perpetual availability of open sourced software is only one of the key benefits provided by open source licences. Another is the ability to take that software and customise it for your own needs. To add new features and to disable features not important to your situation. In other words to take a “close fit” solution and mould it into a “better fit” solution.Web services that provide open Application Programming Interfaces (APIs) present similar mix-and-match benefits, at least on the surface, that open source provides, but does it provide the same level sustainability in your solutions?This was the topic of a workshop session I hosted with Andrew Savory at the Institutional Web Management Workshop 2007 entitled “Sustainable Services: Solidity based on Openness?“. In this session we asked how participants measured the sustainability of their chosen software solutions. The list of criteria produced included items such as:

  • reliability
  • reputation
  • scale of the provider
  • significance of us as a customer
  • data ownership and openness
  • fashion
  • community
  • flexibility

The full list was far too long to detail in this post, but a few were clearly more important than others. This became particularly evident when we proceeded to evaluate a number of well known Web services against the defined criteria.

For example, data access was critical in most Web services. Was the data available in an open standard that made it interoperable with other services? Having put data into the service, could you get it out again? Flexibility was another major concern for the API approach. Did the API allow us to achieve what we want to achieve?

I would argue, like Mark Pilgrim, that this should not be an issue, we should have access to our data, and all derived data, as a matter of course – it’s our data after all. Mark observes that “praising companies for providing APIs to get your own data out is like praising auto companies for not filling your airbags with gravel.

Workshop participants also noted that there is no guarantee that a service will be provided in the future. A topic that Brian Kelly discussed here in this blog when Splashblog closed its doors. Brian suggested that such closures could be considered by some to be a clear justification for not making use of such external Web 2.0 services – a point made by a number of our session participants. Indeed, many services were marked down quite heavily since they are largely unproven beta services with no clear business model. Despite this healthy concern over the longevity of service offerings, workshop attendees felt that some services, such as Shibboleth, are more sustainable because they have public money behind them. However, as Brian goes on to observe, even public sector services are not guaranteed to be there forever. To support his point Brian cites a BBC news article describing the closure of 551 government Web sites and wonders what happens to data held by the AHDS when funding ceases.

The overall conclusion of our workshop attendees was that Web services should only be relied upon for non-critical functions in your institution. Over time we may become more comfortable with relying on third party services, but for now we need to be careful. I liken it to the development of voice communications technologies. We don’t worry about having a dial tone the next time we pick up the phone, but the recent Skype outage shows we can’t rely on the newer voice communications services. The result is that Skype is not suitable for emergency calls.

Reaching Sustainability Through Openness

In my opinion one way of moving towards more sustainable services at a sensible pace is through openness in the development of those services. That is, if a service uses open data standards, provides fully open access to all its data and its APIs and encourages users to participate in the ongoing development of the service, I, as a user, am more likely to stick with it past my initial, experimental, use. For example, I love the idea of Dopplr, but I haven’t gone past exploration because it fails to provide the data in format that is useful to my objectives (Editor’s Note: Phil Wilson pointed out that a Doppler API has recently been announced at http://dopplr.pbwiki.com/. This comment was added at the request of Ross Gardler on 6 Septmeber 2007). Conversely, just 10 hours after the announcement of a beta API for OhLoh I had integrated OhLoh data into Simal, the OSS Watch project cataloguing tool. As soon as OhLoh produces an API for submitting data I’ll ensure the flow is two way, making both projects more likely to survive.

However, openness should not stop at the data and the APIs. I need to ensure that the service remains aligned with my strategic objectives. I want to be able to contribute directly to the flexibility and sustainability of the service in ways that suit my needs. This is where Oh Loh falls down, it is not open source and so my contribution options are limited.

Open source enables us, as users, to choose how to invest our resources in sustainable solutions. We can purchase related products such as support and hosting, or we can fund strategic development, or we can ensure our own staff help support and sustain the product through direct contribution of use cases, documentation, feature requests, bug fixes and even new feature implementations. All of these actions help ensure the product survives and continues to be available to our own organisation.

Web service companies will gladly accept similar contributions from us. The big difference between the two approaches is that with open source we have the freedom to decide where our resources are invested. We can maximise the impact our investment has on our individual utilisation of the service, thus making the service more useful. We are even free to take the software and create our own version should our objectives diverge considerably from the originating service provider (although this can usually be avoided if the project is well managed and cultivates a healthy community).

Most of us want the convenience of a service provider, but such convenience comes with the risk of potential lock-in and, even worse, the loss of a critical service. Having access to the source code means that we increase competition and consequently increase innovation in the code base. It does not prevent companies from differentiating themselves through the provision a more reliable and usable service within their chosen market niche.

Given the choice, I will always use a Web service that makes its source code available under an open source licence, even if that service is less developed than closed competitors. In most cases I will still purchase the service from a provider, but I want to keep my options open in order to ensure my own offerings are sustainable.

Our workshop participants largely agreed with this view, they too were more concerned about having control over their own organisations future in the long term than they were about the short term gains of adopting closed service models.

Ross Gardler
OSS Watch
OUCS
13 Banbury Road
University of Oxford
Oxford
OX2 6NN

OSS Watch Web site: http://www.oss-watch.ac.uk/
OSS Watch blog: http://involve.jisc.ac.uk/wpmu/oss-watch/

14 Responses to “Guest Blog Post: Web 2.0 and Sustainability”

  1. What is it about Dopplr’s API you don’t like? MattB eagerly welcomes feedback.

    If you’re going to quote Mark Pilgrim, perhaps it would have been useful to link to Freedom 0?

  2. Phil – as far as I am aware, there isn’t a Dopplr API – having one would e a start (my personal tops needs are: adding trips via a rest interface and getting details of who is where on any given day). I have spoken personally to MattB about this and have also put a feature request in via the correct channels. Dopplr is in beta, my comments are not meant as a criticism of the idea (which is great) only of the current lack of API. Hopefully, by the time it is a full product there will be an API.

    With respect to the link to Mark Pilgrim… errr… there is one – click on his name and search for the quoted phrase, it’s on the linked page.

  3. The API is on http://dopplr.pbwiki.com/

    Ah, double-checking it’s only been there a few days, I just got early wind of it.

    There’s a link to Mark, but not his Freedom 0 post, which I think is a good summary of some of the issues you voice here.

  4. I agree with most of your post, but especially in terms of service provision there are always the practical considerations to bear in mind, which may mean that a closed-source commercial package is actually more appropriate. Some obvious targets would be short-lived projects or where the open version is woefully behind a commercial version and you can’t dedicate the resource to bring it up to speed.

  5. Anthony Leonard said

    I heard a talk recently by someone from the Open Knowledge Initive. Their approach to sustainability is simple: software systems should plug together without the need for a developer, period.

    For free-riding optimist hackers like myself it was a hard message to take, but it’s true. Think about it. A domestic plug isn’t “open-source” so a savvy engineer can fiddle with it to make it work in his house in ten years time. It’s a boring standard that sits in a large book of building regulations somewhere that means people in ten years people can use it without needing to know how it works.

    Sustainability = Engineering. It’s time that software developers started having to live with being constrained by more standards – and funding bodies of open source projects insisted on them – so that we can start engineering sustainable software that, like buildings, are good for the next 50 years, not the next 5.

  6. Paul Walk said

    I would argue, like Mark Pilgrim, that this should not be an issue, we should have access to our data, and all derived data, as a matter of course – it’s our data after all.

    Well…. yes and no. I’m particularly interested in the long running debates about ‘attention’ data. If Amazon, for example, chooses to note which products I’ve looked at, which I’ve bought etc. and use this attention data to build a better, personalised service, does it follow that I should have access to all such ‘derived’ data? The idealist in me says ‘yes’, but what little legal analysis there has been of this position says ‘not necessarily’.

    A service provider capturing such attention data might argue that the user is perfectly able to do likewise – it’s just that the service provider has bothered to do so. Some of the early work by attentiontrust.org was to develop a browser extension (Attention Recorder) to allow the user to record their own ‘gestures’.

    What ‘rights’ do we, or should we, actually have to such ‘derived’ data?

  7. Ross Gardler said

    Paul,

    An interesting point and I agree that who owns the data can be confusing.

    My own personal take is that data derived from my own activities is my data. In fact, the Data Protection Act empowers me to insist that data held about me is provided upon request (and payment of up to £10). I am not a lawyer, but I believe that this means that derived data must also be provided. Of course, much of this will be useless, “you like the same music as account 11198283” is pretty meaningless unless you also have access to account 11198283. Being able to transfer that information from one service to another, where Amazon account 11198283 may also be represented, increases the value of your data to you and to the new service.

    Note, this is not anti-competitive, quite the reverse. It still allows plenty of scope for companies to build a business around their unique services, but now that the data is at least partially portable we allow new competitors to enter an established marketplace. This increases competition and increases innovation.

    [as a slightly OT interest point, even CCTV footage of me has to be provided upon request, regardless of the cost of retrieval to the footage owner. See the story of a movie made from CCTV footage]

  8. Ross Gardler said

    Phil, thanks for the Dopplr API pointer. Brian, in fairness to Dopplr can you please add an editors comment to the main text. I will now be revisiting my use of Dopplr and embedding it in Simal – cool!

  9. One of the biggest problems is that these web services like Dopplr, Flickr, Slideshare, etc. tend to be very bespoke both in software and hardware because they are designing a service specifically to cope with millions of concurrent users. In many cases, even if you had all the source code, implementing your own version would require you owning or paying for hosting of several acres of data centre, all connected in the right way with several NetScalers. Or a major rewrite of the backend of the service. I simply can’t think that, in comparison with writing/buying a system more suited to scale and purpose of your organisation, that this would be worth it. I’d doubt any web 2.0 service would see this as a sound part of a business plan either.

    I understand what you are saying about APIs, but don’t accept them to be an argument to do with sustainability. We shouldn’t be giving them our data but A COPY of our data, surely? If you are, for example, putting a presentation on SlideShare, it should also be available from your own site. All the source code and APIs in the world are of no use to get that data back the day the CEO walks into their office and goes “we haven’t got our latest round of investment, you’re all sacked, switch everything off”.

  10. Hi Mark
    When you wrote “We shouldn’t be giving them our data but A COPY of our data, surely?” my initial reaction was toi agree strongly. This is the approach I take with Slideshare, with the videos I upload to YouTube and Google Video, etc. And, in the case of Slideshare, I also ensure that the slides and the accompanying metadata provide a URI of the location of the master copy.

    But then I realised that I don’t do this with my use of del.icio.us or, indeed, this blog. In these cases I am taking a risk that the services won’t disappear overnight (and, for these services, I am willing to take this risk). But I’ll also need an API (or data export functionality) to ensure that I can access my data.

  11. Hi Mark,

    I concur with Brian. Some data is not in other locations to start with, such data is not limited to the examples Brian gives though. If I put a presentation on SlideShare it will only be a copy, sure, but the data added later, by other users, such as comments, favourite flags and related items are not copies.

    With respect to your point about scalability I partially agree with your points. However, we must recognise that most users will only care about a fraction of the accounts available. They do not need hosting of several acres of data centre, all connected in the right way with several NetScalers. What most players need is a few thousand connected accounts at most.

    That being said. most people do want the convenience of a hosted service. My assertion that this should not be at the expense of being tied to a specific service provider. To undersand why I would encourage you to consider how long it is taking to get true competition, along with all the cost savings and innovations that brings, into the telecoms industry.

    Open source code means that if there is a market niche available, other companies can operate in that niche. Such a niche may be a specialism or it may be that there are dissatisfied customers of the dominant player.

    This does bring the problem of creating multiple silos of data, which in turn makes the reduces total value of all data. We also need open standards to ensure interoperability between providers. Again open source helps here as a driver of open standards adoption.

    I’m not claiming that a solution from a company that only provides an API is necessarily unsustainable, instead I am claiming that one that provides a hosted service based on open source code is more sustainable since it opens many more opportunities.

  12. […] Furthermore, I don’t want to maintain my network manually, in multiple places and in an externally owned data store. The vast majority of this information can be derived from my mail communications logs and […]

  13. […] all we want is a set of open source social network tools that allow us to build the niche social networks we […]

  14. […] Furthermore, I don’t want to maintain my network manually, in multiple places and in an externally owned data store. The vast majority of this information can be derived from my mail communications logs, networking […]

Leave a comment