Policies on Drugs, Open Standards and Web Accessibility

Over the weekend we’ve been hearing about the squabbles between the Government and the Advisory Council on the Misuse of Drugs. Professor David Nutt, chairman of the Advisory Council, argued that cannabis was less harmful than alcohol and tobacco and that it was upgraded by the Government to Class B against the council’s advice – for political reasons. In response, as described on the BCC News, the Home Secretary “Johnson defends drugs row sacking“  saying that Professor David Nutt went against a long established principle by straying into politics.

An example of a political expediency taking precedence over evidence, surely? After all, we can predict the headlines in papers such as the Daily Mail if the Advisory Council’s recommendations had been accepted by the government. 

But if we feel that evidence and the need to acknowledge the accompanying complexities should outweigh an approach based on simple slogans would such an approach also be used in the context of IT development work? 

This thought came to me earlier today after reading a tweet from Wilbert Kraan which stated

“RT @PeterMcAllister: EU wants to get rid of open standards: http://is.gd/4KMUi (via @brenno) Leaked draft: http://bit.ly/2tTN7X #EUopenS“

The accompanying blog post , headlined “EC wil af van open standaarden“ begins

“De Europese Commissie schrapt in stilte open standaarden voor interoperabiliteit. Het draait nog slechts om ‘open specificaties’, waarbij patenten en betaalde licenties geen taboe meer zijn.”

Friends on Twitter have responded to my request for a translation and suggest that the post on”The European Commission silently scraps interoperability standards” begins with the view that:

“The EU has quietly changed its view on open standards and no longer sees patents and paid licensing as taboos”

The EU has changed its mind on open standards?  That sound intriguing! So I’ve skimmed though the ”European Interoperability Framework for European Public Services (version 2.0)“  document (PDF file) – which, I should add, is clearly labelled as a work in progress.

This report is of interest to me as I recently gave a talk at the ILI 2009 conference entitled “Standards Are Like Sausages: Exploiting the Potential of Open Standards“.  In the talk I described how my early work in promoting open standards (which date back to my contributions to the eLib Standards document back in 1995) can, in retrospect, be seen to be naive. Over the years I have found myself recommending open standards, especially those developed by the W3C, which have failed to gain significant acceptance in the market place. And, just as, the Daily Mail knows it is safe to promote a zero tolerance approach to drugs to its core audience, I was also aware that promoting open standards is a safe thing to do in a public sector IT development context.  But over the years I have begun to realise that such recommendations need to be informed by evidence – and if the evidence is lacking there may be a need for a more refined approach, rather than a continuation of the “One final push” approach. 

These views also apply in the context of Web accessibility. I have argued for several years that an approach based solely on technical conformance with a set of accessibility standards, which fails to acknowledge the diversity of use cases, definitions of accessibility, limitations of relevant tools available in the market place and the resource implications of conforming with such flawed approaches, is the wrong approach to take.

In light of this I was very interested in what the EU’s draft document on the European Interoperability Framework for European Public Services had to say.

What did I find in this document about the European Interoperability Framework (EIF) which aims to promote and support the delivery:

1.5.1 The Political and Historical Context of Interoperability in the EU:: I welcome the section which acknowledges that political and historical issues have a significant role to play in enhancing the delivery of interoperable services.  

2.2 Underlying Principle 1: Subsidiarity and Proportionality. This section goes on to add that “The subsidiarity principle implies that EU decisions are taken as closely as possible to the citizen. In other words, the Union does not take action unless EU action is more effective than action taken at national, regional or local level“. It the context of IT services, I see this as endorsing a user-focussed approach to development work, rather than the centralised imposition of solutions. Section 2.3 Underlying Principle 2: User Centricity reinforces this approach.

2.4 Underlying Principle 3: Inclusion and Accessibility. This section goes on to add that “Inclusion aims to take full advantage of opportunities offered by new technologies to overcome social and economic disadvantages and exclusion. Accessibility aims at ensuring people with disabilities and the elderly access to public services so they can experience the same service levels as all other citizens.“

We then read that “Inclusion and accessibility usually encompass multichannel delivery. Traditional service delivery channels may need to co-exist with new channels established using technology, giving citizens a choice of access.” Hurray – we’re moving away from the WAI perspective that suggests that all Web resources must be universally accessible to all, to an inclusive approach which endorses a diversity of delivery channels!

2.10 Underlying Principle 9: Openness. This section goes on to add that “openness is the willingness of persons, organisations or other members of a community of interest to share knowledge and to stimulate debate within that community of interest, having as ultimate goal the advancement of knowledge and the use thereof to solve relevant problems. In that sense, openness leads to considerable gains in efficiency.” I’m pleased to see this emphasis on the benefits of openness of content and engagement endorsed in the document.

This section than states that:

Interoperability involves the sharing of information and knowledge between organisations, hence implies a certain degree of openness. There are varying degrees of openness.

Specifications, software and software development methods that promote collaboration and the results of which can freely be accessed, reused and shared are considered open and lie at one end of the spectrum while non-documented, proprietary specifications, proprietary software and the reluctance or resistance to reuse solutions, i.e. the “not invented here” syndrome, lie at the other end.

The spectrum of approaches that lies between these two extremes can be called the openness continuum.

We are seeing an appreciation of complexities and a “spectrum of approaches [to openness]” rather than a binary division which is promoted by hardliners.

2.12 Underlying Principle 11: Technological Neutrality and Adaptability. This principle leads to “Recommendation 7. Public administration should not impose any specific technological solution on citizens, businesses and other administrations wh n establishing European Public Services.” Having acknowledged the needs to be user-centric and to encourage openness, whilst recognised that there may be a spectrum of approaches which need to be taken, the document spells out the implications that specific technical solutions should not be imposed.

Chapter 4 of the document introduces four Interoperability Levels, as illustrated.

Although not depicted in the diagram for me this indicates the team for the technical discussions and decisions about interoperability need to be formed within the context of political, legal, organisation, and semantic considerations. Surely self-evident when stated like this, but not when we hear mantras such as “interoperability through open standards” being promoted at a policy level which can lead to discussions taking place in which other considerations can become marginalised.

Has the  ”EU quietly changed its view on open standards and no longer sees patents and paid licensing as taboos“. Or might we suggest that the “The EU is now taking a pragmatic approach to the relevance of standards in ICT development. It now feels that the technical considerations need to be placed in a wider context“?

The Network Effect Is Missing From The Standards Debate

In a recent post I asked “Do We Want A Standards-based Voice/Video Service?“. The post suggested that the failure of the JANET Talk service to gain significant support or interest provided evidence of the failure of a development approach based solely or primarily on support for open standards.

In a response to the post, Nick Skelton provided  his explanation for why JANET Talk didn’t take off – the lack of positive network effects. Nick pointed out that as network grow “its usefulness increases in proportion to the number of potential connections between people in the network – the square of the number of people“. Nick felt that JANET Talk’s failure was inevitable as it “was only for people in UK HE to talk to others in UK HE“.

Although Nick’s point specifically addressed telephone networks I feel his arguments are also applicable to social networks in general – an argument I made at the JISC Digitisation Conference back in July 2007 in a talk on “Globalisation Of Social Networks and Networked Services.

We are now beginning to appreciate the importance of the network effect in a range of application environments – saving bookmarks used to be a function of the user’s browser but now we are seeing advantages of social sharing services such as del.icio.us.

But this seems to be missing from the approaches which have been taken to support IT development activities. In a post about the JISC e-Framework, for example, Andy Powell  questions whether the e_framework is of “any value to anyone“.  In a response Wibert Kraan felt that we can’t “forget about [the e-Framework] and pretend it never happened” – rather there’s a need to “look at what went well and why and what went wrong and why“. And this is equally true when considering the failure of open standards to live up to their expectations.

We need a better model for the adoption of open standards in our development activities since the current approach, which tends to assume that an open standard from a trusted and mature standards body will inevitably be accepted by the marketplace, is clearly flawed. And the network effect would appear to be a significant aspect in solutions which do become widely deployed and used.

Do We Want A Standards-based Voice/Video Service?

Last year JANET(UK) launched a trial of a voice, video and collaboration application called JANET Talk. As described in JANET News No.8 June 2009 (PDF format):

“The aims of the trial were to understand the precise requirements and service provisioning model for an ‘on net’, standards-based SIP service that could be used for communication between JANET users via a software PC client interface, mounted on the user’s PC or a SIP-based traditional phone handset“.

A survey of potential users also “showed a requirement for a feature rich collaboration tool for exclusive
use by JANET connected users that didn’t use peer-to-peer technology“.

Sounds good doesn’t it? A standards-based solution should avoid the problems caused by use of proprietary services and access would be available on both a PC and a phone handset which supported the SIP (Session Initiation Protocol) standard. Who, apart possibly Macintosh and Linux users who seem to have been excluded from the trial, would not wish this trial well (which attracted over 100 institutions) and look forward to deployment of the service across the JANET community?

However, as described in JANET News

“The results from both trial feedback and market research showed that the appetite for a service like JANET Talk had diminished. The reasons cited include a preference for alternative solutions that are now available from the commercial sector. These solutions were deemed easier to use, reliable and free.“

So now we know. Users don’t care about standards. Users care about solutions that work, are easy to use and, ideally, are free!

I know this is true for me, as I was an early adopter of Skype. At one stage use of Skype was frowned upon here at Bath University due to the load it could place on the campus network as well as the concerns about its proprietary nature, and the licensing conditions. However over time the local network team deployed solutions to manage the network load and we now seem to have happy Skype users, such as myself.

The University has also deployed a SIP solution which is available on SIP-compliant phones in various halls of residence. I must admit that when I heard about this offering I was interested. Was there a service based on open SIP standards which would enable me to talk to others without being constrained by a particular client? Sadly it seems that with the Freewire service used at Bath calls are free “when they’re made from one Freewire user to another” although you can “download the Freewire Telephone software for nothing“. But if you want to talk to someone on another service (Skype, for example) you’ll have to pay for the call :-(

So let’s remember, open standards don’t always succeed. And users may reject standards-based solutions in preference to other alternatives. There are risks in investing in open standards. And there should be lessons to be learnt from examples such as this. But I sometimes feel that we will ignore evidence which does not fit in with established dogma.

Standards are for Catholics

Being brought up in an Irish Catholic environment in the 1960s meant that life was full of religious and moral absolutes. If you were good you’d go to heaven (with some time in purgatory a likelihood) whereas protestants would go to hell. Black babies, who never had the opportunity for redemption, would go to limbo (it was only in 2006 limbo that limbo was abolished). And I can recall the Irish missionary priests who came to school collecting for the black babies – peer group pressure meant that the 12-sided 3d coin from your pocket money was the expected contribution. (The local catholic junior school, incidentally, hadn’t been rebuilt after being bombed in the war which meant we had the upstairs classroom in a protestant school – and we had staggered breaks so we wouldn’t mix. Little did we realise in the annual ‘Wessie Road’ upper vs ‘Wessie Road’ lower grudge football matches that the the over-the-top tackles were reflecting disagreements over the Virgin birth and Papal infallibility).

Now although I have already confessed to losing my religion the Jesuits may well have been right in their views on the power of indoctrination in early years. So although I no longer believe that I must not eat meat on Fridays, I am aware of the meaning and power of the word must and can differentiate it from should.

Such an understanding is very relevant in the works of standards. If a programming language requires statements to be terminated with a “;” then you must do so, otherwise your progam with fail (or, as is often said these days, FAIL). It’s not a fuzzy choice – it works or it doesn’t. Period.

But it seems that the meaning of must is slowly being lost. This first struck me several years ago when UKOLN was involved in the development of the standards and guidelines which support the national NOF-digitise programme. We were told that the document should state that “All Web sites must be available 24×7″ (or words to that effect). Our protestations were ignored – until projects reported that responses to the invitation to tender were rather over budget (to put it mildly).  We then described that 24*7 availability requires duplication of servers, backup networking capacity, backup power supplies, etc. and was only likely to be required by international organisations. It subsequently turn out that the requirement was that servers should not be turned off at 5 pm on Friday evenings, as had been the case in some circumstances in the past. The document was updated with the mandatory requirement being replaced by “Projects should seek to provide maximum availability of their project Web site” – as there was a contractual requirement to implement all of the ‘musts’ in the document this was needed in order to safe the entire NOF-digi budget being used to ensure 24×7 access for a single project!

Now I recently asked the question Is The UK Government Being Too Strict? as it similarly seemed to be requiring a must in circumstances in which the evidence suggests that such strict conformance very seldom occurs.

Is this just me and my background, I wonder?  When I see the word must in a standard, I think it really means must – otherwise you’ll be dammed forever in a non-interoperable hell.

But maybe I should chill out a bit? Maybe when I read must I should think of the kind friendly maths teacher I had at school who told me I should try harder, but he knew that it was sometime difficult, so he wasn’t too concerned if I gort it wrong. After all, I’ll probably find it easier in the future.

So tell me, are there policy makers and authors of standards and specifications who really do feel that must means must, whereas the developers interpret must as should? Is the problem that we have a non-interoperable mix of religions involved?

Are You Able?

There were two invited keynote speakers who travelled from Europe to speak at the OzeWAI 2009 conference. As well as my talk (which I described recently ) Dr. Eva M. Méndez (an Associate Professor in the Library and Information Science Department at the Universidad Carlos III de Madrid and not the American actor!) gave a talk entitled “I say accessibility when I want to say availability: misunderstandings of the accessibility in the other part of the world (EU and Spain)“.

Eva’s research focuses on metadata and web standards, digital information systems and services, accessibility and Semantic Web. She has also served as an independent expert in the evaluation and review of European projects since 2006, both for the eContentPlus program and the ICT (Information and Communication Technologies) program and her talk was informed by her knowledge of the inner working of such development programmes funded by the EU.

Her talk explored the ways in which well-meaning policies may be agreed with the EU, although such policies may be misinterpreted or misunderstand and fail to be implemented, even by the EU itself.

I don’t have access to Eva’s slides, so I will give my own interpretation of Eva’s talk.

We might expect the EU to support the development of a networked environment across EU countries across a range of areas. These areas might include:

Available: Have resources been digitised? Are they available via the Web?

Reusable: Are the resources available for use by others?  Or they it trapped within a Web environment which makes reuse by others difficult?

Findable: Can the resources be easily found? Have SEO techniques been applied to allow the resource to be indexed by search engines such Google?

Exploitable: Are the resources available for others to reuse through, for example, use of Creative Commons licences?

Usable: Are the resources available in a usable environment?

Accessible: Are the resources accessible to people with disabilities?

Preservable: Can the resources be preserved for use by future generations?

Since the acronym ARFEUAP isn’t particularly memorable (and ARE-U-API would be too contrived) we might describe this as the Able approach to digitisation. But there is 0ne additional concept which I feel also needs to be included:

Feasible: Are the policies which are proposed (or perhaps mandated) feasible (or achievable)? We might ask are they actually possible (can we make all resources universally accessible to all?)  and can they be achieved with available budgets and with the standards and technologies which are currently available?

There is, of course, a question which tends to be forgotten question: is the proposed service of interest to people and will it be used?

The worrying aspect of Eva’s talk was that the EU don’t appear to be asking such questions – or even used the same vocabulary.  We need to have the bigger picture in order to address tensions between these different areas and the question (and power struggles) of how we prioritise achieving best practices – for example, should we be digitizing resources, even if we can’t make them accessible; should we regard access by people with disabilities as being of  importance than ensuring the resources can be preserved?  And let’s not fudge the issue by suggested that each is equally important and all can be achieved by use of open standards. That simply isn’t the case – and if you doubt this, ask managers of institutional repositories. They will probably say that they are addressing the available, reusable, findable, preservable and, perhaps, exploitable issues, but I suspect that the repository managers would probably admit that many of the PDFs in the repositories will not be accessible.

“Standards Are Like Sausages”

“Standards are like sausages” suggested Charles McCathieNevile at the OzeWAI 2009 conference. “I like sausages” he went on to say “but I’m not keen on exploring too closely how they’re made“.

This was a wonderful metaphor which appealed to several Twitterers at the conference, including scenariogirl and RuthEllison.

A quick Google suggests the origin of this saying is “Laws are like sausages, it is better not to see them being made” by Otto von Bismarck (although this origin is disputed) with the Healthcare Standards blog applying it to standards-making in a post on The Making of Standards and Sausages published in August 2008.

Paul Downey, an advocate of Web Architecture at BT and formerly BT’s Chief Web Services Architect, chair of the W3C XML Schema Patterns for Databinding Working Group and BT representative at various organisations including OASIS and the WS-I, may has some sympathy with this view judging by the title of his talk at the QCon conference  ”Standards are Great, but Standardisation is a Really Bad Idea“. The abstract for this talk is worth quoiting in full:

Standards arise from consensus between competitors signaling maturity in a marketplace. A good standard can ensure interoperability and assist portability, allowing the switching of suppliers. A widely adopted standard can create new markets, and impose useful constraints which in turn foster good design and innovation. Standards are great, and as the old joke goes, that’s why we have so many of them!

If standards represent peace, then formal standardisation can be war! Dark, political, expensive and exclusive games played out between large vendors often behind closed doors. There are better ways to forge consensus and build agreements and the notion of a committee taking often a number of years to writing a specification, especially in the absence of implementation experience appears archaic in today’s world of Agile methods, test driven development, open source, Wikis and other Web base collaborations.

This talk will draw upon Paul’s personal experiences forged in the wonderful world of XML and Web service standardisation, examine the risks of premature standardisation, unnatural constraints, partial implementations and open extensions, puzzle how to avoid cloud computing lock-in, and contrast formal activities with lightweight open processes as exemplified by open source, Microformats, OpenID, OAuth and other Web conventions being ratified through open, lightweight, continuous agreement.

Now I’ve heard it suggested that in order to avoid choosing the wrong standard, you simply need to look at the worthiness of the organisation which produced the standard, perhaps on the assumption that a reputable standards-making organisation is like an approve sausage-making company. But as Paul Downey suggests, and Keith Boone seems to confirm in his post on the Healthcare blog,  the unsavoury standardisation processes take place in an organisation responsible for delivering globally-accepted standards such as HTML, CSS and XML.

Selecting the standards that will not only work as specified but will be widely accepted and supported in the marketplace is not an easy task.  And it is good to see that evidence of such concerns is now becoming more widely available.

Is The UK Government Being Too Strict?

I recently noticed a blog post published on the home page of the WASP (Web Standards Project) Web site.  The blog post, UK government browser guidelines: good sense prevails by Bruce Lawson, Opera applauded the UK Government for responding to pressure from the Web standards and Web development communities on its guidelines aimed at providers of UK Government Web services. The document initially stated that

… webmasters need not test in less popular browsers (those with less than 2% in that site’s usage statistics) and that there should be a page on the site listing the popular browsers which had been tested with the message “We advise you to upgrade your browser version as far as your computer allows and if possible to one of those listed above”.

Following over 400 emails made in response to a plea from Bruce, Adam Batenin, author of the document,  published a revised browser testing guidelines, and, according to Bruce “he’s done a great job of including best-practice development.” I too welcome that change.

However the guidelines also state (paragraphs 21-23) that:

All (X)HTML content must validate with respect to your chosen DTD.

Now although I’d agree with Bruce in his comments on the ”importance of valid code” I feel that a formal requirement that all (X)HTML content must validate with the appropriate DTD will be counter-productive.

We should recognise that the vast majority of HTML content does not comply with HTML standards – and it will be difficult for one sector to deviate substantially from the norm. This situation is likely to be made worse as use of embedded Web 2.0 technologies grows (e.g. YouTube videos of the Prime Minister embedded in UK Government pages) as embedding these services  typical causes HTML validation problems.

Now such problems are (primarily) the responsibility of the third party Web 2.0 providers. And here we should be lobbying them to ensure that code to embed their content does not break HTML standards. But they might argue that, as global services, they need to be very conservative in making changes to services which work, even if they don’t necessarily comply to published HTML DTDs.  The companies could argue that they are being user-focussed in such considerations, as isn’t there some truth in this?  I can recall one hard-line ’standardista’ who, on being told that a (University-developed) service didn’t render correctly in Internet Explorer, was told that the user should upgrade to a standards-compliant browser. And of course the university’s provided browser, was Internet Explorer! Such indulgences may occur in the public sector, but a commercial company which behaved likewise would soon find itself out of business.

As well as concerns that a formal requirement that UK Government Web pages must be fully HTML compliant may mean that pages aren’t rendered by the (flawed) browsers which people use, there is also a danger that this requirement will stifle developments and innovation in Government.

HTML itself has, sadly, proven a difficult language to evolve over time.  We are now in a position in which the usability and accessibility benefits which sensible use of AJAX technologies can provide and being made accessible to, for example, screen-reading software and assistive technologies through a standard known as ARIA. However use of WAI-ARIA (to use its official name) will normally mean that strict HTML compliance will not be possible. And when I’ve raised this issue with people involved in development of the standards and assistive technologies the response I have consistently received is that accessibility benefits which can be provided shown be prioritised over strict HTML. And this view has been endorsed in WCAG 2.0, which has dropped WCAG 1.0’s formal requirement for HTML compliance, requiring, instead, that markup elemnts are currectly opened and closed.

I would therefore suggest that the guidelines document should state that:

(X)HTML content should validate with respect to your chosen DTD.

After all, if the Web Standards Project Web site isn’t able to fully comply with the standards, should we expect every government Web site to?

And let’s also remember that these requirements only apply to (X)HTML content. If these requirements are too difficult to achive, won’t we see content being trapped in PDFs? You might, for example, like to think that the Digital Britain – Interim Report would be available as a HTML resource, but no, it’s only available in PDF and MS Word formats. But at least the such PDF documents won’t fail the government guidelines I’ve described. Let’s not pretend that mandating conformance with HTMLK guideines will result in better HTML documents. I’m convinced that it won’t – it will result in documents being provided in formats such as PDFs. And who bothers checking that PDFs conform with PDF standards?

Are Your PDFs Conformant?

I’ve never been much of a fan of the PDF format. Back in the early days of the Web I had hoped that the proprietary PDF format would be replaced by HTML and CSS. Back then there was an expectation that CSS would be developed to provide the fine control over page layout that is available using word processing and DTP applications.  The development of the Document Object Model (DOM) for HTML/XML various also promised to deliver an environment in which such resources could be interrogated and manipulated in ways which would not be possible with more monolithic resources such as PDFs. And finally HTML and CSS provided accessibility benefits not available in PDF.

However over the years it became apparent that HTML/CSS wouldn’t provide such fine layout control. And we found that HTML as used in the real world tended to be a structural mess, sometimes referred to as ‘tag soup’.

We also discovered that in many cases users preferred PDFs, especially for resources which designed as printed documents.

And last year PDF became an ISO standard, following on from the standardisation of PDF/A as an archival format.

So PDF is now an open standard, is suitable for archival purposes, has widespread support, accessible PDFs can now be created – and there is also an Adobe SDK which supports the development of applications to create and process PDF files.

Sounds good, doesn’t it? But in practice, do PDF files actually conform to the PDF standard? And although PDF files can be accessible, in practice do the PDF files which are produced in normal work flow processes  actually comply with accessible PDF guidelines?

I recently searched for PDF validation tools.  I found that a number of tools were available, many of which were expensive to purchase. I made use of one free email-based tools (Validatepdfa) and used it to report on the conformance of a couple of PDF files for recent peer-reviewed papers which I had submitted to journal / conference organisers. Although these files may have conformed with the publisher’s layout and house style requirements, I found the tool found quite a number of error As you can see the error messages aren’t particualrly helpful and it is difficult to see how such errors can be remedied:

Issues addressed (1) File structure Incorrect delimiter used for indirect object 340 0
Issues addressed (2) File structure Incorrect delimiter used for indirect object 370 0
Issues addressed (3) File structure Missing ID in trailer dictionary
…
Issues addressed (118) Fonts Font ‘TrebuchetMS-Bold’ was successfully substituted and embedded
Issues addressed (119) Fonts CID font subset without CIDSet
Issues addressed (120) Fonts CIDToGIDMap has been successfully embedded in Type2 font LHCKAJ+SymbolMT.
Issues addressed (121) Fonts CID font subset without CIDSet

I then used the Adobe Acrobat software to report on any accessibility problems with the papers. I used this tool to analyse all of my peer-reviewed papers which I have written in the past 10 years – and found that none of the papers actually conformed with Adobe’s accessibility guidelines.

The error messages provided in Adobe Acrobat were mostly helpful and it seemed that one consistent problem was the lack of a language to describe the contents of the document. Fortunately Adobe Acrobat does allow some of the accessibility problems to be fixed with the software – so I assigned the language English to all of the documents. Some of my papers now do conform with PDF accessibility guidelines (at least as far as automated checking tools can detect) – but the documents which had been uploaded to the University of Bath’s institutional repository a few months ago will be the non-accessible versions. There are issue about the workflow processes for uploading papers to institutional repositories: who should have a responsibility for ensuring compliance with guidelines; at what stage should appropriate metadata be added; who should ensure that the metadata is correct; what tools can be used to create and maintain such metadata; what level of detail should be provided; how do we ensure that the metadata isn’t corrupted during workflow processes; etc. Did you really think that using PDF was easy?

I suspect that most people aren’t particularly interested in conformance of such resources with PDF standards and accessibility guidelines – although it was reassuring to see the post on”Survey on malformed PDFs?” on the DCC blog.

But if we are serious about the importance of standards, particularly in the context of digital preservation, and if we are serious about the accessibility of digital resources, we will need to ensure that our workflow practices result in resources on our Web sites and institutional repositories which are conformant.

Or perhaps strict conformance with standards and accessibility guidelines is over-rated. Should we simply acknowledge that the ease of creation of PDF resources is key to the creation of such resources and adding additional steps into the workflow processes will add unnecessary complexities and barriers?

Thoughts On Erik Duval’s Post On Standards

In a post entitled “Standards for Technology Enhanced Learning” Erik Duval gives his thoughts on the issues related to the standards which have a role to play in providing technology enhanced learning.

Erik feels that:

1. The main issue is no longer that we do not have sufficient standards. Rather, we have maybe too many and, more importantly, we don’t make use of them in very advanced ways… Tools are lacking or too much let the standard shine through, rather than focusing on the user experience.
2. We should avoid continuing the ‘not invented here’ approach that  has made us develop learning specific standards when there may be quite appropriate standards already out there or being developed.
3. Standards should not be research oriented but rely on proven practice. Of course, standards enable deployment at large scale, and  therefor make it possible to do research on global infrastructures.
4. Standards enable openness, and that enables innovation – that is another way for standards to be relevant to research.

I would not only agree with Erik’s comments, but suggest that they are relevant beyond the e-learning environment.

Erik’s comment that “Standards should not be research oriented but rely on proven practice” does, I feel, need to be reflected upon by the research community and by organisations such as W3C.  I’ve comment previously on the failure of W3C standards to have any significant impact, and I feel that this is due to a failure to take into account practical issues in preference to developing innovative or elegant solutions.  And I feel that there may be problems with funding streams which seek to encourage the development of new standards (which will, of course, promise a whole set of rich possibilities) at the expense of encouraging greater uptake of standards which are already available (and failing to exploit the rich possibilities which bright them abpout in the first place).

Erik’s suggestion that there’s a need to “rely on proven practice” does, to me, emphasise the need to engage with the software developer community. In the past recommendations of standards had been taken by policy-makers, often with little involvement of those engaging in using such standards. But now, I feel, this is beginning to change.  And I’m particularly pleased to see that JISC are sponsoring Developer Happiness Days in February 2009.  I hope we will see more of such events – and that these will provide an opportunity to share proven practice. And if the proof demonstrates that standards don’t work or are too complex to use, that will have been valuable in itself.

And finally when Erik “We should avoid continuing the ‘not invented here’ approach” I would suggest that we need to ensure that standards evolve slowly, with only minor fixes – and the fifth edition of the Extensible Markup Language (XML) 1.0 provides a good example of this. It’s good that we aren’t at XML 5.0, with a new generation of tools needing to be developed to support each new version.

W3C’s Financial Difficulties Affects Their Validators

Molly Holzschlag recently wrote a blog post entitled W3C Validators in Jeopardy in which she pointed out that “As many folks who follow the W3C are aware, financial and bureaucratic issues have challenged the organization for many years“. Molly went on to describe how “It’s come to pass that the funding necessary to maintain and grow validation services at the W3C has become overwhelming to the W3C’s operational budget. As such, the validators are in jeopardy.“

A donation system has been set up which is described on the W3C Validator Donation Program page.

As Molly says “we’ve had the use of validation tools via the W3C for so long and without cost has been a significant component in the teaching and evangelism surrounding Web standards and best practices“.

But who will have the resources to support this request? And if funding for the validators is uncertain, what next? Is W3C in a position in which the long term sustainability of its standards can  be guaranteed? And didn’t we feel that open standards brought about freedom from the uncertainties  of commercial pressures? It’s time for the risk assessment of standards organisations, I feel, and not just the providers of networked services.

Why Did SMIL and SVG Fail?

Following my blog post on Open Standards and the JISC IE which I wrote back in September Stephen Downes responded with some comments which I include below:

“In retrospect many of the W3C standards which I had felt should form the basis of the JISC IE have clearly failed to have any significant impact in the market place – compare, for example, the success of Macromedia’s Flash (SWF) format with the niche role that W3C’s SMIL format has.” Just so. But these standards didn’t fail because they were open. They failed because, for various reasons, they didn’t do what people wanted. Open standards are still better – but the lesson here is that standards are not necessarily better just because they’re open.

Absolutely, the standards didn’t fail because they were open. The point I was making in my post was that the openness of a standard is no guarantee that it will be successful.  And it is important to remember this to avoid policy makers mandating open standards which in reality may fail to have any significant impact.

But why do open standards, such as SMIL and SVG, fail? Stephen suggests they failed “because, for various reasons, they didn’t do what people wanted“.  There may be something in this, but I feel there are other potential reasons why standards may fail, which I’ve listed below.

Failure to promote the standards: A standards body may fail to promote the benefits of its standards to the user community or to potential vendors.  I don’t think this is the case for SMIL and SVG as W3C is very good at promoting its technical developments.

Standards are not accessible:  In an environment in which the accessibility of digital resources is becoming important in the selection of formats by user organisations, especially in the public sector, there may be reluctance to make use of standards which are not felt to be accessible. This is definitely not the case for SMIL and SVG, which have been developed with the needs of users with disabilities being addressed right from the start.

Failure to get vendor buy-in: Potential software vendors, such as Microsoft, Macromedia, Adobe, etc. are W3C members and have been actively involved in the development of these standards.

Failure by vendors to promote: Tim Berners-Lee, in a post entitled “MS IE “slow in supporting SVG” pointed out that “If you look around at browsers, you’ll find that most of them support scalable vector graphics,” Berners-Lee said. “I’ll let you figure out which one has been slow in supporting SVG.“  The lack of SVG is all Microsoft’s fault, you may feel.  However an article on “SMIL Standards and Microsoft Internet Explorer 6, 7, and 8” touches on some of the complexities of vendor support for rapidly developing standards. As described in this article other vendors have their doubts regarding the the effectiveness of W3C standards such as SMIL, with the Macromedia Product Manager stating that Macromedia “[doesn't] feel that SMIL integrates well with HTML and the current evolution of the DOM, SMIL is a decent standard for synchronizing audio and video, but isn’t really a multimedia standard. And it does not enable an author to create a rich, interactive multimedia presentation with any kind of sophistication.”“

Lack of interest by the users: And perhaps Stephen Downes is correct when he says that such standards don’t do what people want.  Do we have real evidence that there is sufficient interest in these standards for the market place to support the standards?

Insufficient motivation to change existing working practices: Even if there is evidence that there is a marketplace for SMIL and SVG are the benefits sufficient for users to be willing to change their existing approaches, purchase new tools, training staff, etc.

I think it is clear that W3C have failed to deliver a solution which is being widely deployed.  Now this may not be of concern to W3C – they may regard their role as simply developing standards and are happy to leave it to the marketplace to adopt or reject the standards. However as user organisations we can’t take this stance.  So we will need to ensure that we have learnt form the failures of well-promoted standards to have any significant impact. Or perhaps we should simply be prepared to wait for a longer period for new standards to gain impact.  Perhaps we may find greater take-up of SMIL and SVG, with the mobile market providing the arena for the standards to demonstrate their worth.

Or have I got this wrong and will I find a horde of happy SMIL and SVG users commenting on this post with examples of how they are successfully using the standards?

Killed By Complexity

“If this is the death of Wall Street as we know it, the tombstone will read: killed by complexity” it was suggested on the front page of the Guardian today (Tuesday 16 September 2008).  A similar question might be asked about the roadmap for a number of Web developments.  Is Tim Berners-Lee’s vision for the Semantic Web over-complex?  Are the metadata standards which are being developed too complex to be used by many software developers? The abstract for a panel session at WWW 2005 suggested that “It has been estimated that all of the Web Services specifications and proposals (“WS-*”) weigh in at several thousand pages by now”. And one of the many objections to ISO’s decision to standardise the OOXML file format was that, at 6,000 pages, it was too complex for developers in small organisations to implement.

So now’s the time for more lightweight approaches, it could be argued.

Not so, comes the counter-argument. We will need to have comprehensive, well-grounded and unambiguous standards and specifications in order to build robust services.

The current uncertainties in the financial markets of course provide more than just a analogy  - they are also giving rise to uncertainties in the IT sector.  This is often used as an argument to point out dangers of the dependencies on externally-hosted Web 2.0 services, as my colleague Paul Walk pointed out recently. But as I mentioned last year in a post entitled “Universities, Not Facebook, May Be Facing Collapse“, universities themselves are not immune to the financial difficulties which the banks and airline sectors are currently facing.   

But into such discussions we should also add the financial stability of the standards-making organisations. Organisations which have government backing may be able to weather the storm, but what about those member consortiums whose sustainability is dependent on the financial backing of the commercial sector. And as the W3C is one such organisation, can we be confident that the development and maintenance of complex standards will be sustainable in the long run.  In light of suggestion in a recent interview with Ian Hickson, editor of the HTML 5 standard, that the standard is unlikely to be a “Proposed Recommendation in 2022″, should we not now be asking the difficult questions regarding the sustainability of such standards which seem to have a long gestation period before they can be regarded as stable.  

Or am I being unduly pessimistic?  Might not any current financial uncertainties be a mere blip, and perhaps will not affect standardisation development processes along the lines I’ve hinted at? Or will a legacy of George W Bush’s economic mis-management (or Tony Blair’s if you are of a different political hue) be the failure of the HTML 5 standard to achieve its proposed recommendation status by 2022?

On the Videos from the Repository Fringe 2008

On the The thoughts of a Code Gorilla blog a post on Videos from Repository Fringe 2008 provides a link to a number of videos of talks given at the Repository Fringe held recently in Edinburgh. The blog is written by a software developed who has been “identified as “a free thinker” by JISC“.

The post states that the videos “will be made available via a Streaming Server at some point, however this is a microsoft-specific platform, so non-windows/non-Internet Explorer users struggle to access the data“. In order to maximise the access to the videos Code Gorilla has “uploaded them to google video“.

As I mentioned in post on the Open Standards and the JISC IErecently at one stage there was a fairly hard line view that open standards must be adopted in order to provide device independence – in the case of multimedia, W3C’s SMIL standard wold seem to be particularly relevant for synching audio, video and other resources such as presentation files. However as we see in this example, the vision that we had several years ago has failed to have any significant impact, as instead it is the popular services such as Google Video and YouTube which are being used to deliver such resources, as well as providing additional functionality, such as user comments and the ability to embedded the resources in other pages, as illustrated below.

It is also interesting to note that this also provides a good example of a pragmatic approach to the accessibility of such resources. At one stage, when the SENDA legislation was being deployed, there was a feeling in some circles that institutions would need to remove videos from their services unless they could provide full captioning. We now, however, widely accept the view that we need to take ‘reasonable measures’ to provide accessible alternatives – and that removing resources does not improve their accessibility.

So my congratulations to the ‘free thinker’ who has so clearly demonstrated that the naive views that we used to have can, in circumstances such as this, be ignored in order to maximise benefits to the user and provide cost-effective solutions.

It is appropriate to embed this video of Dorothea Salo’s keynote talk at the Repository Fringe 2008, with her comments that “idealism isn’t enough” and “programmers are moving towards flexibility”.

And finally I should add that at the end of this video clip (45 minutes in) Dorothea mentions the impacts that both Paul Walk and Andy Powell are having in questioning some of the assumptions which have been made in the past regarding the technical approaches taken to institutional repositories.

We do need more ‘free thinkers’, I feel.

The George Bush and Microsoft Parallels

Back in May 2008 I published a blog post entitled George Bush IS President And Microsoft’s Office Open XML Format IS An ISO Standard which described how Microsoft’s Open Office XML (OOXML) had been approved as an ISO standard. However in the period between first writing the post and then publishing it South Africa, Brazil, India and Venezuela lodged appeals against this decision claiming that the voting process was marred by irregularities. So until ISO had addressed these appeals we could say that OOXML was not an ISO standard. However as described in an article on OOXML Gets Final Nod After Standards Body Rejects Appeals ISO has now has formally rejected these appeal.

The analogy I drew with George W Bush was even more appropriate than I had anticipated – just as there were doubts over the legitimacy of Bush’s first election victory which were eventually rejected, so the appeals against the legitimacy over the standardisation of OOXML have been rejected, with OOXML now becoming an official ISO standard. I suspect many readers of this blog would have preferred it if neither of these decisions had happened, but they have.

Whether this is the end of the matter is not yet clear: a article on CONSEGI 2008 Declaration — Open Letter to ISO Reveals More OOXML Issues published on the Grocklaw site describes how South Africa, Brazil, Venezuela, Ecuador, Paraguay and Cuba have signed and sent an open letter to ISO condemning this decision. Further information about the standardisation process is available in a Wikipedia article on Standardization of Office Open XML.

But although the standardisation process may have been flawed with, no doubt, political skullduggery going on, the technical merits of the standards questionable and the likelihood that the standard will actually be implemented by vendors questioned by some, we now, I would say, have to accept that it is an ISO standard. But as I’ve argued for other reasons recently, we should in any case be questioning the significance and merits of open standards much more questioningly than we have done in the past, when slogans such as ‘interoperability though open standards’ seem to have been used to stifle discussions and debates on the extent to which open standards actually deliver their stated goals.

It was also pleasing  to read Ross Gardler, manager of the JISC OSS Watch service’s comment on my recent post in which suggests that “it is possible to diverge from [open] standards without enforcing locking. This is a huge advantage when it takes so long for standards to be specified and agreed by committees and standards bodies” – he could, of course, have added caveats regarding the political nature of  standardisation processes.

I therefore welcome Ross’s statement that “OSS Watch would be happy to explore these ideas further. Just what are the advantages and disadvantages of formalised standards against open implementations of data formats?” And over the next few weeks I will publish a number of posts in which I’ll invite discussions on  standards issues.

For this post, however, I’d welcome comments specifically on the OOXML standardisation process and the implications of ISO’s decision.  My view is that it’s a good thing when proprietary formats become standardised (as has also happened recently with the standardisation of Adobe’s PDF format which was announced on 2 July 2008) as this can be beneficial for, for example, long term preservation. However this doesn’t necessarily mean that the format will be appropriate in many circumstances – we do need to decouple the view that because an open standard is available in a particular area that it should necessarily be deployed, I feel.

Open Standards and the JISC IE

The Ariadne article on Lost in the JISC Information Environment has generated some interesting discussions, including my colleague Paul Walk’s post in which he suggests that all models are wrong, but some are useful and Andy’ Powell’s post entitled Lost in the JISC Information Environment?.

I’ll leave the discussions on the technical architecture to others, but thought I’d pick up on Andy’s comment that:

.. the technical standards certainly were intended to be prescriptive.  I can remember discussions in UKOLN about the relative merits of such an approach vs. a more ad hoc, open-ended and experimental one but I argued at the time that we wouldn’t build a coherent environment if we just let people do whatever the hell they wanted.  Maybe I was wrong?

Myself, Andy, Pete Johnston and Paul Miller were the ones who had those long discussions about the role of open standards and the JISC Information Environment (IE).  I was the person, who had been introduced to standards through my involvement with the Web from its early days, who was the most adamant on the need to use open standards, where open meant the standard had been ratified by a trusted neutral standards organisation, such as the W3C. I was therefore never in favour of standards and protocols which weren’t open in this sense, including Adobe’s PDF or Sun’s Java. On the other hand, I was always fairly relaxed about the technologies used to implement the services, not being too concerned if licensed software was felt to provide advantages over open source alternatives, for example.

It was Paul Miller who suggested than my stance on open standards was too inflexible, suggesting that there was a spectrum to openness, rather than a fixed binary divide. As a result of Paul’s comments and subsequent discussions in UKOLN I wrote a briefing document which suggested that rather than seeking a formal definition of open standards, we needed a more flexible approach based on an understanding of the characteristics of open standards.  And the need for such flexibility became even more apparent when the success of RSS had to be balanced against the lack of formal standardisation of RSS (both 1.0 and 2.0).

And in retrospect many of the W3C standards which I had felt should form the basis of the JISC IE have clearly failed to have any significant impact in the market place – compare, for example, the success of Macromedia’s Flash (SWF) format with the niche role that W3C’s SMIL format has.

Just as the open source debate seems to have matured (and I think that the JISC OSS Watch service has helped to move that debate from the polarised opinions we were seeing several years ago) we still need, I feel, to have a much more sophisticated understanding of the role open standards have to play in development activities. And, as with the decisions institutions (and individuals) have to make regarding their use of externally-hosted Web 2.0 services, so funders, developers and project managers will need to give more thought to the risks as well as the promised benefits of use of open standards.

I’ve written, in conjunction with staff from CETIS, OSS Watch and the AHDS, a number of peer-reviewed papers on this topics ( Openness in Higher Education: Open Source, Open Standards, Open Access, Addressing The Limitations Of Open Standards, A Contextual Framework For Standards, A Standards Framework For Digital Library Programmes and Ideology Or Pragmatism? Open Standards And Cultural Heritage Web Sites). I suspect it is time to revisit this topic.

When W3C Web Pages Break

I was looking at a page on the W3C Web site recently to update my knowledge of the SVG specification and SVG tools.  I noticed a link at the bottom of the Scalable Vector Graphics (SVG) page to an RSS feed for the page, and, as a fan of RSS syndication, thought it might be worth adding this feed to my RSS viewer. However when I clicked on the link, rather than seeing the RSS feed and having the option to add this to my preferred RSS reader, an error message was displayed:

Now validating this RSS feed with the RSS validator on the W3C Web site informs me of an error with the feed:

Sorry

This feed does not validate.

• line 227, column 87: Undefined named entity: reg (5 occurrences) [help]
  

  ... ability as well as the Internet Explorer® Plugin and the Windows® ...

This feed does not validate.

It seems that either W3C’s workflow process has failed to removed the registered trademark character for the term “Internet Explorer®” or the RSS schema has failed to included a declaration for this character entity.

No big deal, you may think – and, as the page isdisplayed in the FireFox browser, this is surely another failure of Internet Explorer to follow Web standards.

But if you view the page in Opera you get an XML parser error message:

And here, I think, both Internet Explorer and Opera seem to be obeying the requirement that user agents aren’t expected to render non-compliant pages.

And this hard line approach has been promoted as a vision of the future of the Web by the W3C.  It has been argued that mandating rigourous compliance with specs would help to maximise interoperabilty.

This may be true – but at  what cost.  As someone who studied engineering at University I am aware of the benefits of a fail-safe approach to design, so that if one small component fails it doesn’t mean that the building will collapse. But in this case one small component (the trademark character entity) which hasn’t been properly defined, has led to a total failure for the page to be rendered in two browsers.

Don’t we need Web resources to be designed so they’ll fail gracefully and will be tolerant if humans make mistakes or, as it seems is the case here, there are failures in the workflow?

whois++ and IAFA templates

SCA Home Nations Forum

I recently facilitated a series of breakout sessions on Standards at the SCA Home Nations Forums, held in Edinburgh, Belfast and Cardiff. The aim of the sessions was to discuss the approaches which are being taken to the use of standards by SCA partners in Scotland, Northern Ireland and Wales.

The first event included a plenary talk on “The Standards Dilemma” given by Alastair Dunning, JISC, and I’ve embedded his slides in my blog post.

Alistair’s blog post about the first event, entitled “Digital Standards: Going beyond Stalin“, summarised some of the difficulties which have been experienced in seeking to deploy open standards in digital library development work.

eLib Standards Document

These concerns were reflected in the breakout sessions at the three events. And when I was preparing the breakout session I though it would be useful to review my involvement in standards work, which date back to my contribution to the eLib Standards document, published in February 1996.

In that document I was fascinated to discover some of the open standards which we thought would lead to interoperability for eLib projects. The document mentioned the Open Document Architecture (ODA) standard but went on to (correctly) predict that “It is unclear what future there is for the ODA standard” and stated that “It is not recommended for use in the eLib programme“.

Rather than using ODA, the standards document “anticipated that SGML will be a key standard for eLib“. The document “encouraged [projects] to work together to agree or, where necessary, develop document type definitions“. Although SGML was used by a number of projects (such as, I think, project which used the TEI DTD) SGML did not have a significant role to play for many of the eLib projects until a simplified version of SGML, XML, became available. The exception to that generalisation was HTML. My contribution to the eLib standards document was to write: “Hypertext Mark-up Language (HTML) is simply a DTD which prescribes formats for presentation and display. Hypertext documents in the World Wide Web are written in HTML. eLib projects will make heavy use of HTML and should use HTML 2 and HTML 3 when it is stable. Netscape and other vendor-specific extensions are deprecated.“

It was in the area of standards identifiers, metadata and searching in which the recommendations are most interesting. The document (correctly) stated that “eLib projects should be able to supply a URL for public services” – although in retrospect we should have said “a static and stable URL”. But the above sentence then went on to say the “… and be prepared to adopt URNs when they are stabilised“. The URN (Uniform Resource Name) was envisaged as “a persistent object identifier, assigned by a ‘publisher’ or some authorising agent“. Now today, 12 years later, project Web sites still have a URL for their resources, with other approaches to identifiers (such as DOIs) only being used in specialised areas, such as providing identifiers for journal articles or, in projects such as E-Bank, molecules.

Regarding metadata standards, the document stated:

Relevant standards for resource description:US-MARC, IAFA, TEI headers

although it immediately added the caveat that “This is an area in which there is still much research and development and where it is premature to suggest one preferred approach“.

The document also suggested that the WHOIS++ cross-search protocol could have an important role to play for searching metadata held in the IAFA templates. Indeed the e-Lib-funded ROADS open source software, which underpinned several of the eLib Subject-Based Information Gateways (such as SOSIG and OMNI), was based on this approach.

Discussion

I feel there is much which can be learnt by reviewing the experiences of digital library programmes such as eLib – indeed eLib projects were themselves expected to be open in reviewing their experiences, both positive and negative. Looking at the standards document with the benefit of 12 years of hindsight we can smile at its naivety. But we should also ask why certain standards, which failed to gain acceptance, were encouraged in the first place? An answer, perhaps, is to be found in the interests of the contributors to the standards document. Anne Mumford (a former colleague of my when I worked at Loughborough University) was actively involved in the development of the CGM (Computer Graphics Metafile) standard, so it’s perhaps not surprising that this standard was included in the standards document.

What have we learnt since 1996? Do we ensure that we have more disinterested processes for recommendations? A recent Tweet from Owen Stephens, related to a TechWatch report on “Metadata for digital libraries: state of the art and future directions” suggested that this is not the case: “[I] was surprised how pro-METS [the report] was until I noted “Richard Gartner is [...] is a member of the editorial board for the METS“. Which current exciting new standard will turn out to be tomorrow’s whois++ I wonder?

George Bush IS President And Microsoft’s Office Open XML Format IS An ISO Standard

On 2^nd April 2008 the IT Week magazine described how “Microsoft’s Office Open XML document format standard has been approved as an ISO standard” in an article entitled “OOXML gets the nod as an ISO standard“.

Everyone who has been critical of Microsoft for continuing to promote its proprietary Office format should be pleased with this news, one might think. And indeed an editorial comment in the same issue of IT Week a piece entitled “Microsoft wins format standards” suggested that the “ISO vote endorsing OOXML ends vicious committee wrangling“. But the article admitted that the “decision means that there are now two ISO document standards“. And further “Supporters of the rival Open Document Format claimed OOXML is not truly open because it was not designed by an open process“. In addition they also suspect “Microsoft will find ways to retain control“.

Rowan WIlson on the JISC OSS Watch blog elaborated on these concerns: “the perception that OOXML is in itself an inadequate standard which has triumphed through Microsoft’s expertise at lobbying ISO member bodies for their votes“; “the standard is itself is incredibly long and complex – over six thousand pages” and “Microsoft’s patent non-enforcement promise that accompanies [the standard]“. Similar concerns are described in a Wikipedia entry on OOXML.

But do such criticisms mean that we should not make use of OOXML? I would say not. If you believe in open standards, then you should be prepared to accept standards which have been ratified by a formal standards body. Just as when George W Bush first became president, despite the concerns regarding the voting process and allegations of corruption in certain states, the Democratic party was prepared at accept this decision.

The criticism that “there are now two ISO document standards” misses the point that duplicated standards are not unusual, as the joke “the great thing about standards is that there are so many to choose” illustrates. Indeed, readers of this blog will probably be familiar with the RSS 1.0 and RSS 2.0 – not two versions of the same standard, but two different standards – RDF Site Summary and Really Simple Syndication Standard (to say nothing of its original name Rich Site Summary). The battles which have taken place over this popular syndication format seem to be typical of the standardisation process in the IT sector. So we should not be surprised to read of dissent in the document format area.

I suspect that a lot of criticism of the standard is really aimed at seeking to persuade organisations that they shouldn’t be using Microsoft Office products. But that, I feel, is a different argument. Rather I’ll leave the final comment to Richard Boulderstone, the chief technology officer at the British Library who has welcomed OOXML’s approval as an ISO standard, as the establishment of an open well-defined OOXML standards will ensure documents can be viewed through future applications: “We think hundreds of years in the future, by which point this standard won’t be supported anymore. But we’ll be able to create an application to views these documents as they’re based on an open format. Under the closed proprietary format previously used by Microsoft we couldn’t do that.“. Amen to that.

What Does Openness Mean To Your Community?

Myself, Mike Ellis (Eduserv) and Ross Gardler (JISC OSS Watch) are the co-authors of a paper on “What Does Openness Mean To The Museum Community?” which has been accepted for the Museums and the Web 2008 conference. And I’m pleased that David Bearman (conference co-chair) response when he read the paper was that it should be discussed in a Professional Forum at the conference. Indeed David’s comment on the paper was “it sounds like it could be the most amazing session at MW this year” :-)

The paper suggests that openness can include open standards, open source, open APIs, open access and an open culture (i.e. a willingess to encourage user-generated content). But the paper also acknowledges that there is a downside to each of these aspects. Some of these concerns were raised by Nick Poole, Chief Executive of the MDA in a thread on “The speculative aspect of using Web 2″ on the MCG JISCMail list. Nick commented:

… ‘how can you be so naïve’? Low cost of entry? We were promised that with Open Source Software and it turned out to be no cheaper. Reaching audiences while we sleep? They told us Z39.50 and interoperability would solve that and we’re still not there. Content Management will make everyone a publisher? You just try and get a username and password out of the Council IT Admin.

I’m pleased that Nick raised such concerns. He’s right when he suggests that the potential benefits of both open source and open standards have been over-hyped. And, similarly, the benefits of Web 2.0 can also be exaggerated. But my response to the concerns raised by Nick are to argue that we need to develop more sophisticated ways of engaging with these aspects of openness – and just because policy makers appear to feel that simply mandating use of open standards and open source software will be sufficient to deliver their benefits, doesn’t mean we are faced with the binary choice of accepting or rejecting such views. Rather we need to engage in discussions and debate on ways in which real benefits can be realised.

I’ve been involved in working collaboratively with others in developing models for exploiting the potential of open standards and open source software. At the Museums and the Web 2.007 conference I presented a paper on Addressing The Limitations Of Open Standards, co-authored with my colleague Marieke Guy and Alastair Dunning (then of AHDS). These ideas were further developed and extended to include open source and an open access in a paper on Openness in Higher Education: Open Source, Open Standards, Open Access co-authored by Scott Wilson (JISC CETIS) and Randy Metcalfe (then of JISC OSS Watch).

But there’s a need to build on these approaches and to develop approaches for exploiting other aspects of openness. And such approaches need to recognise the dangers and difficulties. But just because there are difficulties, doesn’t mean we should reject openness – rather it means we need to continue having the debate, whether it’s on mailing list such as the MCG list, on this blog or at the professional forum at the Museums and The Web 2008 conference. So I’ll ask here the questions w’ll be discussing in a few day’s time: what does openness mean to your community, what are the benefits it can provide, what are difficulties which are likely to be faced and, most importantly, how do you feel such difficulties should be overcome.

Your feedback is warmly welcomed.

Losing My Religion

I discovered the Web in December 1992 and, after Christmas, helped to set up the institutional Web site at the University of Leeds. Later that month I met Robert Cailliau, a colleague of Sir Tim Berners-Lee, when Robert was in Leeds visit relatives. Robert gave me the background to the developments of the Web and it was around that time I subscribed to the www-talk mailing list. This was the start of my belief in a Web based on standards developed by an open community.  And I can remember the controversy caused when NCSA, in their development of the Mosaic browser, broke with the consensus in the format of the IMG tag. Marc Andreessen made a proposal which generated debate. However Marc chose to ignore Tim’s suggestions:

Tim Berners-Lee writes:
> Let the IMG tag be INCLUDE and let it refer to an arbitrary document
> type. Or EMBED if INCLUDE sounds like a cpp include which people
> will expect to provide SGML source code to be parsed inline — not
> what was intended.
We’re not prepared to support INCLUDE/EMBED at this point; it raises a
number of nasty issues that are quite separate from the idea of
inlined images.

What happened was that Mosaic was released to universal acclaim. But later, when the lack of extensibility of the IMG tag became apparent, the Netscape browser was released and introduced a more effective way of embedding content other than images, using the EMBED tag. And Marc promoted supported support for this proprietary tag over the limited IMG tag as a killer feature of Netscape.  Similar tactics which Microsoft have been guilty of over the years.

It’s not just Microsoft, you have to be wary of software vendors in general as they all have vested interests in proprietary lock-in, has been my belief over the years. Stick with the W3C, I’ve felt. They are independent of vendors and will be best positioned to provide open standards which everyone can use, I’ve argued over the years.

But over time I’ve begun to question the wisdom of this view. I raised this issue last June in a post entitled “Are W3C Crazy?” in which I picked up on a comment made by Phil Wilson, a Web developer based at the University of Bath. Phil told me, based on his attendance at the XTech 2007 conference that:

There seemed to be a couple of big fat W3C elephants in the room.

The first was that the w3c are doing stuff for use in five or ten years’ time whereas most of the other talks are about things you can do today or next year, which makes them seem like futurologists.

The other is that they really didn’t seem that happy that HTML5 was going ahead, and what the hell was wrong with XHTML2 anyway?

It must be nice to work in a standards organisation where everything you do meets some Platonic Idea of perfection.

Are W3C working in a purist world in which everything needs to meet a Platonic idea of perfection? Others, including long standing Web standards evangelists, seem to be raising similar concerns. Molly (of Molly.com, a well-known author of dozens of books on Web standards) is the latest to raise her concerns. In a post on “From Web Standards Diva to Web Standards Devo“ she makes a startling suggestion:

I’m going to design my new site with frames, tables, spacer gifs, lots of flash embedded into framed pages via iframes. I’m going to use non-semantic, presentational HTML, table based layouts, and lots of inline CSS.

The frightening issue is that I can build such a site so it will validate, pass at least WCAG priority 1 accessibility and have effective SEO.

However she goes on to say:

The mere fact that I can actually do all that and be in compliance with specs should help clarify my point, I hope. It’s not the specs that define Web Standards. We are talking about best practices. We use the term “standards” fast and loose, and for an industry that is so interested in semantics, I find it endlessly ironic that we have chosen such a piss poor description to define a certain level of professional practices.

This post is a follow-up from one on “Web Standards Aren’t” which, as with many of Molly’s posts, succeeds in generating much debate, including contributions from some of the leading lights in Web standards development work.

I met Molly at the W4A 2005 conference when I gave a paper on “Forcing Standardization or Accommodating Diversity? A Framework for Applying the WCAG in the Real World“. This was my radical paper in which I suggested, to a room full of Web accessibility experts, that Web accessibility wasn’t about conforming with a technical set of universal standards, but in identifying best practices which would support users in the particular tasks they were engaged in. Molly, who I didn’t know at the time, supported various comments I made at the conference, which led to various late night drinking sessions at the conference (but I won’t go into that!)

And now Molly is taking the debate even further. and other leading standards-based developers are raising similar concerns, such as Andy Clark’s post dated 11 February 2008 on “transcending the web of today” in which he suggests:

Transcending is about moving away from outdated notions, for example that a design should look the same in all browsers. It is about designing the best possible visual experience for people using the best browsers (and then considering what happens for people using outdated technologies). This is the opposite of progressive enhancement where a designer would design for the most common, lowest common denominator browser (even it is the least capable), and then add extra visual decoration to reward people who use more modern software. Transcending about designing the best for the best.

If leading lights such as Molly and Andy (who have both published books on Web standards, given many prresentations on this topic and beern active in W3C working groups) are questioning the W3C vision, we should pay heed. Have W3C lost the authority they once had? Have the dangers posed by software vendors leading the development of standards simply been replaced by the dangers of a group of researchers and purists who are happy to develop  sophisticated solutions which may fail to gain acceptance in the marketplace?

It’s not longer just a question of passively accepting the vision of the standards developers, I’m afraid. And if you don’t believe me, tell me -do you think the future lies in W3C’s XHTML 2 standard (July 2006 draft) or W3C’s HTML 5 standard (hmm, latest draft came out on 11 February 2008)? If there’s a schism within W3C and W3C Consortium Members such as Microsoft, Sun, Opera and Google, which sect will you follow? Or do you feel the need to avoid the religious wars and join the agnostics?

TechWatch Report on XML-based Office Document Standards

A JISC TechWatch report on XML-based Office Document Standards (TSW0702) has just been published.  As described on the TechWatch Web site:

This TechWatch report explains these issues and some of the standards involved. It proposes that although the UK higher and further education sector has, for a long time, understood the interoperability benefits of open standards, it has been slow to translate this into easily understandable guidelines for implementation at the level of everyday applications such as office document formats. As far as education is concerned, the use of modifiable office document formats has now reached a crucial stage. There is an urgent need for co-ordinated, strategically informed action over the next five years, if the higher education community is to facilitate a cost effective approach to the switch to XML-based office document formats.

The report, written by Walter Ditch,  (and featured in The Register and on ZDnet) provides a very useful background to the needs for open document formats, a discussion about what openness means in this context (which references two of my papers on “Openness in Higher Education: Open Source, Open Standards, Open Access” and “A Contextual Framework For Standards“) and provides a summary of the strengths and weakness of the Open Document Format (ODF) and the Office Open XML format (OOXML).

The report argues that it is now timely for the HE sector to address the issue of how we should move away from use of proprietary office file formats. The report doesn’t make a recommendation on which format(s) we should  adopt or on the deployment strategies we will need – and I think the report is wise in this respect, as any decisions taken now may be made redundant by decisions to be made by ECMA regarding the standardisation of OOXML in the near future.  However the report does provide very useful information which will help to inform future discussions.

Recommended reading for a topic which, as Paul Anderson, the technical editor of the report, says on his blog there is ”a searing debate about which particular XML format all these software packages should make use of and which standard they should use“.  Paul goes on to say “It’s an indication of how deeply these issues are felt and how bitter the XML standardisation battle has become. It really is a war of words.”  Paul’s editorial role and the peer-reviewing process for this report have helped to ensure that the content of the report provides a neutral summary of the background to the standardisation processes.

Use Of Open Standards In JISC Development Programmes

A recent update sent to the E-Framework JISCMail list contains the following summary of a paper written by myself, Scott Wilson (CETIS) and Randy Metcalfe (formerly of JISC OSS Watch) :

Openness in Higher Education: Open Source, Open Standards, Open Access

June 2007. Anyone wishing to make achieve a better understanding of the “open” agenda in higher education should read this recent paper by Brian Kelly, Scott Wilson, and Randy Metcalfe presented at the ELPUB2007 Conference on Electronic Publishing in Vienna. The “open” word is of course used by all of us in incredibly different contexts and as the authors note: “For national advisory services in the UK (UKOLN, CETIS, and OSS Watch), varieties of openness (open source software, open standards, and open access to research publications and data) present an interesting challenge.”

More information at: http://elpub.scix.net/data/works/att/140_elpub2007.content.pdf

This paper, which was presented by Scott Wilson at the ELPub 2007 conference, builds on previous work which has sought to address the tensions between the potential benefits which open standards can provide and the dangers of using standards which fail to gain acceptance in the marketplace, are too complex, are superceded by alternative approaches or are used too soon. The paper argues that there is a need to be flexible (in order, for example, to avoid repeating the mistakes made when the UK higher education community was committed to use of Coloured Book networking protocols as a stepping stone to OSI network standards – a decision which was eventually overturned by the success of Internet networking standards).

The paper describes the parallels with pragamatic and user-centred approaches to use of open standards with the selection and use of open source software and providing open access to scholarly publications and data. In all these cases there are clear benefits to be gained by the sector, but there are also a whole host of complications which would be foolish to ignore.

These issues are very pertinent to the current JISC call for projects in its Capital Programme. The JISC Circular 02/07 document (MS Word format) clearly states JISC’s commitment to open access:

B21: JISC supports unrestricted access to the published output of publicly-funded research and wishes to encourage open access to research outputs to ensure that the fruits of UK research are made more widely available.

and goes on to demonstrate a pragmatic approach to use of standards:

B25: JISC mandates the deposit of the native version (Word, PPT, etc.), with PDF as well if wanted, but certainly with a format from which usable xml can in principle be derived (not PDF).

The approach to use of open standards which JISC requires projects to take is clearly stated:

B29. The institution and its partners must use the technical standards stipulated by JISC and where unstipulated open standards wherever possible, Any deviation should be justified in the proposal and any alternative be designed with re-use by others in mind. Easy of interoperability between systems is key to the provision of next generation technologies for education and research, and projects are expected to work with JISC to address this issue. It is the responsibility of the lead institution to inform its project partners accordingly. Relevant standards can be founded in the JISC Standards Catalogue
http://standards.jisc.ac.uk/.

This paragraph provides the flexibility needed to address potential problems which use of open standards may cause. The requirement to document any deviations is important, and reflects the approach developed by UKOLN in its work (with AHDS) in providing a technical advisory service to support the NOF-digitise programme. As described in the paper A Contextual Framework For Standards for that programme a documented report on deviation from mandated open standards was required as part of the reporting process, and an accompanying FAQ was produced.

I hope this post will be of use to anyone who may be considered submitting a proposal to this call.

Are W3C Crazy?

Phil Wilson recently reported on his trip to the XTech 2007 conference. Phil’s report included a bullet point which said that “The W3C are crazy”. In response to my request for clarification Phil said that:

There seemed to be a couple of big fat W3C elephants in the room.

The first was that the w3c are doing stuff for use in five or ten years’ time whereas most of the other talks are about things you can do today or next year, which makes them seem like futurologists.

The other is that they really didn’t seem that happy that HTML5 was going ahead, and what the hell was wrong with XHTML2 anyway?

It must be nice to work in a standards organisation where everything you do meets some Platonic Idea of perfection.

I think it is clear that W3C have had a very purist approach to the development of Web standards. Indeed Chris Lilley admitted in a talk on HTML Reloaded at the WWW 2007 conference that “99.99999% of the Web was invalid HTML. W3C pretended that didn’t exist.“

The W3C’s purist position is under pressure from companies such as Mozilla and Google, who feel that it is foolish to ignore that Web environment as it is today and build a new version of XHTML which is incompatible with HTML 4 and XHTML 1. Instead these companies, together with others who wish to build on existing tehcnologies, have been pushing evolutionary development of HTML 4, under the name HTML 5.

Under such pressure, the W3C has been forced to back both camps, with the chartering of a HTML Working Group (which will develop HTML ‘classic’) and a XHTML 2 Working Group.

Despite this concession, I feel that there is a culture at W3C which is uncomfortable will the need to address real world constraints and, as Phil describes it, prefers a world  which conforms to a “Platonic Idea of perfection“.

Are W3C crazy? No, not crazy, I would say, but idealist – and perhaps teasing the user community with a vision of perfection which is unlikely to be realised. And when Phil states they are “doing stuff for use in five or ten years’ time” it would seem he underestimates the timescales, as the WHATWG FAQ states, in response to a question on when HTML 5 will be finished: ”Around 15 years or more to reach a W3C recommendation (include estimated schedule)“.

Open Standards – Are We There Yet?

When will all the open standards we need be finalised? What will happen when that day arrives?  And are we there yet?

Daft questions, you may be thinking.  But if that’s the case – and we’ll never arrive at a position in which the open standards we need are all done – what does this mean for the development community?  Is the seemingly never-ending development of standards simply a way of providing ‘jobs for the boys’ – so that software developers and standards developers will be guaranteed of a job?

Or, to ask a related question, are the standards which are available today good enough for most uses.  Andy Powell, in the eFoundations blog, raised this issue recently when he commented “I’m very mindful of the tension between the relatively complex … and the relatively simple, tag-based, approaches taken by Web 2.0 repository-like applications such as Slideshare and Scribd.“

Andy went on to admit that “Unfortunately, I lean uncomfortably in both directions!“  I think that many of us involved in development work would admit to similar doubts – and perhaps those who have no doubts are those with a blinkered vision who were responsible for leading the UK HE sector down the cul de sac of Coloured Book network protocols in the 1980s.

What should be done?

Addressing The Limitations Of Open Standards

Open standards are great – they can provide machine- and application-independence, thus avoiding vendor lock-in and they can help to ensure services are interoperable and are widely accessible. Unfortunately open standards don’t always work – they can be too ambitious, fail to gain market acceptance, may be too costly to implement or be superceded by alternatives. So how do development programmes ensure they make use of open standards which will be successful and avoid making costly mistakes when selecting standards? This is the theme of a paper on “Addressing The Limitations Of Open Standards” by myself, my colleague Marieke Guy and Alastair Dunning, AHDS which will be given at the Museums and the Web 2007 Conference on 12 April.

The paper and accompanying slides are available. Your comments are welcome.

Technorati Tags: MW2007