What Can Web Accessibility Metrics Learn From Alt.Metrics?

Research Report on Web Accessibility Metrics

A W3C WAI Research Report on Web Accessibility Metrics, published on 30 August 2012, is currently open for review, with feedback requested by 30 September 2012.

The introduction to the report describes how:

Recently, a plethora of metrics has been released to complement the A, AA, and AAA Levels measurement used by the WAI guidelines. However, the validity and reliability of most of these metrics are unknown and those making use of them are taking the risk of using inappropriate metrics. In order to address these concerns, this note provides a framework that considers validity, reliability, sensitivity, adequacy and complexity as the main qualities that a metric should have.

The introduction concludes:

A symposium was organized to observe how current practices are addressing such qualities. We found that metrics addressing validity issues are scarce although some efforts can be perceived as far as inter-tool reliability is concerned. This is something that the research community should be aware of, as we might be making efforts by using metrics whose validity and reliability are unknown. The research realm is perhaps not mature enough or we do not have the right methods and tools. We therefore try to shed some light on the possible paths that could be taken so that we can reach a maturity point.

David Sloan and I contributed to an Website Accessibility Metrics symposium held last year with a paper Web accessibility metrics for a post digital world (available in PDF, MS Word and HTML formats). The abstract for our paper stated:

This paper argues that, as we move towards a ‘post-digital’ world where use of the Web becomes normalised, there is a need to address Web accessibility measurement challenges within a wider real-world context. Strategy and policy that defines Web accessibility purely by the conformance of digital resources with technical guidelines can lead to a danger that ‘good enough’ solutions may fail to be deployed; they also fail to consider a wider measure of user experience in accessibility measurement. We propose that metrics should draw on aspects of user experience to provide a more meaningful, real-world measure of the impact (or not) of accessibility barriers and therefore priority in addressing them. Metrics should also consider context in terms of the quality of effort taken by organisations to provide an inclusive experience; one option for doing so is the framework provided by British Standard 8878 Code of Practice for Web Accessibility. In both cases, challenges exist in the complexity of defining and implementing such metrics.

Or, as we described in a follow-up paper entitled A challenge to web accessibility metrics and guidelines: putting people and processes first:

This paper argues that web accessibility is not an intrinsic characteristic of a digital resource but is determined by complex political, social and other contextual factors, as well as technical aspects which are the focus of WAI standardisation activities. It can therefore be inappropriate to develop legislation or focus on metrics only associated with properties of the resource.

In the light of our involvement in last year’s research symposium we intend to respond to the W3C’s request for feedback on the Research Report on Web Accessibility Metrics. It should be noted that responses must be submitted by 30 September 2012 to the public-wai-rd-comments@w3.org list which has a publicly visible mailing list archive. I am therefore posting some thoughts on this blog in advance of that date in order to get feedback before making the formal response.

What Can Web Accessibility Metrics Learn from Alt.Metrics?

Section 1.2 of the report is entitled The Benefits of Using Metrics. This section should, I feel, be followed by a section on The Risks of Using Metrics. It would be useful to base such a section on the experiences gained in other areas in which metrics are being developed. Areas in which useful comparisons could be made include metrics for online reputation (i.e. services such as Klout) and assessment of research impact (e.g. alt.metrics); in both of these areas the potential benefits of metrics have been identified, but their limitations are also acknowledged.

Last Friday (21 September 2012), a live chat on Twitter, peer review and altmetrics: the future of research impact assessment took place on the Guardian’s Higher Education Network. During the discussion DrGunn (Dr. William Gunn, the Head of Academic Outreach for Mendeley) pointed out the need to recognise the limitations of metrics for the assessment of research impact:

First, it’s important to note that the point of altmetrics isn’t to suggest that it may be possible to use any measures of impact or influence to fully judge the merit of individuals. Citations, tweets, bookmarks, etc are all indicators of influence, but influence isn’t merit.

and highlighted the need for evidence which demonstrates relationships between metrics and tangible real-world outcomes:

The second point is addressing Stevan’s comment about coupling metrics to real positive outcomes. Science Exchange, Mendeley, PLOS, and Figshare have joined together and launched the Reproducibility Initiative, which aims to provide a positive incentive for doing work that’s robust and reproducible.

Such approaches are needed in the development of metrics for Web accessibility. But we should be clear about what is being measured. Is the work in identifying areas of research aiming to develop ways of measuring conformance with Web accessibility guidelines, such as WCAG. Or is the aim to develop metrics which relate to real-world experiences of people with disabilities seeking to make use of Web products?

In the conclusions the report makes the point that:

Employing metrics whose validity and reliability is questionable is a very risky practice that should be avoided. We therefore claim that accessibility metrics should be used and designed responsibly.

The statement that “accessibility metrics should be used and designed responsibly” is meaningless as nobody would argue that “accessibility metrics should be used and designed irresponsibly“! The report needs to  be clear about the reasons why Web accessibility metrics are being developed and who the beneficiaries of such work would be. The development of an international standard for Web accessibility metrics might benefit large software vendors, which would have a global market for selling tools for measuring conformance with such standards. There might also be benefits for organisations which would like to be able to display a badge demonstrating conformance with such standards. But what are the benefits for the user community, especially users with disabilities?

In addition there is a need to consider the risks in developing Web accessibility metrics. Might the development of such metrics lead to organisations failing to provide Web services if they failed to conform fully with such metrics, even if such services may still be of value to people with disabilities?

I’d welcome your comments, but more importantly, I’d encourage people with an interest in this area to respond to the call for comments  by sending a message by 30 September 2012 to the public-wai-rd-comments@w3.org (note this list has a publicly visible mailing list archive).

Twitter conversation from: [Topsy]