Metrics, This Time For Web Accessibility

Metrics For Web Accessibility

A recent tweet from @LouWoodley alerted me to a post which described “Here is how you can game Klout“.  The post described how an automated bot seems to have been successful in gaining a high ranking on Klout.  A clear example of the limitations of automated use of metrics in order to seek to establish some kind of value – in the case related to the impact, outreach and influence of individuals using Twitter.

In this post, rather than revisiting a discussion of the pros and cons of metrics for analysing interactions on the social web I’d like to drawn attention to a call for short papers for an “Online Symposium on Website Accessibility Metrics“. The call for papers describes how:

The W3C/WAI Research and Development Working Group (RDWG) will hold an online symposium on measuring website accessibility and invites your contribution. The goal of this symposium is to bring researchers and practitioners together to scope the extent and magnitude of existing website accessibility metrics, and to develop a roadmap for future research and development in the field.

The background to this area of work describes how:

Measuring the level of web accessibility is essential for assessing accessibility implementation and improvement over time but finding appropriate measurements is non-trivial. For instance, conformance to the Web Content Accessibility Guidelines (WCAG) is based on 4 ordinal levels of conformance (none, A, AA, and AAA) but these levels are too far apart to allow granular comparison and progress monitoring; if a websites satisfied many success criteria in addition to all Level A success criteria, the website would only conform to level A of WCAG 2.0 but the additional effort would not be visible.

and goes on to admit that:

Using numerical metrics potentially allows a more continuous scale for measuring accessibility and, to the extent that the metrics are reliable, could be used for comparisons. However, it is unclear how metrics can be developed that fulfill requirements such as validity, reliability, and suitability. For example, is a web page with two images with faulty text alternatives out of ten more accessible than another page with only one image with a faulty text alternative out of five? While such a count may be a fairly simple and reliable metric it is generally not a valid reflection of accessibility without additional information about the context in which the faults occur, but identifying this context may introduce complexity, reduce reliability, and raise other challenges.

The online symposium is invited submissions for papers which will “constitute the basis from which to further explore a research and development roadmap for website accessibility metrics”.  Papers are invited which discuss of the relationship of two approaches: (1) measuring ‘accessibility in terms of conformance’ with guidelines such as WCAG or Section 508 and (2)  ‘accessibility in use’ metrics that reflect the impact that accessibility issues have on real users, regardless of guidelines.  Papers are invited to address the following types of questions:

• What sort of techniques can we explore to combine metrics that are computed automatically, semi-automatically (with input from humans), and manually (where the judgement is made by humans, even if with input from software)?
• How can we build an infrastructure (such as IBM Social Accessibility) that allows experts to store accessibility information (metadata) for use with metrics that are computed during subsequent audits?
• What metrics, or combination of metrics, can be used as predictors of accessibility?
• How shall we characterize the quality of such predictors in terms of properties such as reliability, validity, sensitivity, adequacy and adaptability?
• Which approaches can be embraced for validating, benchmarking, and comparing web accessibility metrics?
• How should we tackle metrics in web applications with dynamic content?

Discussion

Myself and several Web accessibility researchers and practitioners have, over the years, written several papers in which we have described reservations regarding use of WAI guidelines as a definitive benchmark for Web accessibility. Our work began in 2004 with a paper on “Developing A Holistic Approach For E-Learning Accessibility” and a year later a follow-up paper on “Implementing a Holistic Approach to E-Learning Accessibility” won a prize for the best research paper at the ALT-C 2007 conference. Since then we have published several further papers with contributions from a number of Web accessibility researchers and practitioners which have developed our ideas further. In brief we might describe our ideas with the Twitter-friendly summary: “Accessibility is about people & their content; it is not an inherent characteristic of a resource“.

I therefore welcome the invitation for ideas on “ ‘accessibility in use’ metrics that reflect the impact that accessibility issues have on real users, regardless of guidelines“.

I should say that I do feel that there is still a need to be able to provide more sophisticated Web accessibility rankings that go beyond the WAI A/AA or AAA scores (which, in a university context seem to have parallels with the first, upper second, lower second and third class ranking we have for undergraduate achievements). We should be able to useful differentiate between Web pages (and Web sites) which  have images which have 95% containing alt equivalents from those with only 5% – currently both such pages will be treated equally as a WCAG failure. Similarly it would be useful to rank the importance of a failure to conform with WCAG guidelines (for example missing alt text on images should be regarded as more significant than having a hr element without the required terminating / in an XHTML document which breaks HTML conformance requirements).

But to prioritise analysis and subsequent actions based on the resource still focuses on the content, in isolation of the context, the target audience and the purpose of the Web resource or service.  Such an approach fails to consider blended approaches (“blended accessibility for blended accessibility” as we described in a presentation back in 2006) or the provision of multiple access routes to content (such as access to content published in the Guardian which, as described in a recent post, can be accessed via the Guardian Facebook, Kindle, iPhone or Android apps, on the Guardian Web site and via Guardian RSS feeds.

The “Online Symposium on Website Accessibility Metrics“, however, provides an opportunity to revisit previous work from a new perspective: the role of metrics. In general metrics may be needed for several purposes including:

• Measuring conformance to agreed benchmarks
• To identify good and bad practices which can lead to sharing experiences and changing working practices.
• To penalise conformance which fails to reach acceptable measures.
• To understand the limitations of benchmarks and to identify ways in which they may be refined.

Tools such as Bobby, WebXact, WAVE, and similar accessibility checking tools tended to focus on conformance with WCAG guidelines and, in some cases, the Section 508 US guidelines which had many similarities to WCAG.

However an initial survey published in 2002 and a follow-up survey in 2004 of  conformance with WCAG guidelines for UK University home pages using Bobby was used to demonstrate the limitations of such tools and the underlying guidelines they were based on: i.e. the 3 universities  in 2002 and the nine universities in 2004 having a WCAG AA conformance rating (based on automated testing which will, in any case, tend to give a higher rating) demonstrated failures ion the guidelines rather than failing across the UK higher education sector.

This example illustrates how the tools were used to demonstrate flaws in the underlying guidelines,  and why it would, in some cases, be inappropriate to use the metrics to highlight good or bad practices or to penalise lack of conformance.

What Is To Be Done For Web Accessibility Metrics?

Whilst our accessibility research highlighted limitations in WAI’s approaches, we did not feel it would be appropriate to abandon efforts to develop guidelines to enhance access to digital resources and services for people with disabilities.   We proposed a less mechanistic approach which we described as a “holistic approach to Web accessibility” which was described in a paper on  “Implementing A Holistic Approach To E-Learning Accessibility” and subsequently further refined and developed in several papers including”Holistic Approaches to E-Learning Accessibility” and “Implementing A Holistic Approach To E-Learning Accessibility“.

Our approaches became validated with the launch of BS 8878; Code of Practice for Web Accessibility.  As described in  a post on BS 8878: “Accessibility has been stuck in a rut of technical guidelines” BS 8878 acknowledges the complexities of Web accessibility and provides a 16 step plan which identifies the necessary steps for enhancing accessibility, whilst acknowledging that it may not always be possible to implement best practices at all stages.

For me, therefore, the focus of future Web metrics work should be based on metrics associated with the 16 stages of BS 8878, rather than the single stage in BS 8878 which addresses technical guidelines such as WCAG.

In a post on Web Accessibility, Institutional Repositories and BS 8878 I described how the UK’s BS 8878 Code of Practice for Web Accessibility might be applied in the content of large numbers of PDF resources hosted in institutional repositories. Some suggestions for  metrics associated with the various stages described in the post are given below.

1. Note any platform or technology preferences:
   Advice: PDFs may not include accessibility support.
   Metrics: Monitor numbers of resources provided in PDF format and measure changes over time.
2. Define the relationship the product will have with its target audience:
   Advice: The paper will be provided at a stable URI.
   Metrics:  Monitor changes in URIs for resources and report on any changes.
3. Define the user goals and tasks:
   Advice: Users will use various search tools to find resource. Paper with then be read on screen or printed.
   Metrics: Monitor terms used in searches and use to identify possible usability problems.
4. Consider the degree of user experience the web product will aim to provide:
   Advice: Usability of the PDF document will be constrained by publisher’s template. Technical accessibility will be constrained by workflow processes.
   Metrics: Feedback on enhancements to the template should be made to the publisher and records kept on implementation of recommendations.
5. Consider inclusive design & user-personalised approaches to accessibility:
   Advice: Usability of the PDF document will be constrained by publisher’s template. Technical accessibility will be constrained by workflow processes.
   Metrics: Records should be kept on personalised preferences selected and feedback should be gathered on the personalised experiences.
6. Choose the delivery platform to support:
   Advice: Aims to be available on devices with PDF support including mobile devices.
   Metrics: Records on usage of platforms should be kept and used to inform and possibly modify the policy decision.
7. Choose the target browsers, operating systems & assistive technologies to support:
   Advice: All?
   Metrics: Selection of target browsers may be determined by popularity of such browsers.  There will therefore be a need to define how to measure browser usage.
8. Choose whether to create or procure the Web product:
   Advice: The service is provided by repository team.
   Metrics: Not appropriate.
9. Define the Web technologies to be used in the Web product:
   Advice: HTML interface to PDF resources.
   Metrics: Not appropriate.
10. Use Web guidelines to direct accessibility web production:
    Advice: HTML pages will seek to conform with WCAG 2.0 AA. PDF resources may not conform with PDF accessibility guidelines.
    Metrics: Use of automated tools to measure conformance with best practices for HTML and PDF resources.  Deviations from best practices should be documents and remedial action taken, if felt to be appropriate.
11. Assure the Web products accessibility through production (i.e. at all stages):
    Advice: Periodic audits of PDF accessibility planned.
    Metrics:  This is the key area.
12. Communicate the Web product’s accessibility decisions at launch:
    Advice: Accessibility statement to be published.
    Metrics: Provide a feedback tool for the accessibility statement and ensure that issues raised are addressed.

Does this seem to be an appropriate response to the question “What should be done with Web accessibility metrics?”