A Challenge to Web Accessibility Metrics and Guidelines: Putting People and Processes First

Martyn Cooper
Institute of Educational Technology
Open University
Milton Keynes, UK
+44 1908 655729 m.cooper@open.ac.uk

David Sloan
School of Computing
University of Dundee
Dundee, UK
+44 1382 385598

Brian Kelly
University of Bath
Bath, UK
+44 1225 383943

Sarah Lewthwaite
King's Learning Institute
King's College London
London, UK
+44 207 848394


This paper argues that web accessibility is not an intrinsic characteristic of a digital resource but is determined by complex political, social and other contextual factors, as well as technical aspects which are the focus of WAI standardisation activities. It can therefore be inappropriate to develop legislation or focus on metrics only associated with properties of the resource.

The authors describe the value of standards such as BS 8878 which focus on best practices for the process of developing web products and include a user focus.

The paper concludes with a case study that illustrates how learning analytics could provide data to support the improvement of the inclusivity of learning resources, providing a broader perspective beyond the digital resource.

Categories and Subject Descriptors
H.5.2 [User Interfaces - Evaluation/methodology]; K.4.2 [Social Issues - Assistive technologies for persons with disabilities]
General Terms
Measurement, Documentation, Human Factors.
Web accessibility, disabled people, policy, user experience, social inclusion, guidelines, development lifecycle, procurement.

1. Introduction

Endeavours to improve the quality of web interaction for people with accessibility needs have focused on developing and refining design guidelines and on providing support for web content creators to create content that can be accessed and used by the intended audience, regardless of any disability they may have. However, as evidence continues to show that disabled people encounter significant barriers when trying to use web sites to complete goals. As such, the complexity of the web suggests that a wider definition of an inclusive web is needed in conjunction with and alternative approaches to achieving such inclusivity.

A recent W3C Web Accessibility Initiative (WAI) online symposium on Website Accessibility Metrics invited submissions for approaches to the development of metrics for website accessibility [FN 1]. As described in [9], accessibility metrics may be useful for several reasons. However there is also a need to ask the question "How do metrics help web authors and developers provide more inclusive online services?" Cooper [4] has provided a critique of the current web accessibility metrics presented at the online symposium, and concluded that there was a need to address the requirements of the user and usage context which are not accounted for with metrics which only address factors associated with the digital resources. Rather than focusing only on development of more sophisticated accessibility metrics for web resources, the authors argue the emphasis should be based on enhancing practices which support the development of processes and policies which can help to provide more inclusive access to resources and services.

We have previously argued that technical accessibility guidelines are only one part of a wider strategy to encourage organisations to use the web to deliver inclusive services [7]. In this paper we look to the emergence of the "Third Wave" of Human-Computer Interaction [3] where focus on system quality extends beyond measures of successful task completion to supporting a positive (albeit subjective, and context-dependent) user experience. We suggest explanations for the challenges in achieving inclusion via the web caused by organisational complexity. We go on to discuss opportunities for taking advantage of existing data about users and usage patterns to gather evidence of quality of user experience of disabled people, with the intention of identifying targeted areas for reducing barriers in a pragmatic fashion. We argue that this analysis of usage patterns provides a more context-oriented, and pragmatic alternative to a prioritised guideline-based approach to accessibility measurement and remedial action.

2. The Benefits of a User Focus

WCAG 2.0 was developed openly and with specific public consultation periods. Indeed there was extensive critique and input into it from motivated individuals with disabilities and organisations of and for disabled people. Although the range of WAI activities include addressing user needs, browser and authoring tool requirements, business requirements, etc., existing legislation and regulation is focused on web content accessibility. However the focus of WCAG is on the technical artefact - i.e. the "web page" - not on users and user goals. The argument for this approach is that in many cases, it is impossible for developers to envisage the diversity of potential users; they understand best the technology and distillation of best practice into technical set of instructions in the form of guidelines best supports accessible content creation. This naturally increases the distance between developer and user [8], may affect true prioritisation of accessibility goals, which in turn can lead to difficulties in pragmatically achieving optimal accessibility.

There are user-focused efforts to assess and address Web accessibility issues. Organisations may actively seek feedback themselves through conducting usability testing with disabled people; Bigham et al [2] have reviewed a number of crowd-sourced methods of identifying and overcoming Web access barriers; though only a few of these methods lead to persistent repairs. The IBM Social Accessibility project is one example where users can report barriers in web content to volunteers who can implement fixes to proxy versions of that content [10]. The Fix the Web [FN 2] initiative provides another mechanism for disabled people to report accessibility issues encountered anywhere on the web, without requiring a technical explanation of the underlying barrier. These are then taken up with the site owners in question by one or more of a team of volunteers. At present, the system has yet to scale to anything like the reach envisaged by its initiators.

A proposed approach at the Open University is termed a Social Software Approach to Accessibility. Here, registered disabled people initially complete a profile of their assistive technology and access approaches. Whenever they visit a resource, within a closed set, say a repository, a Virtual Learning Environment (VLE), or potentially for the entire web, they nominate on a simple 3 point scale how accessible it was for them (accessible, problematic but partly accessible, inaccessible). Then, when any registered user searches the "domain" in question alongside any ranking for relevance, they receive an indication of the level of accessibility as experienced by all users with a similar access profile, and who have previously visited and rated those resources. As well as being of direct benefit to the users in finding accessible resources, this approach would enable cumulative accessibility reports to be generated and communicated to the owners of the resources in question.

In the above cases, the driver for change comes from people who may be reporting barriers in the context of difficulty or inability to complete a goal - but depending on the nature of the feedback mechanism, potentially also a reflection on user experience.

3. From Resource to Process

At present, accessibility discourse focuses heavily on useful-but-simple accounts of user experience that focus on web resources as the location of accessibility. Such technical approaches are frequently localised, to the detriment of more productive, nuanced and critical understanding. In practice, for example, this means a developer, or commissioner, newly engaging with WCAG gains very little sense of the social, economic, political or cultural nature of accessibility.

In this paper we argue for a relational approach to accessibility. Accessibility is a property of the relation between the user and the resource in the context of how that is mediated; not a property of the resource. Accessibility must be situated within a real world context, and acknowledge the unequal power structures that constitute disability and accessibility. To illustrate this, it is useful to consider how present guidelines that strive for universality may result in counter-productive, paternalistic outcomes that may displace alternative approaches.

Previous work by the authors has highlighted the importance of context to accessibility outside areas of mainstream information provision, for example, by identifying a need to prioritize accessible learning outcomes, rather than discrete digital resources, as key to developing accessible and equitably experiences of education [5]. Further work reports the importance of recognizing the cultural context of the user to enhance accessibility of culturally situated resources in galleries or museums [7]. A critical engagement with the politics of accessibility discourse is proposed by emergent work from Disability Studies [6]. This locates disability and the experience of accessibility within a relational socio-cultural frame of competing economic, cultural and political forces and subsist alongside other indices of exclusion (for example, age, gender, sexuality, ethnicity, class). Such work highlights that all digital resources are situated, and that experience of the resource is likewise framed. From this critical viewpoint, there is a concern that technical guidance is not only partial, but that conformance can be counter-productive. If advocacy and legislation promote adherence to web standards as the sole determinant of disabled people's experiences of technology, this could undermine the importance of significant cultural, political, social and other 'real world' issues.

As a result, rather than suggesting the development of more sophisticated accessibility metrics for web resources, the authors propose that the emphasis should be based on enhancing practices which support the development of policies and processes which in turn can help organisations and authors to provide more inclusive access to resources and services.

4. Accessibility as a Process

The development of web assets or applications is a process. Accessibility considerations need to be built into the everyday practices across the full web product life-cycle from conception and specification through development to delivery and maintenance. Recognising this, the British Standards Institute developed BS 8878: 2010 Web Accessibility Code of Practice [1]. As described in [9] this document provides:

"... a framework that allows definition - and measurement - of the process undertaken by organisations to procure an optimally accessible web site, but is at present a copyrighted work and not freely available. In comparison to a purely technical WCAG conformance report, the nature of the data being gathered for measurement means that inevitably the measurement process is longer; but it also provides a richer set of data giving context - and therefore justification - to current levels of accessibility."

Section 6 is the core of the standard. It makes recommendations for accessibility being addressed across a 16 Step Model of the web product development and maintenance process (see Figure 1). These steps span: initial conception and requirements analysis (steps 1 to 6); strategic choices based on that research (steps 7 to 11); the decision to procure or develop the web product either in-house or contracted out (step 11); production of the web product (steps 12 and 13); evaluation of the product (step14); the launch (step 15); and post-launch maintenance (step 16).

Step 1: define the purpose of the web product
Step 2: define the target audiences for the web product
Step 3: analyse the needs of the target audiences for the web product
Step 4: note any platform or technology preferences and restrictions of the web product's target audiences
Step 5: define the relationship the product will have with its target audiences
Step 6: define the user goals and tasks the web product needs to provide
Step 7: consider the degree of user-experience the web product will aim to provide
Step 8: consider inclusive design and user-personalized approaches to accessibility
Step 9: choose the delivery platforms to support
Step 10: choose the target browsers, operating systems and assistive technologies to support
Step 11: choose whether to create or procure the web product in-house or contract out externally
Step 12: define the web technologies to be used in the web product
Step 13: use web guidelines to direct accessible web production
Step 14: assure the web product's accessibility through production
Step 15: communicate the web product's accessibility decisions at launch
Step 16: plan to assure accessibility in all post-launch updates to the product

Figure 1: 16 Step Model of BS 8878

This model has been drawn up based on real-world experience in companies and organisations that have effectively addressed accessibility. BS 8878 addresses accessibility both at the organisational level and the individual product level. It needs to be adapted to any situation it is applied. Here we use it to briefly analyse the role of web accessibility guidelines and metrics. In the notes expanding the 16 step model only in Steps 8, 9, 12, and 13 is WCAG is referred to. Step 8 is a consideration of whether to adopt inclusive design or personalisation or approaches and suggests inclusive design is represented by WCAG. Step 9 is about choosing the delivery platforms targeted. It encourages a review of the relevance and applicability of WCAG, depending on whether the platforms concerned have implemented W3C WAI ATAG and UAAG guidelines.

Further, illustrating a theme in this paper, it is readily possible for an individual to find a particular web resource accessible on one platform, say a smart phone, but presenting them with accessibility challenges or barriers on different platforms, say a PC or web-enabled TV. Hence, universal statements about levels of accessibility of that resource become meaningless without qualification about the users' functional abilities and the properties of the technologies they are using to access the resource. Step 12 is about defining the web technologies adopted in the product. It encourages questioning as to whether the technologies facilitate accessibility and if suppliers provide techniques for conforming to WCAG.

It is only Step 13 that refers to WCAG in the way its use is normally envisaged in guiding the adoption of accessible approaches in development. This confirms that there is much more to achieving accessible web products than referring to WCAG in isolation. Step 14 is about assurance of accessibility but its notes do not specifically mention WCAG. Section 8 of BS 8878 treats assurance of accessibility not as something achieved by testing towards the end of the development phase, but as requirements gathering and a series of testing made throughout the life-cycle of the product. It makes recommendations for: gathering requirements from disabled users; creating an accessibility test plan; accessibility testing methods; post-launch programme of accessibility testing. It should be noted that BS 8878 makes no direct reference to accessibility metrics.

5. Learning Analytics And accessible user experience

Whilst BS 8878 focuses primarily on the process of commissioning and implementing a web site with accessibility in mind, the post-launch programme of accessibility testing is significant (Step 16 in the BS 8878 model). How can organisations gather an ongoing indication of quality of user experience by disabled people?

One approach is to take advantage of analytics - data gathered automatically on user behaviour when interacting with a particular web resource. The 1st International Conference on Learning Analytics & Knowledge [FN 3] defined Learner Analytics as:

... the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.

The Open University Learner Analytics Project is reviewing data the university currently collects, or could easily collect, that could be integrated meaningfully in a learner analytics approach. One area of interest is withdrawals - e.g. when students stop study before completion of a module towards a degree. Such "drop-outs" are a high-stakes issue for universities, because they form part of the assessment of the quality of their teaching, which in turn impacts on levels of funding from government and from student-paid fees.

It is envisaged that, with a learner analytics approach, it will be possible to map for the whole student body what points on paths of study withdrawals occur. It will be further possible to analyse this data comparing withdrawals by disabled students in particular with the student body in general. It is worth noting that the Open University is a large institution with more than 200,000 students, more than 12,000 of whom declare a disability. Hence there is a reasonable chance of data about the point of withdrawal across the educational context revealing that there is something of significance relating to that context over and above the more random distribution of withdrawals for non-study related issues such as health problems and family circumstances.

An analysis has been made of completion and pass rates on all undergraduate modules presented in 2010-11, with a minimum of 10 students who had declared at least one disability (164 modules). The differences in module completion and pass rates between disabled students and non-disabled students falls in the range [-33% to +29%]. On the majority of modules (67%) disabled students fail to complete or do not pass the course in a greater proportion than non-disabled students. If this data was routinely reviewed in a learner analytics approach investigations could be triggered of what might be the factors in module design that might be leading to poor completion and pass rates for disabled students. Further, what are factors in the modules where disabled students are doing as well or better than the non-disabled members of their cohort? The reasons for these could be diverse, but might include issues of accessibility at the teaching and learning level or at the technical level of how the teaching and learning is mediated; which increasingly is web-based.

The learner analytics here only indicate where there might be a problem, not what it is. Notwithstanding the reservations expressed about accessibility metrics in [4], a focused use of web metrics could be useful here. If the web assets of the module in question were all reviewed by a metrics approach it could be determined if web accessibility is likely to be a significant factor in the high level of withdrawals in this case. A fuzzy threshold for the particular metric chosen indicating where web accessibility impacts on withdrawal may be established over time, but comparisons with the metrics applied to modules with a low level of withdrawal by disabled students would be the starting point. Thus an analytics and metrics approach could support continued improvement in accessibility in the case of legacy assets. This is a significant issue within eLearning generally where content reuse and persistence is high.

A further point that supports linking learner analytics and web metrics is that there are insufficient resources (staff time for those with expertise) to apply web metrics to existing web assets and activities across the university. The learner analytics approach enables available effort to be focused where it has most impact.

Any issue identified by a learner analytics approach could be further analysed to compare the experience of disabled students with those of the student body as a whole. We argue this can highlight issues needing further investigation and suggest in the case where courses are making extensive use of the web, and accessibility metrics may be part of this further investigation. Heuristic and end-user evaluation may then be used to reveal more in-depth aspects of underlying barriers.

The key message is that if web components of a course are problematic for disabled students, this could be because of failures that can be measured against WCAG or they could be because of: usability issues; insufficient support and training for the student; or poor inclusive learning design. The emphasis of the learner analytics approach here is therefore as an initiator of a process of improvement based on better understanding of the user experience, not in the first place reviewing properties of the web resource.

6. Conclusions

In this paper we have asserted the pre-eminence of the users' experiences in assessing the accessibility of any web resource. This follows from the understanding of accessibility as a property of the relationship between the users and the resource, and that context is important. In seeking to ensure good experiences for disabled people in interacting with web resources, we highlight the importance of consideration of their needs throughout the development cycle, and the value of systems that collect and respond to user-provided feedback on accessibility, in their own terms. We commend the approach articulated in BS 8878 that helps organisations build this into their processes.

Recognising that accessibility is an area for continued improvement and the challenges organisations face in optimising accessibility of content, we have highlighted learner analytics as one example where an analytics approach could help organisations identify and focus on where remedial action is most urgently required. Further work in this area is ongoing to explore the efficacy and level of applicability of this approach as a contributor to a process of designing for inclusive web experiences.


  1. British Standards International (2010). BS 8878:2010 Web Accessibility - Code of Practice, (charged for publication available through http://www.bsi-publications.com/)
  2. Bigham, J., Ladner, R. and Borodin, Y. (2011). The design of Human-Powered Access Technology, Proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (ASSETS '11). New York: ACM Press.
  3. Bodker, S. (2006). When second wave HCI meets third wave challenges. In Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. pp.1-8.Oslo, Norway.
  4. Cooper, M. (2011). Web accessibility metrics - "What are they for then?", http://martyncooper.wordpress.com/2011/12/06/web-accessibility-metrics/
  5. Kelly B., Sloan D., Phipps L., Petrie H. and Hamilton F. (2005). Forcing standardization or accommodating diversity? A framework for applying the WCAG in the real world. Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Chiba,Japan, 10 May 2005). New York: ACM Press.
  6. Kelly, B., Lewthwaite, S. and Sloan, D. (2010). Developing Countries; Developing Experiences: Approaches to Accessibility for the Real World. Proceedings of the 2010 International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Raleigh, USA, 26-27 April 2010). New York: ACM Press
  7. Kelly, B., Sloan, D., Brown, S., Seale, J., Petrie, H., Lauke, P and, Ball, S. (2007). Accessibility 2.0: People, Policies And Processes. Proceedings of the 2007 International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Banff, Canada, 7-8 May 2007). New York: ACM Press.
  8. Lewthwaite, S. and Swan, H. (2012). Disability, Web Standards and the Majority World. In: L. Meloncon (Ed.) Rhetorical AccessAbility: At the intersection of technical communications and disability studies. Baywood Publishing.
  9. Sloan, D. R. and Kelly, B., (2011). Web Accessibility Metrics For A Post Digital World, RDWG Symposium on Website Accessibility Metrics, 5 December 2011 http://www.w3.org/WAI/RD/2011/metrics/paper10/
  10. Takagi, H., Kawanaka, S., Kobayashi, M., Sato, D., and Asakawa, C. (2009). Collaborative web accessibility improvement: challenges and possibilities. Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility (ASSETS '09). New York: ACM Press 195-202.


[1] See http://www.w3.org/WAI/RD/2011/metrics/

[2] See http://www.fixtheweb.net/

[3] See https://tekri.athabascau.ca/analytics

Citation Details

A Challenge to Web Accessibility Metrics and Guidelines: Putting People and Processes First, Cooper, M., Sloan, D., Kelly, B. and Lewthwaite, S. WWW 2012 Lyon, France, 16-17 April 2012. <http://www.ukoln.ac.uk/web-focus/papers/w4a-2012/>

Copyright Details

W4A2012 - Submission Type, Communications. April 16-17, 2012, Lyon, France. Co-Located with the 21st International World Wide Web Conference. Copyright 2012 ACM. ISBN 978-1-4503-1019-2