Why Automated Accessibility Testing Isn’t Enough

When we do accessibility audits at b13 our main focus is on manual testing. But we also have incorporated automated testing tools, like Axe or Wave into our testing workflow. These tools are great. They help us to find common failures of the WCAG success criteria quickly and get a feel for the Accessibility of the page. 

Automated tools for accessibility testing

Axe or Wave are great in finding certain issues. They tell you about: 

• empty links and buttons,
• missing alt-text,
• if elements are not focusable,
• if forms are invalid,
• if form labels are missing,
• if color contrast is not enough. 

In short: they find the most common issues almost 96% of all websites have, according to the WebAIM Million report.

If time and budget are valid concerns for your accessibility testing and you want to do something quickly about your accessibility issues, making leverage of automated testing tools is a way to go. So is that enough to fix all the accessibility issues you have on your website? If you use these automated testing tools and fix the most common issues that’s a great way to handle this and you won’t need to test manually anymore, right?

Sorry to destroy your hope at this point, but no. Automated testing tools only catch between 20 and 30% of all WCAG failures, depending on who you ask. Gen Herres summarized a study by Deque. Deque is the developer company behind Axe and the study prevails that Axe is able to test 15 out of the 50 WCAG success criteria, or about 30%. 

Melwyn Joseph summarizes several studies and draws a similar conclusion and states:

Adrian Roselli conducted a study himself where he tested all applicable success criteria manually and used automated tools. He comes out with even lower numbers, but also shares a valuable lesson: 

What automated tools can’t test properly

Why can’t we get higher numbers in the range, automated tools are able to test for? The big problem with tools like Axe or Wave are: they lack the context of a page and the user. They can only test the code of the current page, not the changes in viewport, orientation or size. Here is an exemplary list of WCAG success criteria, automated tools are not able to test properly.

1.1.1 Non-text Content

• Automated tools are great in telling you if images are missing alt text, as stated in the beginning. But the thing about alt text is, it’s not a simple boolean comparison: having an alt text or not. It’s more complex. Alt text needs to incorporate context and emotion.
• Automated tools can’t figure out if the present alt text is meaningful. They can’t figure it if an image is decorative and therefore empty alt text is sufficient.

1.2.2 Captions

• Captions are important for deaf people if they watch a video. Automated tools are not able to determine if a video has captions and if this captions are actually meaning- and helpful to deaf people. 

1.3.1 Info and Relationships

• This success criterion is about the proper usage of HTML Elements. It’s about characterizing headings as headings, tables as tables, paragraphs as paragraphs, lists as lists, block quotes as block quotes and so on. Automated tools are not able to detect if specific information should rather be presented as a table or as a text block. Or if a text block should rather be shown as an ordered list. Humans can, because the can evaluate the context of the content.

1.4.3 Contrast (Minimum)

• Automated tools are able to determine if the color contrast on an element is insufficient.
• But they still have huge problems to figure this out for semi-transparent and gradient backgrounds. 

2.1.1 Keyboard

• Automated tools may be able to tell if elements are focusable (2.4.7) and if the focus is visible to users.
• But they can’t tell if a whole website is fully usable with a Keyboard. 

2.4.2 Page Titled

• Automated tools are able to test if a website has a page title.
• But this success criterion is not only about having one, it’s about having a meaningful title. 

2.4.3 Focus Order

• The way users with a screen reader read content can differ from how sighted people do it. Automated tools cannot detect the correct focus order because they lack the context.  

3.2.1 On Focus, 3.2.2 On Input

• These success criteria are about unannounced context switching. If you focus an element, or change the setting of a component, the context should not changed unless clearly advised about the behavior before. This can lead to huge cognitive load and confusion for users. Something automated tools can’t test at all.

3.3.3 Error Suggestion

• If a user fills out a form and there’s an error message, it should be clear to the user and understandable in a way that they are able to correct their input. How are automated tools able to test this? It’s again about context and these tools can’t check if error messages are helpful or not. Only humans can.

4.1.2 Name, Role, Value

• Automated tools can check if components have WAI-ARIA roles and properties. This makes it easier for screen reader users to understand them.
• But they are not able to fully simulate how an actual screen reader is used. So while adding the correct WAI-ARIA Role to a custom element satisfies the Axe report, it’s still possible that Screen reader users cannot operate this part of the website.

This is just a small sample of the success critera automated tools can’t test for. Unfortunately the list could go on and on. The important thing that remains is, if something is accessible when it’s meaningful or needs the correct context to test it, automated tools are not suited for that task, only manual testing is. 

Conclusion

Automated testing is a valuable tool—it’s fast, cost-effective, and helps detect common accessibility issues. However, fixing accessibility is not a quick fix.

As shown above, automated tests cannot catch every WCAG failure, nor can they ensure a website is fully accessible. That’s why an effective accessibility testing strategy combines both automated and manual testing.

These two approaches work best together:

• Automated testing alone is quick but misses many critical issues.
• Manual testing alone is thorough but costly and time-intensive.

By combining both methods, a more accurate testing result can be achieved andthe most WCAG violations can be identified.

But we need to remember: WCAG compliance is just the baseline. Meeting the guidelines does not automatically mean a website is fully accessible for all users.