Automated vs Manual Accessibility Testing: How to Make Your Digital Products Usable to Everyone

We met up with Jose Martinez, a member of one of Briteweb’s long-standing clients and strategic partners, The Chicago Lighthouse. Since 1906, they have served the blind, visually impaired, disabled, and Veteran communities of Chicago. The Chicago Lighthouse provides assistive technology and Digital Accessible Experience (DAX) consulting to pave solutions for accessibility to everyone, everywhere. Jose is the Lead DAX Analyst and today we interviewed him about his thoughts on automated vs. manual accessibility testing.

Meet Jose of the Chicago Lighthouse

Jose’s journey started at The Chicago Lighthouse as a baby when a counselor came to his parents’ home to advise them on how to interact and support a child with blindness. We fast forward to 2015 when Jose joined The Chicago Lighthouse team as a call center agent. From there, management took heed of his abilities, soon promoting him to IT as a desktop support specialist.

He became a de facto ambassador for The Chicago Lighthouse’s community of VI employees. From his personal experience of living with blindness, he has provided valuable insight on IT barriers for people with disabilities and assisted the organization’s exploration of accessibility compliance testing.

After working on an internal accessibility project to allow career progression for agents who use screen readers in one of The Chicago Lighthouse’s call center social enterprises, Jose and other team members realized the potential to provide critical accessibility services outside the organization as well. Jose was integral in developing a training curriculum to start and grow accessibility compliance and usability testing services at The Chicago Lighthouse that are now sought after across almost all industries. Jose now leads a team that helps web and app developers build inclusion into their digital properties.

The IT expert shared with us his wisdom of the various ways to test the accessibility of a product. In an important conversation, he offered his insight on the difference between automated and manual accessibility testing.

What is accessibility testing?

Accessibility testing evaluates if a product can be usable by everyone. Not just a certain segment of people, not people with specific disabilities, but the largest user base possible.

How to automatically test accessibility:

Accessibility can be tested by running your system through an automated program designed to point out any errors, bugs, and missing accessibility requirements.

Computers are excellent at following rules, they can tell if you broke a rule or if something doesn’t comply with a rule. It is recommended to use the Web Content Accessibility Guidelines (WCAG) to write the automated test based on your expected output. WCAG standards are established by W3C to ensure websites are accessible worldwide.

There are free Chrome browser extensions like, Axe Devtools, that will automatically run your webpage through its system to test your compatibility. The program will tell you if it finds any errors, pinpointing a direct problem area that needs fixing. These tools for automated accessibility testing provide your team with fast answers on the compatibility of your software.

Jose explains that, generally, when you run automated software, it’s going to scan through your page and present you with a rating. It will display how close your website is to the goal of one hundred percent compliance with WCAG standards. It will also provide results showing which pages do or do not have issues, which will be marked directly on the web page itself.

How to manually test accessibility:

Manual testing for accessibility requires having a person physically evaluate the user experience. This experimentation produces the best results when conducted by people with disabilities, however, it can also be tested by simulating a disability, like blindfold testing.

The people testing this product will be looking to first, find out if it can be used by people with disabilities, and second, that the usability is practical. When manual testing, you will want to know: Is this product usable for a wide range of abilities? Is it intuitive? Does it enhance a user’s accessibility to this product? What is missing or what can be improved?

Benefits to manually testing accessibility:

Disadvantages to manually testing accessibility:

Is manual testing or automatic testing better?

Jose’s opinion is that you can’t have one without the other. He says, “You could do all automated testing and yet your user base is going to come back to you and say there’s something wrong here. This site is impractical.”

The theory of your accessibility elements might seem great, but once it is actually put to use, it may not actually help people in the way intended. You can test a product through computer automation, then spend a large amount of time and money to make it compliant, but if you never consult human beings, then you’re at risk of the usability being poor. If the usability is poor, then it may not actually do the job of making the product available to people with disabilities.

Jose gives us an example of this with a throwback to using MP3 players. He remembers using Rockbox, an audio player that provided an audible interface for accessibility. He says it was great to be able to use this product like everyone else, but when it came to file management, that was no longer the case. There was no text-to-speech synthesizer, which means he could not read the file names, causing the product to not be practical for use. Rockbox made MP3 players accessible to the visually impaired, however, there was a significant barrier that made it hard to use. While accessibility was available, usability was not practical.

The goal of accessibility is to bridge the gap between what resources are available to people with disabilities and that no one is disadvantaged. In fact, if assistive technology is implemented correctly, then we can vastly increase opportunities for people with disabilities.

What does it look like to combine automated and manual accessibility testing?

Accessibility cannot be confined to a box. A holistic testing approach ensures that product usability is practical for users of all abilities. Once your product is ready for testing, first start by running it through an automated program to check that it aligns with WCAG standards. The automated test will provide a report to review and solve.
Next, round up a group of manual testers with a diverse range of abilities. Have them provide feedback on their personal user experience. Have your engineers cycle through that process until you have something that is both seamlessly accessible and usable.

How can we create solutions for accessibility?

It is a social responsibility that what we are building is available for everyone to use.

Jose recommends that companies hire people with disabilities to help in the process of creating practical accommodations. Hire them to find out how they live their lives and what workarounds they use to overcome everyday obstacles. It is one of the easiest ways to best understand accessibility and create more innovative solutions.

When working with a person with disabilities, it is important to keep in mind that they will have different comfort levels regarding an invitation into their personal life. Jose reminds us that he is speaking from just one perspective and there is a huge variety of differences when it comes to disabilities.

There will be varied cognitive and physical abilities. There will be people who have lived with a disability their entire lives and people who are newly adapting to their disability. While it is critical to consult people with diverse experiences, it is also necessary to consider their comfortability with the conversation. So how can we mindfully create these inclusive solutions?

The best practices to incorporate accessibility will be to:

Make the step toward improving your accessibility:

Improving your digital accessibility is not only important for the inclusion of people with disabilities, but it will also widen your audience by making your product attractive to more people.

Briteweb is committed to diversity and inclusion, which is why our web design projects partner with The Chicago Lighthouse’s DAX team. When developing your digital products with Briteweb, you will experience this holistic accessibility testing process in action.

Do you need help adapting your accessibility? Connect with the Briteweb team to start your commitment to inclusivity today.