Introducing: an Eleventy starter project for WCAG reports

When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that. In this post I’ll provide some context.

At eleventy-wcag-reporter, I have published my Eleventy starter for reports about conformance to the Web Content Accessibility Guidelines (WCAG). It is issue oriented, promotes rich text, comes with a PDF export script and supports two languages (English and Dutch).

There is an example report available.

Note: this is just one specific way to create accessibility conformance reports. This works for me and my clients. Other approaches and requirements exists, and I can’t guarantee this works for you.

Issue-oriented reporting#heading-1

WCAG conformance reports document whether a website meets the criteria in the WCAG standard. It will tell the reader which Success Criteria met the standard, which did not.

Many WCAG conformance reports list WCAG in its entirety and then specify the result for each. Say, a website is tested for WCAG 2.1, Level AA and 10 problems are found. The report will show all 50 Success Criteria (SCs): 10 say “Failed”, 40 “Passed” (or something else, like “Cannot Tell”). The accessibility problems are metadata of the success criteria, so to say.

In this Eleventy project, you’ll create one file for each issue

In my personal experience, teams that commission a report are mostly interested in the actual accessibility problems. They want to know “what needs fixing”. They may also want to know: how severe are the issues in comparison, which expertise can solve it (design? development?) and which assistive technology is affected. And, where relevant, what the issue or solution looks like in code. This is why I provide clients with an issue-oriented report: a list of issues, rather than a list of SCs. The Success Criterion is added as metadata of the issue, rather than the other way around.

A benefit of the issue-oriented approach, is that it can integrate better with a team’s workflow. Issue trackers are more common in workflows than SC-trackers. It also, as mentioned, makes it easier to include other meta data like responsibility and severity, and to include code examples and screenshots.

WCAG-EM, the standard methodology for accessibility conformance reporting, does not specify either of the two approaches. What is important for a conformance report is that all Success Criteria were evaluated, and that this fact is made clear. You want to be sure that “no issue filed for an SC” really means the sample did not contain any issues for that SC.

In other words: the report is only done when you’ve checked all sample pages for all WCAG Success Criteria in your scope (i.e. all 50 criteria for WCAG 2.1, Level A + AA). Always test them all. If your report follows the SC-based approach this is explicit, in the issue-based approach it is implicit.

The power of Markdown and YAML#heading-2

One of the benefits of using Eleventy for my reports, is that it makes it easy to write rich text. That may sound like a no-brainer, but in many reporting tools I’ve seen, issues are described in plain text. The main reason I want to use rich text to describe issues, is that they can make tips more clear. Lists, links, images, code examples and even text level stuff like bold and italics can make a report more readable. This is important, because it increases the chance the team can fix the issues found.

Markdown is my favourite syntactic sugar for rich text, so that’s what I use, but you can use whatever else. HTML should work, too.

For each issue that is described, there is also a set of meta data, like which Success Criterion it relates to. It is described in Yaml front matter, so at the top of an issue.md, you would add something like:

---
sc: 2.1.1
severity: High
difficulty: High
title: Main menu does not work with keyboard
sample: homepage
---

Spelling out the meta data in front matter makes it powerful. We can do things like add up the issues filed for a specific page, or count how many were filed for a given Success Criterion.

Some essential metadata in WCAG conformance reports#heading-3

Pointing out accessibility problems and solutions is the bread and butter of my conformance reports, as my goal is usually to help the team fix all issues. But accessibility reports are used in various contexts. There is some important metadata that helps someone looking at the report what’s going on, including monitoring bodies like governments. They may require publication of accessibility conformance reports to help them perform their regulatory duties, good metadata aids this monitoring.

Logius, the Dutch government department that keeps track of WCAG conformance reports that Dutch governmental organisations provide, uses a checklist to verify if conformance reports meet their requirements:

• is the methodology (eg WCAG-EM) mentioned?
• is the scope mentioned (domain and/or sub domains that are in scope, and those outside scope)
• does the report declare that all 50 Success Criteria in WCAG 2.1, Level A + AA were evaluated?
• is the level of accessibility support described (for instance: “the website can be used by all common browsers and assistive technologies“)?
• is there a list of technologies used (eg HTML, CSS, SVG, etc)
• is there a list of pages that were included in the evaluated sample?
• is the person or company that did the evaluation listed?
• are browsers and other software, including versions, listed?
• is it dated?
• are the results published in a human-readable or machine-readable format (EARL)?

The above is my translation of the checklist on the Controle door Logius page; many of these are also recommended in WCAG-EM; the requirements likely vary where you or your client are based.

The eleventy-wcag-reporter starter project has variables for each of these checkpoints. A lot of the meta data is added to the report’s index.njk as YAML front matter, so that it can be displayed.

FAQ#heading-4

After reading this, you may have questions before trying to use the code. The questions below weren’t actually asked frequently, but I hope they have useful answers.

Will this find accessibility issues?

No, you find issues, create a Markdown file for each and then add some frontmatter. The code will then include it in a pretty report. Or really, a report that you can make pretty.

Why not integrate with automated tooling?

There are tools that can find issues, for ~20% of WCAG criteria (other estimates exist). You can use those tools to find issues and add them manually. It is usually easier for a human to group issues in a way that is most useful to readers of the report. For instance, if a brand’s primary colour contrasts badly with a white background, you may not need to file that more than once.

Can more languages be supported?

Yes, feel free to contribute. We need all of WCAG in JSON format (I have code to scrape) and translations for strings.

How many issues should I add to a report?

There is no minimum or maximum, but you should check all pages in your sample for all Success Criteria in your scope, and have issues for all situations where a SC is not met, in order to create a full report.

What about PDFs?

I like HTML as a report format. Some organisations prefer to receive a PDF document. Like web pages, PDFs also need to be accessible, and simply pressing “Save as PDF“ in a browser will be unlikely to include all semantics. So I use PrinceXML, which supports a lot of print CSS. There are licenses for a machine or server, or you can use the SaaS version DocRaptor. The starter project has a Shell script to generate a PDF with PrinceXML, you will need a license or end up with a watermark in your resulting PDF.

Wrapping up#heading-5

So, find my starter project at eleventy-wcag-reporter, please don’t laugh at my code. Feedback is very much welcomed by email, DMs or in GitHub. I may not be able to get back very quickly, as the open version of this is a side project. Happy accessbility conformance report writing!