Introducing: an Eleventy starter project for WCAG reports

When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that. In this post I’ll provide some context.

At eleventy-wcag-reporter, I have published my Eleventy starter for reports about conformance to the Web Content Accessibility Guidelines (WCAG). It is issue oriented, promotes rich text, comes with a PDF export script and supports two languages (English and Dutch).

Example report shows an issue with code example, a listing of wcag criterion, severity and difficulty and what the sample is There is an example report available.

Note: this is just one specific way to create accessibility conformance reports. This works for me and my clients. Other approaches and requirements exists, and I can’t guarantee this works for you.

Issue-oriented reporting

WCAG conformance reports document whether a website meets the criteria in the WCAG standard. It will tell the reader which Success Criteria met the standard, which did not.

Many WCAG conformance reports list WCAG in its entirety and then specify the result for each. Say, a website is tested for WCAG 2.1, Level AA and 10 problems are found. The report will show all 50 Success Criteria (SCs): 10 say “Failed”, 40 “Passed” (or something else, like “Cannot Tell”). The accessibility problems are metadata of the success criteria, so to say.

Screen3 In this Eleventy project, you’ll create one file for each issue

In my personal experience, teams that commission a report are mostly interested in the actual accessibility problems. They want to know “what needs fixing”. They may also want to know: how severe are the issues in comparison, which expertise can solve it (design? development?) and which assistive technology is affected. And, where relevant, what the issue or solution looks like in code. This is why I provide clients with an issue-oriented report: a list of issues, rather than a list of SCs. The Success Criterion is added as metadata of the issue, rather than the other way around.

A benefit of the issue-oriented approach, is that it can integrate better with a team’s workflow. Issue trackers are more common in workflows than SC-trackers. It also, as mentioned, makes it easier to include other meta data like responsibility and severity, and to include code examples and screenshots.

WCAG-EM, the standard methodology for accessibility conformance reporting, does not specify either of the two approaches. What is important for a conformance report is that all Success Criteria were evaluated, and that this fact is made clear. You want to be sure that “no issue filed for an SC” really means the sample did not contain any issues for that SC.

In other words: the report is only done when you’ve checked all sample pages for all WCAG Success Criteria in your scope (i.e. all 50 criteria for WCAG 2.1, Level A + AA). Always test them all. If your report follows the SC-based approach this is explicit, in the issue-based approach it is implicit.

The power of Markdown and YAML

One of the benefits of using Eleventy for my reports, is that it makes it easy to write rich text. That may sound like a no-brainer, but in many reporting tools I’ve seen, issues are described in plain text. The main reason I want to use rich text to describe issues, is that they can make tips more clear. Lists, links, images, code examples and even text level stuff like bold and italics can make a report more readable. This is important, because it increases the chance the team can fix the issues found.

Markdown is my favourite syntactic sugar for rich text, so that’s what I use, but you can use whatever else. HTML should work, too.

For each issue that is described, there is also a set of meta data, like which Success Criterion it relates to. It is described in Yaml front matter, so at the top of an issue.md, you would add something like:

---
sc: 2.1.1
severity: High
difficulty: High
title: Main menu does not work with keyboard
sample: homepage
---

Spelling out the meta data in front matter makes it powerful. We can do things like add up the issues filed for a specific page, or count how many were filed for a given Success Criterion.

Some essential metadata in WCAG conformance reports

Pointing out accessibility problems and solutions is the bread and butter of my conformance reports, as my goal is usually to help the team fix all issues. But accessibility reports are used in various contexts. There is some important metadata that helps someone looking at the report what’s going on, including monitoring bodies like governments. They may require publication of accessibility conformance reports to help them perform their regulatory duties, good metadata aids this monitoring.

Logius, the Dutch government department that keeps track of WCAG conformance reports that Dutch governmental organisations provide, uses a checklist to verify if conformance reports meet their requirements:

  • is the methodology (eg WCAG-EM) mentioned?
  • is the scope mentioned (domain and/or sub domains that are in scope, and those outside scope)
  • does the report declare that all 50 Success Criteria in WCAG 2.1, Level A + AA were evaluated?
  • is the level of accessibility support described (for instance: “the website can be used by all common browsers and assistive technologies“)?
  • is there a list of technologies used (eg HTML, CSS, SVG, etc)
  • is there a list of pages that were included in the evaluated sample?
  • is the person or company that did the evaluation listed?
  • are browsers and other software, including versions, listed?
  • is it dated?
  • are the results published in a human-readable or machine-readable format (EARL)?

The above is my translation of the checklist on the Controle door Logius page; many of these are also recommended in WCAG-EM; the requirements likely vary where you or your client are based.

The eleventy-wcag-reporter starter project has variables for each of these checkpoints. A lot of the meta data is added to the report’s index.njk as YAML front matter, so that it can be displayed.

FAQ

After reading this, you may have questions before trying to use the code. The questions below weren’t actually asked frequently, but I hope they have useful answers.

Will this find accessibility issues?
No, you find issues, create a Markdown file for each and then add some frontmatter. The code will then include it in a pretty report. Or really, a report that you can make pretty.
Why not integrate with automated tooling?
There are tools that can find issues, for ~20% of WCAG criteria (other estimates exist). You can use those tools to find issues and add them manually. It is usually easier for a human to group issues in a way that is most useful to readers of the report. For instance, if a brand’s primary colour contrasts badly with a white background, you may not need to file that more than once.
Can more languages be supported?
Yes, feel free to contribute. We need all of WCAG in JSON format (I have code to scrape) and translations for strings.
How many issues should I add to a report?
There is no minimum or maximum, but you should check all pages in your sample for all Success Criteria in your scope, and have issues for all situations where a SC is not met, in order to create a full report.
What about PDFs?
I like HTML as a report format. Some organisations prefer to receive a PDF document. Like web pages, PDFs also need to be accessible, and simply pressing “Save as PDF“ in a browser will be unlikely to include all semantics. So I use PrinceXML, which supports a lot of print CSS. There are licenses for a machine or server, or you can use the SaaS version DocRaptor. The starter project has a Shell script to generate a PDF with PrinceXML, you will need a license or end up with a watermark in your resulting PDF.

Wrapping up

So, find my starter project at eleventy-wcag-reporter, please don’t laugh at my code. Feedback is very much welcomed by email, DMs or in GitHub. I may not be able to get back very quickly, as the open version of this is a side project. Happy accessbility conformance report writing!

Thanks to Roel, Peter, Rian, Jeroen and many others for discussions, ideas and suggestions related to accessibility conformance reports. (Thanks do not imply endorsements)

Comments, likes & shares (185)

Ross Mullen, Terry O'Brien, Sara Soueidan, Joe Lanman, Matthias Ott, Myles Lewando, Michael Hastrich, Starfall Projects, Anneke Sinnema, Ronald Aveling, Gaël Poupard, Sarah Federman, Sami Keijonen, Wassim Chegham 🌱, Peter van Grieken, Matthew Holloway | Black Lives Matter, Guillaume Deblock, Princiya 👧‍, Martin Laker Ockovsky, Robert Spier, Tommie Woef, Alicia Jarvis (She/Her/Hers), CPACC, CSM 🇨🇦, Ruben Bos, Taeke, Arturo Castillo Delgado, Grigor 🇵🇸, Søren Birkemeyer 🦊, Darek Kay, Stephanie Eckles, Tomas Pustelnik, Katie Langerman, Joan Joly, Adam Juran, Francesco Giudice, Karmugil, Jens Geiling, PK 🐢 👩🏻‍💻, Fotis Papadogeorgopoulos, Brett Jankord, emily 🪱🌔, Bernard Nijenhuis, Dan, Towhid, Vincent, Gaëtan, Pat Ramsey “it STILL goes over the nose”, Matsuko, Mario Vasquez, Rafael Calvo 🔰, Jess Peck 🐍👾✨, 👻 Zac Vineyard, Ruben Nic, Erwin Verbruggen, Florian Geierstanger, Dana Byerly, Eleventy, Phil Wolstenholme, Phillip Faniola, mason conkright, corey benson, lisica, ned, Juha Karttunen, Niek, Jens Grochtdreis, Patrick Grey, Dirk Sidney Jansen, Károly Szántai, Jeremy Van Staalduine, Sven Wolfermann, Saptak Sengupta, Scott McKee, Bunch, audrey maniez, Danisaurio, Colin, Matt Hobbs, Karan Ganesan, T Carter Baxter, Marco useCauseAndEffect(), Zach Leatherman, AJ Klein, Patrick Haug, Yannis Güdel, Heather, Hélio Correia, Iain Bean, Riccardo Erra, Dennis Lembrée, Peter Antonius, ASIC Rocky 🪙⛏, Dave Cunningham, Honkel Steffi, Jasper Moelker 🔋🌳🇪🇺, Juan Javier, Scifibrarian, MS, MFA, Peter, Who me?, Alex Edwards, Christian Gloddy, Ollie Boermans, Natalie Chin, Stephanie Hagadorn, TRST_Blog, Martin Underhill, Bogdan Cerovac, Brian Crumley, Jamie and Lion, a 🧈, Marcos de Miguel, Laura Whitehead 💙, Gode vinkjøp, negi4a, João Figueiredo, Corey Megown, Clare Hyam (she/her), Bruno Pulis, Melanie Sumner, Craig Abbott, Chris, Stephanie Eckles, Gaël Poupard, Matthias Weston, Somascope, Carie Fisher, Pauline Vos, Tom Hermans🤘🌈, Conor and Joe Lanman liked this

Starfall Projects, Michael Hastrich, Sami Keijonen, Sarah Federman, Alicia Jarvis (She/Her/Hers), CPACC, CSM 🇨🇦, Comandeer, Jeanine Kay Lineback, mallory, alice & bob, Darice de Cuba, Yakim, Chris Power, Rian Rietveld, Jess Peck 🐍👾✨, Eleventy, Bram Smulders, Saravana, Juha Karttunen, Patrick Grey, Sven Wolfermann, Scott McKee, Eric Bailey, Zach Leatherman, Who me?, Hadley, Marcos de Miguel, Clare Hyam (she/her) and Web Axe reposted this

Myles Lewando wrote on 24 May 2021:
Ooooo nice!
Robbert wrote on 24 May 2021:
I love that every issue can be linked to with the # fragment identifier! Would like to link to relevant issues on the component backlog for NL Design System. No more PDFs!
Wassim Chegham 🌱 wrote on 24 May 2021:
Hey @eleven_ty folks, check this out!
Peter van Grieken wrote on 24 May 2021:
I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Sophie wrote on 24 May 2021:
wow!
Wassim Chegham 🌱 wrote on 24 May 2021:
That's great! Thank you for sharing.
Hidde wrote on 24 May 2021:
heb de ”Set up” zojuist vast ietsje uitgebreid, maar ik stuur je ff een DM voor support! :-)
Jules Ernst wrote on 24 May 2021:
Just missing the installation for very dummies. It's me I know, but tried hard to get it installed with no success.
Sara Soueidan wrote on 24 May 2021:
An @eleven_ty starter project for WCAG reports, kindly open-sourced by @hdv Code: github.com/hidde/eleventy… Intro & about: hiddedevries.nl/en/blog/2021-0… #a11y
Peter van Grieken wrote on 24 May 2021:
I’ve been using this project for a while and it has made writing WCAG reports so much easier for me.
Hidde wrote on 24 May 2021:
thank you!
Colin wrote on 24 May 2021:
nice work!
Allen vaccinated Hoffman! wrote on 24 May 2021:
Nicely done. Take a look at ACRT Accessibility Conformance Reporting Tool from the U.S. FEeral government. It uses JSON to store the test checkpoints and also the report data for sharing.
Peter wrote on 24 May 2021:
Good work pal. I’ll look forward to kicking the tyres on this 👍🏻
Danisaurio wrote on 24 May 2021:
Hi! Loved your project and I'll add spanish support :) thanks for doing this!
Sun lover wrote on 25 May 2021:
This is an excellent #ally resource! Thank you so much for sharing.
bertrandkeller wrote on 25 May 2021:
hiddedevries.nl/en/blog/2021-0… - belle initiative, mais un peu compliqué à suivre dans le temps. RGAA C’est 106 critères, 106 fichiers MarkDown a éditer, c’est un peu lourd.
Shawn Thompson wrote on 25 May 2021:
Thanks @mgifford for sharing this with me!
Priya wrote on 25 May 2021:
Introducing: an Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0…
Pablo Lara H wrote on 31 May 2021:
Introducing: an Eleventy starter project for WCAG reports by Hidde de Vries @hdv #Eleventy #Eleventystarterproject #WCAG #webdev #WebContentAccessibilityGuidelines hiddedevries.nl/en/blog/2021-0…
Max Antonucci wrote on 2 June 2021:
So @hdv shared their @eleven_ty template for WCAG reports, and it is pretty damn awesome - hiddedevries.nl/en/blog/2021-0… #a11y #accessibility
Bruno Pulis wrote on 2 June 2021:
Awesome initiative @hdv soon as possible I'll send a translation to Brazilian portuguese! :)
Ben Myers 🦖 wrote on 2 June 2021:
Ohhh, I love this.
Melanie Sumner wrote on 2 June 2021:
This is very cool.
Web Axe wrote on 27 July 2021:
An Eleventy starter project for WCAG reports hiddedevries.nl/en/blog/2021-0… #wcag #a11y #dev #reports
Hidde wrote on 19 August 2021:
Haha this sounds very close to home! Makes a lot of sense.
Carie Fisher wrote on 30 August 2021:
😻😻😻
Joe Lanman wrote on 7 September 2021:
awesome, thanks!
The A11Y Project wrote on 20 October 2021:
"When I audit websites for accessibility, I write Markdown files that Eleventy turns into reports. Today, I’ve shared the code I use for that." @hdv hiddedevries.nl/en/blog/2021-0…