Improve logging and debugging of violations

[Nathan Hammond]

Initial implementation of logging for the audits has been minimal. Currently, it just outputs the violations in numeric order follow by a dump of the object. Example:

This isn't the worst, but can definitely be made better before the official 1.0 release. Following are proposed features for better logging of violations (subject to change):

General Improvements

• Present information in an easy-to-digest format before dumping the object.
  • Potential format: [impact]: [help]. Offending markup is: [nodes.html].
  • Ex: CRITICAL: Form elements must have labels. Offending markup is: <input id="test" type="test">.

Component Improvements

• Include the Component name/reference (e.g., component.toString()) so that it is clear who the offender was
  • Provide hook to break on a specific violation

Acceptance Test Improvements

• For Acceptance Test audits, specify the run number of the audit
  • Provide hook to break on a specified run number to aid in debugging

[Ken Sin]

Not sure if there's already a way to do this in development, but perhaps an object that keeps track of all violations and can be accessed in the console? Ideally, it updates as a violation appears.

ie: After submitting a form, an error message appears where the error message triggers a violation. The violations tracker is then updated with the violation or reference of the violation. The dev can call something like violations.count or violations.lastObject to query for any state changes to the tracker.

My main desire for this is that when I see a ton of violations on a page, I feel a bit overwhelmed seeing something like:

-- especially as a first-time user of the addon and newbie to a11y concepts. I feel like a simple API like this would make seeing a ton of violations on a page at once a little easier to manage.

Anyway, I'm sure the improvements listed above will help a lot, but just wanted to throw my suggestion in the box! If there's an easy way to get something like this up and you can point me in the right direction, I'd be happy to work on a quick PR.

(I know it's my own fault for letting the violations get so bad... 😭)

[Nathan Hammond]

@ksin Hokay, that's a complete and total mess, I agree. Would you be interested in creating a util object for this addon that has a few helper methods?

Proposed approach: /addon/utils/violationHelper.js

let violations = [];

var thing = Object.create(Array);

// Doesn't work, but you get my point.
thing.prototype = {
  add(violation) { violations.push(violation); },
  filter(node || rule) {},
  yourOtherGoodIdeas() {}
}
export default thing; 

Then in the initializer you can import it and push it into global or app space, making it available in the user's console, possibly logging how to do it the first time a violation gets added to it. What do you think?

[Melanie Sumner]

I'm getting the same violation on the same element reported multiple times in the console. Is this related to this issue or is it a different issue @nathanhammond ?

[Brian Sipple]

@MelSumner Interesting. I caught similar behavior related to audits being run multiple times here. Perhaps that's also related.

[Michael Rarela]

FYI
It appears that some of the improved logging fixes were removed in May 2016... from Issue #2.

[Trent Willis]

@majew7 during that time ember-a11y-testing underwent a major rewrite. There was previously another addon named ember-axe which, similarly to ember-a11y-testing, provided automated accessibility testing but leveraged the much more robust axe-core library.

Thus, since axe-core is much more full-featured and has dedicated support from a company, we elected to adopt the approach ember-axe took and merged it into ember-a11y-testing.

This issue was then opened because we were aware that the logging provided by the new approach wasn't as approachable and robust. The PR #78 should improve this situation (I just keep forgetting to review it).

[Trent Willis]

Just merged #78. Will release shortly, but this issue will remain open as there are a few other tasks still.

[Michael Rarela]

Thanks @trentmwillis, I appreciate the quick reply and merge. I just did npm install and confirmed that I have the latest v0.4.3, but I'm still not seeing the improvement yet. Here's what I found...

I observe that:

• Older 'Offending Markup (#)' string is in function a11yAuditCallback() called from method runA11yAudit(). This low information log is still executing (See image).
• The new 'Offending markup is' string is in function audit()audit... but where it's called from isn't clear. It appears to be a code path not executed.

Am I calling the code incorrectly, or missing a setting?

I'm using import a11yAudit from 'ember-a11y-testing/test-support/audit';

[João Farias]

Same issue than @majew7

[Sean Massa]

You are seeing that because the test log output wasn't updated to the new format. I've updated this in a new PR: #103