tests do not fail on a11y violations

[Sarbbottam Bandyopadhyay]

the errors get logged into the browser's console (ember test --serve) but tests do not fail.

[Bill Pullen]

@sarbbottam which version of ember, ember-cli, & ember-a11y-testing are you using? I was able to get my project to fail on errors by switching back to 0.1.4

[Sarbbottam Bandyopadhyay]

I'm using 0.1.6.

I just tried with 0.1.4, but tests didn't fail.

[Bill Pullen]

This seems to be an issue with axe-core for me anything above 1.1.1 simply logs the error to the console rather than reporting it to Qunit.

[Sarbbottam Bandyopadhyay]

I tried with axe-core@1.1.0 and the tests seems to fail. However, I can't seem to identify what is th reason for failure from the output.

ember test

Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
Built project successfully. Stored in "/Users/sabandyo/GitCorp/ember-a11y-testing-demo/tmp/core_object-tests_dist-JsLmnkfl.tmp".
not ok 1 PhantomJS 2.1 - Global error: Error: Assertion Failed: The page should have no accessibility violations. Please check the developer console for more details. at http://localhost:7357/assets/vendor.js, line 17311
    ---
        Log: |
            { type: 'error',
              text: 'Error: Assertion Failed: The page should have no accessibility violations. Please check the developer console for more details. at http://localhost:7357/assets/vendor.js, line 17311\n' }
    ...
not ok 2 PhantomJS 2.1 - Acceptance | welcome: visiting /
    ---
        actual: >
            null
        stack: >
            http://localhost:7357/assets/vendor.js:17311
        message: >
            Error: Assertion Failed: The page should have no accessibility violations. Please check the developer console for more details.
        Log: |
    ...
ok 3 PhantomJS 2.1 - JSHint | acceptance/welcome-test.js: should pass jshint
ok 4 PhantomJS 2.1 - JSHint | app.js: should pass jshint
ok 5 PhantomJS 2.1 - JSHint | helpers/destroy-app.js: should pass jshint
ok 6 PhantomJS 2.1 - JSHint | helpers/module-for-acceptance.js: should pass jshint
ok 7 PhantomJS 2.1 - JSHint | helpers/resolver.js: should pass jshint
ok 8 PhantomJS 2.1 - JSHint | helpers/start-app.js: should pass jshint
ok 9 PhantomJS 2.1 - JSHint | resolver.js: should pass jshint
ok 10 PhantomJS 2.1 - JSHint | router.js: should pass jshint
ok 11 PhantomJS 2.1 - JSHint | test-helper.js: should pass jshint

1..11
# tests 11
# pass  9
# skip  0
# fail  2
Not all tests passed.
Error: Not all tests passed.
    at EventEmitter.getExitCode (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/lib/app.js:434:15)
    at EventEmitter.exit (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/lib/app.js:189:23)
    at /Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/lib/app.js:103:14
    at tryCatcher (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/promise.js:691:18)
    at Async._drainQueue (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/async.js:138:16)
    at Async._drainQueues (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/async.js:148:10)
    at Immediate.Async.drainQueues (/Users/sabandyo/GitCorp/ember-a11y-testing-demo/node_modules/ember-cli/node_modules/testem/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)
npm ERR! Test failed.  See above for more details.

[Bill Pullen]

You have to open up the console inside chrome inspector and look at the violations

[Sarbbottam Bandyopadhyay]

I am trying to get this into the build pipeline with a headless browser. So I have to solely rely on the console log.

[Iain D Broadfoot]

I can't see an issue on the axe repository relating to this; should we file one?

[Buck Doyle]

I used ember-axe in an application and forked it to add console logging of aXe violations, maybe something like that can be used here. You can see the code here. I hope to use ember-a11y-testing soon and could look at incorporating an option like this if there’s interest.

[Trent Willis]

This has been fixed and released in v0.2.0.