Running version 3.0.0

Question

Running version 3.0.0

dmundra opened this issue 3 years ago · comments

After #128, I updated to 3.0.0 but I am getting a puppeteer error. Am I missing something?

npm run pa11y-ci:sitemap 

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude "/*.pdf"

Running Pa11y on 107 URLs:
 > http://127.0.0.1:4000/guide/documents - 2 errors
 > http://127.0.0.1:4000/guide/glossary - Failed to run
/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });
                                                              ^

Error: Protocol error (Target.disposeBrowserContext): Failed to find context with id 1573D003ABD6824A875BA4554C9131FA
    at /Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
    at Browser._disposeContext (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:168:32)
    at BrowserContext.close (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:523:29)
    at testRunner (/Users/dmundra/workspaces/accessibility/node_modules/pa11y-ci/lib/pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Node.js v17.0.1

Jose Bolos · Answer 1 · Mon Nov 29 2021 01:14:28 GMT+0800 (China Standard Time)

Hi @dmundra,

Could you please provide a public URL that we can test? It's impossible for us to debug the issue otherwise.

I've noticed you're using Node 17, could you please test it again using an LTS version of Node, as those are the only ones that we support?

If I'm reading your example correctly and the URL that you're testing has the same content as https://accessibility.civicactions.com/guide/glossary, this is working correctly with 3.0.0:

Broadly speaking, the error that you show there is a puppeteer error, so there's very little that we can do about it besides updating puppeteer to a latest version. I've seen similar errors in the past with puppeteer choking on servers that are configured in a specific way. If pa11y runs successfully on a public facing URL, but fails to run on a localhost one, this may be the same problem.

Daniel Mundra · Answer 2 · Mon Nov 29 2021 01:34:05 GMT+0800 (China Standard Time)

Hi @josebolos,

I am testing locally and in GitHub actions for the site you mentioned, CivicActions/accessibility#580 (comment)

I got the same error when trying it against the live sitemap.xml

./node_modules/pa11y-ci/bin/pa11y-ci.js --sitemap https://accessibility.civicactions.com/sitemap.xml
Running Pa11y on 119 URLs:
 > https://accessibility.civicactions.com/guide/events - 2 errors
 > https://accessibility.civicactions.com/guide/glossary - Failed to run
/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });
                                                              ^

Error: Protocol error (Target.disposeBrowserContext): Failed to find context with id F6EC73A10C23F6F16F75D9DCE662CC99
    at /Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
    at Browser._disposeContext (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:168:32)
    at BrowserContext.close (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:523:29)
    at testRunner (/Users/dmundra/workspaces/accessibility/node_modules/pa11y-ci/lib/pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Node.js v17.0.1

I am also running with axe but I get the error even when not running with axe. I will try different version of node as well.

I did find this puppeteer issue puppeteer/puppeteer#6258 if it helps. Maybe some of those versions needs to be changed for pa11y-ci.

Daniel Mundra · Answer 3 · Mon Nov 29 2021 03:01:19 GMT+0800 (China Standard Time)

Same error with node LTS version v16.13.0

Joey Ciechanowicz · Answer 4 · Mon Nov 29 2021 04:56:27 GMT+0800 (China Standard Time)

I couldn't re-create it locally by running

npx pa11y-ci --sitemap  https://accessibility.civicactions.com/sitemap.xml  --sitemap-exclude "/*.pdf"

If possible, could you share your pa11y-ci config file? And also the output of npx pa11y --environment

Dave Skender · Answer 5 · Mon Nov 29 2021 06:15:04 GMT+0800 (China Standard Time)

If possible, could you share your pa11y-ci config file? And also the output of npx pa11y --environment

This is @dmundra's config file for that site:

{
  "defaults": {
    "concurrency": 4,
    "standard": "WCAG2AA",
    "runners": ["axe"]
  }
}

He's also using ubuntu-latest in his test action, so it would be similar to my environment:

  System:
    OS: Linux 5.11 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
    Memory: 4.13 GB / 6.79 GB
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.13.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.1.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/npm
  pa11y: 6.1.1

I'm testing on a local GitHub Actions --detached site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).

For me, this runs fine on my local Windows 10 machine. On a side note, I'm running html-proofer on this same --detached site on ubuntu-latest without problems, so I don't think it's the host.

Dave Skender · Answer 6 · Mon Nov 29 2021 06:57:38 GMT+0800 (China Standard Time)

I suspect this has to do with concurrency. I changed my config file to "concurrency": 1 and it ran all the way through with Exit Code 2; however, it's producing false negatives that do not show up when I run locally.

Dave Skender · Answer 7 · Mon Nov 29 2021 07:17:04 GMT+0800 (China Standard Time)

Changing config "runners": ["axe"] to "runners": ["htmlcs"] solves my false negative problem.

Dave Skender · Answer 8 · Mon Nov 29 2021 07:24:37 GMT+0800 (China Standard Time)

In summary, there are likely two things that don't work in this scenario:

Concurrency > 1
Axe runner

The workaround is to change configurations in .pa11yci to:

{
  "defaults": {
    "concurrency": 1,
    "standard": "WCAG2AA",
    "runners": ["htmlcs"]
  }
}

Filippo Liverani · Answer 9 · Mon Nov 29 2021 20:04:09 GMT+0800 (China Standard Time)

Same error with node LTS version v16.13.0

Same issue on CircleCI's cimg/ruby:3.0-browsers latest image with node v16.13.0

/home/circleci/project/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });

.pa11yci:

{
  "defaults": {
    "concurrency": 4
  }
}

Daniel Mundra · Answer 10 · Mon Nov 29 2021 22:52:53 GMT+0800 (China Standard Time)

Thanks @DaveSkender.

I'm testing on a local GitHub Actions --detached site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).

I am doing the same in GitHub Actions, i.e. running against a local detached Jekyll site. The relevant files:

Daniel Mundra · Answer 11 · Mon Nov 29 2021 22:59:13 GMT+0800 (China Standard Time)

Confirmed that reducing concurrency to 1 allows the process to run to completion.

Aaron Goldenthal · Answer 12 · Fri Dec 10 2021 10:55:36 GMT+0800 (China Standard Time)

@joeyciechanowicz @josebolos I believe it's the same core issues I came across tonight working on a reporter, and in longer running reporter.afterAll functions (with a couple of reporters, with layers of async) it fails with:

E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71
            this._callbacks.set(id, { resolve, reject, error: new Error(), method });

Error: Protocol error (Target.disposeBrowserContext): Target closed.
    at E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:70:16)
    at Browser._disposeContext (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:168:32)
    at BrowserContext.close (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:523:29)
    at testRunner (E:\project\node_modules\pa11y-ci\lib\pa11y-ci.js:135:21)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

It looks to be a missing await at the specified line in the call stack - browserContext.close is an async function. Running with "useIncognitoBrowserContext": false seems to be a workaround since it avoids the line, but adding the await looks like it solved the core issue (at least in my case).

I have a PR if you want it, but it is just the one line.

Jose Bolos · Answer 13 · Thu Mar 24 2022 01:25:20 GMT+0800 (China Standard Time)

This should be fixen in v3.0.1 via #170 thanks to @aarongoldenthal.

Feel free to reopen the issue if you're still experiencing this problem.