Tests don't fail gracefully when encountering errors after test execution and stall the CI

Question

Tests don't fail gracefully when encountering errors after test execution and stall the CI

sethvanwykJET opened this issue 4 months ago · comments

What are you trying to achieve?

Test pipelines should continue gracefully when encountering an error during "after" test execution

What do you get instead?

When the test fails, codeceptJS fails to properly scale down the test run and keeps the browser session open, essentially turning the test thread into a zombie codeceptJS process.

Tests running in pipelines will time out as the thread that's running these tests don't know the test has completed and that they should start the next test.

   Requesting menu-item sold-out status change to false
      I send put request "https://uri/api/v2/restaurant/1289255/category/15516284/product/487560132", {"name":"Sold Out","description":"Will be changed to sold out by automated tests","external_id":"","sort_id":4,"delivery_price":12,"pickup_price":12,"shipping_method":"pickup_delivery","sold_out":false,"image_url":null,"soft_deleted":false,"addon_set_id":null}, {"accept":"application/json","content-type":"application/json","Authorization":"Basic =="}
      › [Request] {"baseURL":"https://uri/api/v2/restaurant/1289255/category/15516284/product/487560132","method":"PUT","data":{"name":"Sold Out","description":"Will be changed to sold out by automated tests","external_id":"","sort_id":4,"delivery_price":12,"pickup_price":12,"shipping_method":"pickup_delivery","sold_out":false,"image_url":null,"soft_deleted":false,"addon_set_id":null},"headers":{"Content-Type":"application/json","Accept":"application/json","X-Session-Id":"8c7db2c0a86b","user_agent":"cw-api-automated-tests","sec-fetch-mode":"cors","access-control-allow-origin":"*","accept":"application/json","content-type":"application/json","Authorization":"Basic =="}}
      › [Response] Response error. Status code: 404
      › [Response] "404 page not found\n"
404 page not found

    [2] Error | Error: Restaurant Menu API call failed! Response: undefined
             Status: 404 (e) => { const err = (recorder.getAsyncErr() === n...
Error processing test.after event:
TypeError: Cannot read properties of undefined (reading 'ctx')
    at AllureReporter.allureTestByCodeceptTest (/Users/seth.vanwyk/workspace/project/node_modules/allure-codeceptjs/dist/index.js:89:62)
    at AllureReporter.testAfter (/Users/seth.vanwyk/workspace/project/node_modules/allure-codeceptjs/dist/index.js:159:33)
    at EventEmitter.emit (node:events:402:35)
    at Object.emit (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/event.js:149:28)
    at errHandler (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:116:39)
    at /Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:162:11
    at /Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/recorder.js:239:9
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    [2] <teardown> Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /Users/seth.vanwyk/workspace/project/output/Desktop_-_item_is_sold_out_after_each_hook__After_for_should_remove_item_that_is_sold_out_from_basket_@site-nl_@delivery_@not-allowed-to-run-in-production_@should-remove-item-that-is-sold-out-from-basket.failed.png
  ✖ FAILED in 434ms

    [3]  Starting recording promises

-- FAILURES:

  1) Desktop - item is sold out
       should remove item that is sold out from basket @site-nl @delivery @not-allowed-to-run-in-production @should-remove-item-that-is-sold-out-from-basket:
     Error: Restaurant Menu API call failed! Response: undefined
             Status: 404
      at Object.changeMenuItemSoldOutStatus (api-calls/restaurant-menu-api.js:36:15)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Test.<anonymous> (/Users/seth.vanwyk/workspace/project/checkout-e2e/scenarios/multi-step-checkout/item-out-is-sold-out-alert-desktop-test.js:52:3)

  Scenario Steps:
  - I.seeElement({"xpath":"//*[@data-qa='jet-privacy-settings-button-full-consent']"}) at ./page-objects/footer/footer.js:16:41
  - I.seeElement({"xpath":"//*[@data-qa='privacy-settings-action-close']"}) at ./page-objects/footer/footer.js:15:42
  - I.say("Checking which cookie banner variant is in use") at Object.acceptCookies (./page-objects/footer/footer.js:14:7)
  - I.say("Opened page https://site/en/menu/test-restaurant-selenium?showTestRestaurants=true&utm_medium=e2e_testing&utm_source=cwplatform&utm_campaign=local&e2e=true&timestamp=1708522969025 at Wed Feb 21 2024 13:42:49 GMT+0000 (Coordinated Universal Time)") at Actor.accessURL (./steps-file.js:174:12)
  - I.grabCurrentUrl() at Actor.accessURL (./steps-file.js:173:37)
  - I.waitForInvisible({xpath: //*[contains(@data-qa,'skeleton')]}, 30) at ./steps-file.js:210:18
  - I.say("Awaiting for 30 seconds for skeleton to not be visible") at Actor.waitPageLoads (./steps-file.js:206:14)
  - I.waitForFunction(async () => {
          const state = await document.readyState;
          return state === 'interactive' || state === 'complete';
        }, 120) at Actor.waitPageLoads (./steps-file.js:197:12)
  - I.say("Awaiting for page to load with javascript") at Actor.waitPageLoads (./steps-file.js:196:12)
  - I.amOnPage("https://site/en/menu/test-restaurant-selenium?lat=53.0651908&lng=16.04760159&showTestRestaurants=true&utm_medium=e2e_testing&utm_source=cwplatform&utm_campaign=local&e2e=true&timestamp=1708522969025") at Actor.accessURL (./steps-file.js:167:14)
  - I.say("Opening the page") at Actor.accessURL (./steps-file.js:164:12)
  - I.sendGetRequest("/restaurant?slug=test-restaurant-selenium", {"X-Country-Code":"pl","Accept-Language":"en","X-Language-Code":"en"}) at getTakeawayRestaurantOpening (./api-calls/restaurant.js:6:28)

  Artifacts:
  - screenshot: /Users/seth.vanwyk/workspace/project/output/should_remove_item_that_is_sold_out_from_basket_@site-nl_@delivery_@not-allowed-to-run-in-production_@should-remove-item-that-is-sold-out-from-basket.failed.png

  2) Desktop - item is sold out
       "after each" hook: After for "should remove item that is sold out from basket @site-nl @delivery @not-allowed-to-run-in-production @should-remove-item-that-is-sold-out-from-basket":
     Error: Restaurant Menu API call failed! Response: undefined
             Status: 404
      at Object.changeMenuItemSoldOutStatus (api-calls/restaurant-menu-api.js:36:15)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (/Users/seth.vanwyk/workspace/project/checkout-e2e/scenarios/multi-step-checkout/item-out-is-sold-out-alert-desktop-test.js:75:3)
      at async promiseRetry.retries.retries (/Users/seth.vanwyk/workspace/project/node_modules/codeceptjs/lib/scenario.js:143:9)




  FAIL  | 0 passed, 2 failed   // 27s

The terminal stalls here and leaves a broken browser session open that doesn't close anymore.

Provide test source code if related

    if (response.data.message !== 'success') {
        console.log(response.data)
        throw new Error(
            `Restaurant Menu API call failed! Response: ${response.data.message}
             Status: ${response.status}`
        );

Details

CodeceptJS version: Tested on 3.5.3 and 3.5.14
NodeJS Version:16
Operating System: MacOS
puppeteer

Horsty · Answer 1 · Mon Mar 11 2024 17:14:37 GMT+0800 (China Standard Time)

Interesting @sethvanwykJET , i'm not sure i have same issue, but have you this kind of error message in report allure or terminal ?

element ({xpath: //*[@data-e2e = 'custom-id-e2e'][@data-e2e-type = 'custom-type']}) still not present on page after 12 sec
locator.waitFor: Target page, context or browser has been closed

or like that

locator.all: Target page, context or browser has been closed

Seth · Answer 2 · Mon Mar 11 2024 17:17:37 GMT+0800 (China Standard Time)

Interesting @sethvanwykJET , i'm not sure i have same issue, but have you this kind of error message in report allure or terminal ?
element ({xpath: //*[@data-e2e = 'custom-id-e2e'][@data-e2e-type = 'custom-type']}) still not present on page after 12 sec
locator.waitFor: Target page, context or browser has been closed
or like that
locator.all: Target page, context or browser has been closed

I've also been seeing those errors, but they're not necessarily the same as this.

Those don't stall the pipeline / stop a test from retrying itself on my end.

Horsty · Answer 3 · Mon Mar 11 2024 17:32:13 GMT+0800 (China Standard Time)

I have both in my case, the pipeline are stuck and i get this kind of error.
So my assumption is that are connected but i don't understand why.

I will check about throw exception

That happens when i go to the 200th test all the begining works fine

Seth · Answer 4 · Mon Mar 11 2024 17:38:49 GMT+0800 (China Standard Time)

I believe my condition relates to a test failing in the after() section

After(async () => {
  throw new error goes here
});

Horsty · Answer 5 · Mon Mar 11 2024 17:40:30 GMT+0800 (China Standard Time)

Ok i will check that, i have teardown on after with async request that can lead to throw error
same in before i have setup that can lead to throw error

Seth · Answer 6 · Mon Mar 11 2024 17:44:08 GMT+0800 (China Standard Time)

I've checked if the fail happens in the before, but that seems to be fine

It's only when test execution fails in after that it fails to close the test

I tried to explain the test failing in after in the base description, but because of the word after it's a bit difficult to narrow it down through words without some weird grammar gymnastics