Running test suites and channel selection

Question

Running test suites and channel selection

you1anna opened this issue 4 years ago · comments

Robin Miklinski commented 4 years ago

Hi,
I have a couple of questions about the tool:

When using npx cypress-slack-reporter or index.ts is it possible to run only a subset of tests from package.json? E.g.
"cy:run:pr": "cypress run --spec 'cypress/integration/PR/*'"
Is there a way to pass in the Slack channel results are posted to?

Thanks,
Robin

Yousaf Nabi · Answer 1 · Tue Jun 23 2020 07:13:32 GMT+0800 (China Standard Time)

Hey,

The running of the tests is seperate to the invocation of the slack reporter.If you run your command cy:run:pr and then npx cypress-slack-reporter, do you not get the desired result?
I believe the webhooks are specific to a channel when created, but stand to be correct. If so, it would be simple to pass a channel value, either name or id. I'd just need to check the slack api

Cheers,
saf

Yousaf Nabi · Answer 2 · Tue Jun 30 2020 07:49:38 GMT+0800 (China Standard Time)

@you1anna what is your use case for posting to a different channel?

Would you want to post to a different channel on failure? If so we could support different webhook urls for test pass / test fail and build fail?

I assume issue 1 is no longer an issue?

Robin Miklinski · Answer 3 · Wed Jul 01 2020 16:47:21 GMT+0800 (China Standard Time)

Hey @YOU54F, one of our use cases is to post to various channels depending on the result, but we also want to post to multiple channels at the same time. Unsure if those are handled by the Slack API.

The first issue is resolved, thanks.

Yousaf Nabi · Answer 4 · Wed Jul 01 2020 17:20:23 GMT+0800 (China Standard Time)

No it doesn't, you would need to provide a seperate webhook per channel iirc and if you wanted to email multiple channels, you can execute the reporter multiple times with a different webhook.

We could support a distinct webhook per result (build fail/test fail/test pass) and support an array of webhooks. If there is an array, the message could be sent once for each webhook?

Robin Miklinski · Answer 5 · Thu Jul 02 2020 04:18:42 GMT+0800 (China Standard Time)

Yes that should work, that would allow us to supply as many webhooks as are needed, and have results sent to as many channels as necessary per test run.

Robin Miklinski · Answer 6 · Sun Jul 05 2020 04:30:11 GMT+0800 (China Standard Time)

Hi @YOU54F, regarding the first issue I notice the following command

yarn cypress run --spec 'cypress/integration/PR/test.spec.js' && npx cypress-slack-reporter

gives an error Error: Cannot find test report @ mochareports

My package.json is using the same dep versions as the one in this repo.

Running
yarn cypress run --spec 'cypress/integration//PR/test.spec.js' && npx ts-node script.ts

Will run the reporter but also seems to run the full test suite in addition to test.spec.js

I have exported the webhook first using export SLACK_WEBHOOK_URL=xxxso not sure why this is failing. Your solution to issue 1 makes sense in theory, haven't seen it working yet however.

Yousaf Nabi · Answer 7 · Sun Jul 05 2020 17:51:59 GMT+0800 (China Standard Time)

Can you show me the contents of your script.ts?

When you run your PR, where is your test report created? Have you created a mochawesome test report in json format? If so you need to tell the slack reporter where your report lives.

With regards to point 2, I have released a new version 0.10.0 which will support a webhook per status ( build fail / test fail / test pass ) and any webhook env var can support multiple webhooks separates by a comma

Robin Miklinski · Answer 8 · Sun Jul 05 2020 19:08:20 GMT+0800 (China Standard Time)

Am running it locally using spec.ts from this repo, in the root of my repo.
There are existing test reports in cypress/reports/mocha/mochawesome.json

Also tried with
yarn cypress run --spec 'cypress/integration/PR/test.spec.js' && ./node_modules/.bin/cypress-slack-reporter --report-dir cypress/reports/mocha (relative to package.json) with the same error, I must be passing something incorrectly?

Regarding point 2, that's awesome thanks. Can't wait to try it out 👍

Yousaf Nabi · Answer 9 · Sun Jul 05 2020 21:28:03 GMT+0800 (China Standard Time)

The report dir is relative to your package root. So you don’t need to do ../../ Spec.ts invokes the cypress runner itself, and uses it own options as set in the script. Which is why you see it perform a full test run. When the cypress runner is called, you can pass in the spec option there.

…

On Sun, 5 Jul 2020 at 12:08, Robin Miklinski ***@***.***> wrote: Am running it locally using spec.ts <https://github.com/YOU54F/cypress-slack-reporter/blob/master/src/cli/spec.ts> from this repo, in the root of my repo. There are existing test reports in cypress/reports/mocha/mochawesome.json Also tried with yarn cypress run --spec 'cypress/integration/PR/test.spec.js' && ./node_stmodules/.bin/cypress-slack-reporter --report-dir ../../cypress/reports/mocha with the same error, I must be passing something incorrectly? Regarding point 2, that's awesome thanks. Can't wait to try it out 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#650 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYCJ4NFOEDEALPNCZWWZETR2BNLBANCNFSM4OCTCVDQ> .

Robin Miklinski · Answer 10 · Mon Jul 06 2020 04:31:34 GMT+0800 (China Standard Time)

Thanks, I had a duplicated report dir that was not longer used, my bad. This is working as needed:

yarn cypress run --spec 'cypress/integration/PR/test.spec.js' && npx cypress-slack-reporter --vcs-provider bitbucket...

Yousaf Nabi · Answer 11 · Mon Jul 06 2020 04:35:10 GMT+0800 (China Standard Time)

@you1anna happy days! Let me know how you get on with testing the new feature for diff webhooks per alert and then we can close this down. Thanks for the feature request as well, and I hope others can make use of it too :) I will be able to at work to alert a main channel when there is a failure and just a healthcheck channel when it passes 🤘🏽

Robin Miklinski · Answer 12 · Mon Jul 06 2020 04:39:04 GMT+0800 (China Standard Time)

Excellent will do, should hopefully be able to get it into our pipeline this week and provide you with some feedback 🙂

Robin Miklinski · Answer 13 · Tue Jul 07 2020 00:39:57 GMT+0800 (China Standard Time)

Hey @YOU54F, consistently getting an error with the report directory when not running via script.ts

rm -rf ./cypress/reports/mocha && yarn cypress run --spec 'cypress/integration/PR/test.spec.js' && npx cypress-slack-reporter --vcs-provider bitbucket --ci-provider custom --report-dir 'mochawesome-report' --video-dir 'cypress/videos' --custom-url 'https://go.testing.build.xxx-cdn.net' --verbose true

The error is
Error: Cannot find test report @ mochawesome-report

When passing the report dir as cypress/reports/mocha
Error: Cannot find test report @ cypress/reports/mocha

Is additional config needed when invoking using this command, as opposed to with script.ts?

Robin Miklinski · Answer 14 · Thu Jul 09 2020 20:58:48 GMT+0800 (China Standard Time)

@YOU54F not sure how to proceed with this one, would you be able to repro locally? Afaik the only way to trigger a subset of tests with the reporter.

Have double checked all package versions are the same, it doesn't seem to detect the reportDir correctly regardless of the setting.

Yousaf Nabi · Answer 15 · Fri Jul 10 2020 04:17:35 GMT+0800 (China Standard Time)

Have you created a html report as well as the Json test report. The script file you run cypress, squash the mochawesome reports into 1 with mochawesome merge and generates a report with Marge. (Mochawesome report generator) you need a json report and html report. I can give you a working example, can you create a branch or repo showing the issue :)

…

On Thu, 9 Jul 2020 at 13:59, Robin Miklinski ***@***.***> wrote: @YOU54F <https://github.com/YOU54F> not sure how to proceed with this one, would you be able to repro locally? Afaik the only way to trigger a subset of tests with the reporter. Have double checked all package versions are the same, it doesn't seem to detect the reportDir correctly regardless of the setting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#650 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYCJ4J4UWYDAQEGIHFIVD3R2W5JPANCNFSM4OCTCVDQ> .

Robin Miklinski · Answer 16 · Fri Jul 10 2020 19:04:03 GMT+0800 (China Standard Time)

Ah didn't realise HTML had to be true, thanks. Have set this in reporterOpts. The issue is now:

Successful runs post the result to slack
failures do not

rm -rf ./cypress/reports/mocha && yarn cy:run:ecards && npx cypress-slack-reporter --vcs-provider bitbucket --ci-provider custom --report-dir 'cypress/reports/mocha' --video-dir 'cypress/videos' --custom-url 'https://go.testing.build.xxcdn.net' --verbose true

All tests pass

Some tests fail

No slack post
error Command failed with exit code 4. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Will try creating a branch in this repo to show the issue.

Yousaf Nabi · Answer 17 · Fri Jul 10 2020 20:04:50 GMT+0800 (China Standard Time)

Thanks mate, are you using the standard slack webhook url or one of the new ones?

…

On Fri, 10 Jul 2020 at 12:04, Robin Miklinski ***@***.***> wrote: Ah didn't realise HTML had to be true, thanks. Have set this in reporterOpts. The issue is now: - Successful runs post the result to slack - failures do not rm -rf ./cypress/reports/mocha && yarn cy:run:ecards && npx cypress-slack-reporter --vcs-provider bitbucket --ci-provider custom --report-dir 'cypress/reports/mocha' --video-dir 'cypress/videos' --custom-url 'https://go.testing.build.xxcdn.net' --verbose true All tests pass [image: Screen Shot 2020-07-10 at 09 33 30] <https://user-images.githubusercontent.com/7108767/87135089-7b209200-c291-11ea-84aa-7c5142f7e540.png> Some tests fail - No slack post error Command failed with exit code 4. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. Will try creating a branch in this repo to show the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#650 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYCJ4LN7IZIQIYHWCHWOF3R23YTDANCNFSM4OCTCVDQ> .

Robin Miklinski · Answer 18 · Fri Jul 10 2020 21:20:59 GMT+0800 (China Standard Time)

We are the payload parameter as JSON in as post request, as opposed to sending a JSON string as the body of a post request. Unsure which is the new way?

Robin Miklinski · Answer 19 · Sat Jul 11 2020 18:45:30 GMT+0800 (China Standard Time)

Hey @YOU54F Here is a repo example, if you export a SLACK_WEBHOOK_URL value first then run the command in the readme, hopefully it should repro the issue, thanks.

https://github.com/you1anna/cypress-slack-reporter-issue

Yousaf Nabi · Answer 20 · Mon Jul 13 2020 20:43:49 GMT+0800 (China Standard Time)

Hey dude, thanks for the repro, will have a look. I think I know what the issue is. You might just need to pass in the specs you want to run, as a parameter to the node script you are running, so that the cli runner for cypress can pick it up.

Might not get a chance until later this week. Thanks for perservering though

Yousaf Nabi · Answer 21 · Mon Jul 13 2020 20:44:11 GMT+0800 (China Standard Time)

We are the payload parameter as JSON in as post request, as opposed to sending a JSON string as the body of a post request. Unsure which is the new way?

I don't understand this comment, could you elaborate please?

Robin Miklinski · Answer 22 · Mon Jul 13 2020 23:02:42 GMT+0800 (China Standard Time)

Hey dude, thanks for the repro, will have a look. I think I know what the issue is. You might just need to pass in the specs you want to run, as a parameter to the node script you are running, so that the cli runner for cypress can pick it up.

Might not get a chance until later this week. Thanks for perservering though

Thanks, the command above is passing cy:run:ecards which runs a /ecards/* command, is there an example of the correct way to do this? Passing specs individually would get messy, I think as there may be many tests.

Whenever you have time that would be a great help, cheers.

Robin Miklinski · Answer 23 · Mon Jul 13 2020 23:05:58 GMT+0800 (China Standard Time)

We are the payload parameter as JSON in as post request, as opposed to sending a JSON string as the body of a post request. Unsure which is the new way?

I don't understand this comment, could you elaborate please?

I'm not sure how to differentiate between the new and old web hooks, devops control that stuff, but I believe we are using the standard hooks.

Yousaf Nabi · Answer 24 · Sat Jul 18 2020 20:18:11 GMT+0800 (China Standard Time)

Hey @you1anna

Updated your repo with examples

https://github.com/you1anna/cypress-slack-reporter-issue/pull/1/files

Basically you need you chain your slack command with || otherwise the exit code thrown from cypress will cause the next step to fail.

Setup some test webhooks against a slack org I used test integrations,they are removed now so you will need to create your own.

We need to pass in the htmlReport path to get the path and file name of the html report for building up a url link.

do you have a remote path and filename for your html report in your custom location? If so maybe we need to support the user providing a customUrl and a custom path inc filename to the html test report.

Or provide a noHtml flag to ignore the html report side entirely (we only need the json test report to determine the test run state, the html report is just reporting sugar in the slack message)

Yousaf Nabi · Answer 25 · Sat Jul 18 2020 21:03:52 GMT+0800 (China Standard Time)

Btw you can run the script like so

SPECS='cypress/integration/pass/*' ts-node script.ts

If you were you add the following option spec: process.env.SPECS, here https://github.com/YOU54F/cypress-slack-reporter-issue/blob/5954b8f79ccea0c4c0a515b184e9e3fb8c70742f/script.ts#L14

CypressNpmApi.run({
  spec: process.env.SPECS,
  reporter: "cypress-multi-reporters",
  reporterOptions: {
    reporterEnabled: "mocha-junit-reporter, mochawesome",
    mochaJunitReporterReporterOptions: {
      mochaFile: "cypress/reports/junit/test_results[hash].xml",
      toConsole: false,
    },
    mochawesomeReporterOptions: {
      reportDir: "cypress/reports/mocha",
      quiet: true,
      overwrite: false,
      html: false,
      json: true,
    },
  },
})

Robin Miklinski · Answer 26 · Sat Jul 18 2020 22:09:21 GMT+0800 (China Standard Time)

Thanks very much for this @YOU54F

My slack webhook doesn't seem to be posting anything currently, will need to confirm it's still working on Monday.

These are the results trying with Makefile:

with make robin-test

  │ ✖  RandR/PR/ecards.spec.js                  00:32        1        -        1        -        - │
    ✖  1 of 1 failed (100%)                     00:32        1        -        1        -        -  

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
 ciProvider:- custom
 customUrl:- https://go.testing.build.rg-cdn.net
 vcsProvider:- bitbucket
 reportDirectory:- cypress/reports/mocha
 videoDirectory:- cypress/videos
 screenshotDirectory:- cypress/screenshots

TypeError: Cannot read property 'split' of undefined
make: *** [robin-test] Error 1

Trying with the script after adding the SPEC param :

SPECS='cypress/integration/pass/*' && npx ts-node script.ts

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  error/error.spec.js                        0ms        -        -        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  fail/fail.spec.js                        00:01        1        -        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  pass/pass.spec.js                        861ms        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  RandR/PR/ecards.spec.js                  00:31        1        -        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  3 of 4 failed (75%)                      00:33        3        1        3        -        -  

Merged report available here:- [
  '/Users/robin.miklinski/rg/cypress-slack-reporter-issue/mochawesome-report/mochawesome.html',
  '/Users/robin.miklinski/rg/cypress-slack-reporter-issue/mochawesome-report/mochawesome.json'
]
Constructing Slack message with the following options {
  ciProvider: 'custom',
  vcsRoot: 'bitbucket',
  reportDir: 'mochawesome-report',
  videoDir: 'cypress/videos',
  screenshotDir: 'cypress/screenshots',
  verbose: true
}
Error: TypeError: Cannot read property 'split' of undefined
    at Object.slackRunner (/Users/robin.miklinski/rg/cypress-slack-reporter-issue/node_modules/cypress-slack-reporter/bin/slack/slack-alert.js:43:15)
    at /Users/robin.miklinski/rg/cypress-slack-reporter-issue/script.ts:69:5
    at tryCatcher (/Users/robin.miklinski/rg/cypress-slack-reporter-issue/node_modules/bluebird/js/release/util.js:16:23)

I am unclear whether this error is causing the slack post to fail, will be able to confirm on Monday. Do you know what Cannot read property 'split' of undefined is referring to?

do you have a remote path and filename for your html report in your custom location? If so maybe we need to support the user providing a customUrl and a custom path inc filename to the html test report.

Our build server is GoCD and the report will sit in a custom location as an artefact, support for this would be great, but non essential.

Yousaf Nabi · Answer 27 · Sun Jul 19 2020 05:03:12 GMT+0800 (China Standard Time)

Ahh the split is trying to split a webhook, it suggests you haven't set one. Basically the split function will split a webhook on the , in order to allow the sending a message to multiple channels.

https://github.com/YOU54F/cypress-slack-reporter/blob/04181cf9dbb91c45acdf46aca1a6d42083b74bf3/src/slack/slack-alert.ts#L110

I can wrap that in a try / catch and throw a descriptive error there to help users

If you can set export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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 you should be able to process the slack report, but it will return a 404 (as those webhooks are been removed now) and then run make robin-test you should be golden

#any
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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
#error
SLACK_WEBHOOK_ERROR_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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
#failed
SLACK_WEBHOOK_FAILED_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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
#passed
SLACK_WEBHOOK_PASSED_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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

Yousaf Nabi · Answer 28 · Sun Jul 19 2020 05:03:47 GMT+0800 (China Standard Time)

you can check if the env var is set with echo $SLACK_WEBHOOK_URL 👍🏽

Yousaf Nabi · Answer 29 · Sun Jul 19 2020 22:44:33 GMT+0800 (China Standard Time)

I've the most recent branch I am working on, I've made it more fault tolerant so that if

if can't find a test report, it will always send a build error message, and log a warning
if it can't a html file, it will no longer error (will need additional param to pass in a html file name) and still report on the test run, and log a warning
if it finds more than one test report, it will use the first and log a warning

Hopefully that should ensure that whatever the state of the test run, it should be able to report a message to slack, as the notification is the most important thing.

Robin Miklinski · Answer 30 · Sun Jul 19 2020 23:16:48 GMT+0800 (China Standard Time)

Ahh the split is trying to split a webhook, it suggests you haven't set one. Basically the split function will split a webhook on the , in order to allow the sending a message to multiple channels.

https://github.com/YOU54F/cypress-slack-reporter/blob/04181cf9dbb91c45acdf46aca1a6d42083b74bf3/src/slack/slack-alert.ts#L110

I can wrap that in a try / catch and throw a descriptive error there to help users

If you can set export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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 you should be able to process the slack report, but it will return a 404 (as those webhooks are been removed now) and then run make robin-test you should be golden

#any
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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
#error
SLACK_WEBHOOK_ERROR_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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
#failed
SLACK_WEBHOOK_FAILED_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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
#passed
SLACK_WEBHOOK_PASSED_URL=https://hooks.slack.com/services/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,https://hooks.slack.com/services/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

I had set the .env file hooks to my test hook and run source etc, but nothing seems to post to slack, so wondering if something else is wrong. Will confirm with the team tomorrow.

Hopefully that should ensure that whatever the state of the test run, it should be able to report a message to slack, as the notification is the most important thing.

Yes those changes all sound sensible, especially if something changes in CI and the report cannot be found for whatever reason.

Yousaf Nabi · Answer 31 · Mon Jul 20 2020 02:46:17 GMT+0800 (China Standard Time)

latest version now released v1.0.2

Yousaf Nabi · Answer 32 · Tue Jul 21 2020 01:06:00 GMT+0800 (China Standard Time)

FYI

https://unix.stackexchange.com/a/90995

basically use && npx cypress-slack-reporter || npx cypress-slack-reporter to allow it to execute the slack message regardless of whether the step before passed or failed

Robin Miklinski · Answer 33 · Thu Jul 23 2020 20:55:35 GMT+0800 (China Standard Time)

Hey @YOU54F

In CI we have the Test Report button linking to artefacts, containing the report, videos etc which is great - that part's working fine.

There are two outstanding issues:

How can the commit link to bitbucket be constructed?

Previously locally it was returning video links in the slack post, but have not seen these generated since 18th June, I think on v0.8 on cypress-slack-reporter. The video's directory is being passed as a parameter, and it should be working.

Yousaf Nabi · Answer 34 · Wed Aug 12 2020 08:51:00 GMT+0800 (China Standard Time)

which ci system do you use for building? can you raise that as a new issue.

2nd issue is fixed, it was a regression.

cheers again @you1anna