Prow Job to Automate Verification of Conformance Log Submissions

Question

Prow Job to Automate Verification of Conformance Log Submissions

hh opened this issue 5 years ago · comments

Hippie Hacker commented 5 years ago

It might help to have some automation around Conformance Log Submissions

Setup Prow for the CNCF org
- @cjwagner has the k8s-test-infra creds / @idvoretskyi has the CNCF org creds
- https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md#add-the-webhook-to-github
Setup tide tags for cncf/k8s-conformance
Setup pre-submit job
Generate Complete list of Conformance Tests within this repo - kubernetes/kubernetes#84330

The verification job should:

Ensure the correct version of the server is tested
Ensure the all Conformance tests have passed by comparing the list of tests for that release to junit.xml

@johnSchnake is looking into using an e2e dry-run reporter combined with the AST walker we use for conformance documentation generation to generate the list of expected tests per release.

/assign @hh

Hippie Hacker · Answer 1 · Fri Nov 01 2019 01:42:52 GMT+0800 (China Standard Time)

Ticket for Generating Full List of Tests with metadata and full test names:

kubernetes/kubernetes#84330 (comment)

Hippie Hacker · Answer 2 · Wed Nov 06 2019 02:31:16 GMT+0800 (China Standard Time)

@idvoretskyi We need to coordinate with @cjwagner to give prow.k8s.io access to the CNCF org as the first step in configuring prow/tide etc.

Hippie Hacker · Answer 3 · Tue Nov 12 2019 05:14:12 GMT+0800 (China Standard Time)

/assign @caniszczyk
Can you work with @idvoretskyi / @cjwagner regarding cncf/org permissions required to setup prow.k8s.io

https://github.com/kubernetes/test-infra/blob/master/prow/getting_started_deploy.md#add-the-webhook-to-github

Hippie Hacker · Answer 4 · Thu Nov 21 2019 21:50:48 GMT+0800 (China Standard Time)

/assign @dankohn

We need to grant @cjwagner temprorary cncf github org admin in order to setup the necessary org <-> prow integrations.

There are secrets on both sides regarding the k8s infra and cncf org event hooks.

We can remove admin rights once the integration is setup.

Cole Wagner · Answer 5 · Wed Dec 04 2019 02:28:02 GMT+0800 (China Standard Time)

CNCF org level webhooks have been configured for prow.k8s.io.

Hippie Hacker · Answer 6 · Wed Dec 04 2019 06:38:24 GMT+0800 (China Standard Time)

/assign @hh

Hippie Hacker · Answer 7 · Tue Apr 21 2020 06:40:03 GMT+0800 (China Standard Time)

We will need to migrate the web hooks over to prow.cncf.io when it's brought up.

Hippie Hacker · Answer 8 · Wed Apr 29 2020 08:38:35 GMT+0800 (China Standard Time)

@idvoretskyi can you add https://github.com/cncf-ci as an admin to https://github.com/cncf/k8s-conformance. WE'll need this to setup a cncf owned bot for use with prow.cncf.io.

Ihor Dvoretskyi · Answer 9 · Wed Apr 29 2020 18:43:01 GMT+0800 (China Standard Time)

@idvoretskyi can you add https://github.com/cncf-ci as an admin to https://github.com/cncf/k8s-conformance. WE'll need this to setup a cncf owned bot for use with prow.cncf.io.

This is solved, I assume.

Bernokl · Answer 10 · Tue May 18 2021 02:12:05 GMT+0800 (China Standard Time)

/close The prowjob has been successfully monitoring k8s-conformance. This work is done