Inconsistency in generation pipeline file copy operation for .NET beta
zengin opened this issue · comments
We have a smoke test step in metadata generation, which copies generated files into .NET beta repo and attempts to compile. The files copied in this step are not the same as the ones copied in the subsequent beta SDK generation step. Investigate whether the files are not generated at all or not copied.
AB#8165
@baywet @MIchaelMainer @nikithauc this is super interesting.
Our multi stage pipeline is broken in a very weird way. If you check-in in any of the stages in one run (even if to the main branch), it will not be picked up by the subsequent stages.
This is what we do in the initial metadata capture stage. But in subsequent stages, those changes are not reflected, because Azure DevOps takes a snapshot at the time of queueing and checks out in detached head state (with the snapshot commit hash) when we use built-in checkout step.
yeah it'd make sense ADO keeps the same "source state" across multiple stages of the pipelines so you don't run into inconsistencies should somebody commit between two stages.
We could address that a different way, instead of having a gigantic pipeline that does everything from updating metadata to generating and testing the SDKs we could have:
- one pipeline that updates the metadata (schedule trigger)
- one pipelines that generates and test the SDKs (CI trigger on updates from the metadata repo), which we could even break further down if we wanted to
An alternative could be to upload the generated metadata as artifacts and use that instead of the repo for subsequent stages.
I think for short term solution, we can checkout master in the subsequent steps explicitly. Having one trigger pipeline works nicely with testing end-to-end scenarios.
We can also add another trigger for this pipeline which could handle the CI in case someone checks into msgraph-metadata directly.
But checking out master brings us back to the consistency issue. In the (unlikely I'll admit) event where someone checks-in something on master between the metadata update stage and the following ones, the following ones will not be consistent with the changes we intended to generate for. If we want to keep a single pipeline, using artifacts is seems like a better approach. And the changes should not be much more than checking out master:
- upload task in the metadata update stage
- download tasks in the generation stages
- update the paths for generation to point to what has been downloaded from the artifacts
- (opt. for performance, remove checking out the metadata repo in the generation stages)
I think for short term solution, we can checkout master in the subsequent steps explicitly. Having one trigger pipeline works nicely with testing end-to-end scenarios.
We can also add another trigger for this pipeline which could handle the CI in case someone checks into msgraph-metadata directly.
+1
@baywet, but why wouldn't we want to pick up the latest changes already in master even if it is checked in a moment ago? It will be reflected in next week's generation anyways.
That said, it makes sense to use artifacts directly from the consistency point of view. Whatever we cleaned as metadata will be used as-is in the subsequent stages. The only issue now is to iterate over this design, because pipelines are not fun to iterate on especially when they directly manipulate the master branch as in this case.
From a pure CI perspective, having the sources being changed by an external event as the process is "running" seems counter-productive.
It'll also be an issue for people working in parallels or on multiple branches where cross-wiring might happen.
Using artifacts would solve all those design issues. We could even make committing the updated metadata back to the repo an option (on by default) which would also make testing easier. (no impacts on the master branch, concurrent tests for different things possible)
We already have the option to skip publishing changes.
BTW, we also already publish the metadata as an artifact, we had it for debugging purposes. I think we just need to download that instead of using from the metadata repo :)
even less work to do then 😂
are you taking on the remaining?
Yes, I am working on it. It'll take some time because we have to special case it for the scenario where we skip the metadata capture-clean step in manual runs.
Meanwhile I will need to rerun V1 generations, because it also didn't pick up the metadata changes due to the same snapshotting issue. That's why we have seen docs-only PRs there.