Package Storage v2: repack existing packages as .zip archives.

[Julien Lind]

From @mtojek

We have to import existing packages to Package Storage. Here are the requirements I identified:

1. We need to keep the migration up until we switch to the new storage = archive the only one.
2. We shouldn’t block ongoing package development.
3. We need to implement the CI logic to check if the package has been already published. Otherwise, we’ll end up with an avalanche of queued publishing jobs.

To do this really quickly I see following options:

Pull model

1. Internal CI: run a daily job to copy packages from legacy Package Storage repository to Google Storage
2. (-) Storages won’t be kept in sync for a day

Push model

1. Beats CI: trigger publishing job for every built and unpublished package
2. Beats CI: trigger publishing job for every stage
3. Beats CI: call signing pipeline before publishing
4. (-) The first run will take some time and resources to publish ~1000 packages
5. (+) Storages will be in-sync as soon as possible

Pull and Push model

1. Variation of both models
2. Use pull model to copy existing packages once
3. Then, enable the push model to include only new packages

[Marcin Tojek]

@jsoriano It looks like a CI-crafting task, so one of our "favorites" :) Do you have any preference regarding the model? I would vote for the push model as it's the closest to the future form - pushing from Integrations to Storage v2.

[Jaime Soriano Pastor]

@mtojek push model SGTM, no need to worry a lot about this not being optimal for the initial import as ideally we only need to do this once.

Would this also cover packages that are not in the integration repository?

I would like to have a clear separation between pushing code to a repo where the code of the package lives, and the publication of a version of this package, so each package developer can decide how and when to publish. If they want CD, they can hook the publication of the package to changes in their branches, if not, they can have a different process to trigger the publish action.

[Marcin Tojek]

@mtojek push model SGTM, no need to worry a lot about this not being optimal for the initial import as ideally we only need to do this once.

👍

Would this also cover packages that are not in the integration repository?

Yes, as this code will be executed as part of the Package Storage job.

If they want CD, they can hook the publication of the package to changes in their branches, if not, they can have a different process to trigger the publish action.

That's the end goal, we can/will prepare Jenkins groovy helper functions to reach out to the new storage. I'd like to configure the full CD for the Integrations repository and just give guidance for the rest.

---

Looks like we agree, so the "push model" wins.

[Jaime Soriano Pastor]

Yes, as this code will be executed as part of the Package Storage job.

But it is still the idea to remove the Package Storage at some moment, right?

[Marcin Tojek]

Yes :D in Github terminology: archive it (no more PRs welcome).

[Marcin Tojek]

All production packages have been pushed to the new storage v2.

There will be a CI task to syncing updating v2 with v1, until we fully switch to the new place (PR).

Once we merge this PR, we can close this issue.