[DISCUSS] Package release and development process
jonathan-buttner opened this issue · comments
Endpoint's current flow for developing the endpoint package is to bump the minor version of the package and push it to staging (and which also gets picked up in snapshot) to allow other developers who do not run the package registry locally to leverage new mappings. During the development cycle multiple "interim" packages could be released to staging. This has the side effect of skipping versions in the package.
Developer1 might pushes a package for some feature at version 0.14.0 and a couple weeks later another developer adds new mappings and pushes 0.15.0 to staging. When the package is finally released to production it might have skipped from 0.13.0 to 0.20.0 which could cause confusion to the user.
The advantage of keeping each set of "features" versioned during a single production release of a package allows testing to determine if a single staging release of the package causes issues. If we keep the package as 0.14.0 for both features and an issue occurs, we have to find out what features are in version 0.14.0 for the package that is in the remote registry. It also means that 0.14.0 in the remote registry is not the same as what we have in our endpoint-package repo at 0.14.0. The endpoint-package repo might have 3 other features that haven't yet been pushed to staging.
To support "interim" packages I think it'd be helpful to allow prereleases of a package. As we're developing features we can create packages with the version 0.14.0-dev.0 and the next one will have 0.14.0-dev.1 etc. Then when we're ready to release a package to production we'll release 0.14.0 to staging give it a quick check to make sure nothing crazy happened and finally cherry-pick that package from staging to the production branch.
If we have prerelease packages and 0.14.0-dev.0 exists in staging and a new package 0.14.0-dev.1 gets released to staging and it causes issues I can quickly find the commit or tag for 0.14.0-dev.1 in the endpoint-package repo and figure out what changes happened between -dev.0 and -dev.1.
Bumping the version in some fashion between interim packages also has the benefit of allowing long standing testing environments built from master in the cloud to pull down new version of the package. If we keep the versions the same kibana will not identify that there is a new version and these cloud environments will not be able to install the new package.
Would all these "prerelease" packages be pushed to staging or snapshot?
Would all these "prerelease" packages be pushed to
stagingorsnapshot?
Actually good point, we could just push them to snapshot since that's what master is pointing at.
@jonathan-buttner Yes, and we could still do you the prereleases but perhaps 0.14.0-{commit-hash} or anything you can automated instead.
@jonathan-buttner Yes, and we could still do you the prereleases but perhaps
0.14.0-{commit-hash}or anything you can automated instead.
Ok cool, I'll start working on the ingest manager changes. I think we'll still what to do <major>.<minor>.<patch>-<prerelease string (dev, alpha whatever)><some number>+<commit-hash>
If we don't have the some number portion we could run into a scenario where the newest commit (newest prerelease) is technically less than the previous prerelease. This probably wouldn't be a huge deal since most of the time snapshot is only going to be serving a single endpoint package, but it could still affect long running testing environments 🤷
Good point about deciding what is newest. Any plans on your end how you will generated the preprelease incrementally or is it manual?
Good point about deciding what is newest. Any plans on your end how you will generated the preprelease incrementally or is it manual?
Yeah my hope is to create a script for the endpoint package that allows a developer to run a make command to release the package. The script would ask whether the release is a prerelease or meant for production. If it is a prerelease it would keep the prerelease version that I'm working on here: https://github.com/jonathan-buttner/endpoint-package/blob/update-readme/.bumpversion.cfg#L2
It would open a PR against snapshot with the current version string. Once the PR is opened, we use bump2version to increment the prerelease number and that gets committed to the endpoint package so that the next developer that needs to do a prerelease is already ready to go.
If it is a production release, bump2version will strip off the prerelease portion, commit that, and then open a PR against either snapshot or staging depending the conclusion of our other discussion. Once the PR is opened, we'll run bump2version again and it'll increment the minor or maybe major in preparation for the next release and add the prerelease section back to the version string and commit those changes.
We've been using bump2version for bumping the version already: jonathan-buttner/endpoint-package@2898e60
I wonder that's the reason why we have the "experimental" tag.
e.g.
version 1.0.5 is "ga"
version 1.0.6-prerelease-alpha is "experimental" , so prods won't expose it.
@mtojek This is a good question. As package versions are not tied to stack release versions, we could indicate experimental and beta releases in the version number.
Taking your example above, also <1.0.0 is beta or experimental. How would you make the differentation between beta and experimtental?
@mtojek This is a good question. As package versions are not tied to stack release versions, we could indicate experimental and beta releases in the version number.
I'm fine with this.
Taking your example above, also <1.0.0 is beta or experimental.
Yes. "Beta" or "experimental" or "ga" are just tags defining stability of the revision. Once defined I can't be changed/upgraded/downgraded.
How would you make the differentation between beta and experimtental?
I'm not sure if it answers your question, but I believe it's up to us and our level of confidence, whether we should release package as beta or it's still experimental.
@mtojek Could you make an example of a revision for beta and one for experimental?
Beta samples:
mysql-0.9.8 - has been released to prod, but we're not confident about this revision and we may change it according to the release policy (beta)
nginx-12.3- previous release was 12.2, now we're heading up with 12.3.
Experimental samples:
nginx-12.3.1 - contain a bugfix for problems spotted in 12.3. It may be tested by community users, but shouldn't be installed in general
endpoint-0.14.0-dev.1 - we can expect more -dev.X revisions. We discussed it above.
haproxy-13.0 - previous release, haproxy-12.8 was ga, but we introduced a new dataset hence this is revision is not advised to be installed.
@mtojek Based on your comment in #252 (comment) I somehow assumed, you meant beta/experimental can be indicated with the version itself. But taking nginx-12.3.1 as an example, this is not the case.
Diving into your example for haproxy-13.0. There is also a part, that a dataset can be experimental / beta but the package is GA, that is at least my view.
Beta samples:
mysql-0.9.8- has been released to prod, but we're not confident about this revision and we may change it according to the release policy (beta)
nginx-12.3- previous release was 12.2, now we're heading up with 12.3.
Just want to make sure I'm tracking correctly. The tag we're talking about that indicates whether it is beta, experimental etc is the release field in the manifest right? https://github.com/elastic/package-storage/blob/production/packages/endpoint/0.13.1/manifest.yml#L8
Diving into your example for haproxy-13.0. There is also a part, that a dataset can be experimental / beta but the package is GA, that is at least my view.
Oh I didn't realize that datasets themselves can be experimental but the package is GA. Is that something we support right now? Or something you're suggesting we add in the future?
Also just a side note, I don't think 12.3 is a valid semver 2 format, which the Kibana ingest manager app enforces now.
@jonathan-buttner Each dataset can have it's own release tag: https://github.com/elastic/package-storage/blob/production/packages/aws/0.2.4/dataset/billing/manifest.yml#L2 We have it in the manifest today but don't expose it in the UI yet.
Also just a side note, I don't think 12.3 is a valid semver 2 format, which the Kibana ingest manager app enforces now.
Sorry, it's a typo. We can consider this as 12.3.0.
@jonathan-buttner I think we can close this issue as we made quite a few decisions since August on this.