[Fleet] Allow uploading zip packages in integrations UI

Question

[Fleet] Allow uploading zip packages in integrations UI

jen-huang opened this issue a year ago · comments

Supersedes #70582

Today we have the endpoint POST /api/fleet/epm/packages which accepts a zip package archive and uploads it to Fleet in the same way that a regular package install would. This feature is important to reduce the total delivery time for package developers. While much of the functionality is complete through enabling bundled packages with Kibana, there are still some outstanding tasks before this API is ready for documentation and wider use:

The package policy editor doesn't use the streams from the uploaded package if another version of the package also exists in the registry (original report)
Ensure uploaded package can be fully uninstalled
Ensure uploaded package can be fully overwritten (similar to a re-install)
Ensure uploaded package does not conflict with bundled or registry packages (whether they are installed or not)
Robust unit tests around the code path (installPackageByUpload, parseAndVerifyArchive)
Improved integration tests
OpenAPI documentation

Sample test instructions can be found at elastic/integrations#4597:

Download elasticsearch-1.1.0-preview1.zip
Upload the package to Kibana:

curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://my-kibana:5601/api/fleet/epm/packages -u elastic:changeme --data-binary @elasticsearch-1.1.0-preview1.zip

Related enhancements that can be done once the API is finished:

Support gzip in addition to zip (as of last testing, seems gzip has some issues)
UI for upload package
CLI integration with elastic-package: elastic/elastic-package#1084

Elastic Machine · Answer 1 · Tue Jan 10 2023 05:53:34 GMT+0800 (China Standard Time)

Pinging @elastic/fleet (Team:Fleet)

Julia Bardi · Answer 2 · Mon Feb 06 2023 21:12:34 GMT+0800 (China Standard Time)

After reviewing the issue, the linked improvements in #70582 are still needed. Started working on them.

[Fleet] Use epm-packages saved object as main source of truth for installed packages #81995
[Fleet] Refactor package installation / upgrade / downgrade and error recovery / rollback #82007

Can't reproduce this issue:

[Fleet] Add error handling: uploaded package missing from in-memory cache #82010

This one I'll hold off on, as it is a change in package-spec repo.

Add package validator in typescript elastic/package-spec#58

Julia Bardi · Answer 3 · Mon Feb 06 2023 21:25:43 GMT+0800 (China Standard Time)

@jen-huang @kpollich @jsoriano Do we have an example package that is not yet in the registry? The elasticsearch package mentioned in the description is already there. I would need one to test some scenarios where the package is not in registry.

Mark Hopkin · Answer 4 · Mon Feb 06 2023 21:50:57 GMT+0800 (China Standard Time)

@juliaElastic you could use elastic-package build --zip -v to build one of our test packages in x-pack/test/fleet_api_integration/apis/fixtures if that works? I know alot of them are not 100% valid per the package spec but they should be good enough

Julia Bardi · Answer 5 · Mon Feb 06 2023 22:38:56 GMT+0800 (China Standard Time)

@hop-dev it worked, thanks.

Julia Bardi · Answer 6 · Tue Feb 07 2023 00:10:54 GMT+0800 (China Standard Time)

@jen-huang @kpollich I am wondering if this issue is still relevant: #82010 Not sure how to reproduce a cache miss. Do any of you know? I'm looking at the uploaded packages after kibana restart in local, and don't see any issues.
I think this code was refactored since the issue was raised, and we fetch the info from source if missing in cache. If you agree, we can close this one.

Kyle Pollich · Answer 7 · Tue Feb 07 2023 00:13:11 GMT+0800 (China Standard Time)

I think the above is likely no longer relevant. From what I remember, all installation paths add packages to the in-memory cache as expected. I'm going to close that issue as I don't think it's relevant to the current state of the EPM codebase.

Julia Bardi · Answer 8 · Tue Feb 07 2023 21:43:05 GMT+0800 (China Standard Time)

Tested the Uninstall of uploaded packages and seems to work fine.

@kpollich What is the expected behavior of the Reinstall button on the UI for uploaded packages?
Now the button calls the POST /epm/packages API that calls the installPackageFromRegistry function.

We could potentially add a drag and drop feature to upload the zip file again on the UI, though this is out of scope for this issue I think. Should we make the button disabled with a tooltip for uploaded packages?

Kyle Pollich · Answer 9 · Tue Feb 07 2023 22:29:23 GMT+0800 (China Standard Time)

We could potentially add a drag and drop feature to upload the zip file again on the UI, though this is out of scope for this issue I think. Should we make the button disabled with a tooltip for uploaded packages?

Agreed on a both. A dropzone implementation could be nice, but seems needless for the corner case of reinstalling an uploaded package. Let's go forward with the disabled button + tooltip stating

This integration was installed by upload and cannot be automatically reinstalled. Please upload it again to reinstall.

Julia Bardi · Answer 10 · Wed Feb 08 2023 18:50:30 GMT+0800 (China Standard Time)

@joshdover I tried to reproduce the issue reported here on 8.7 but couldn't:

The package policy editor doesn't use the streams from the uploaded package if another version of the package also exists in the registry

I uploaded the elasticsearch-1.3.0-preview1 package from the linked pr and I'm seeing the new data stream on the Package Policy Create UI. Tested the Edit and Upgrade package policy forms as well, looks good.
Did you see the issue in a different scenario?

Josh Dover · Answer 11 · Wed Feb 08 2023 19:36:13 GMT+0800 (China Standard Time)

@juliaElastic it must be fixed, I was testing this against 8.5 at the time which I think was before some of the changes we did for input packages were in place that may have fixed this.

Julia Bardi · Answer 12 · Mon Feb 13 2023 21:16:54 GMT+0800 (China Standard Time)

Tested uploads with elasticsearch package, that has latest 1.2.0 GA version in registry, initially the package was not installed.

uploaded version 1.3.0-preview1
uploaded an older 1.0.0 version (downgrade)
upgraded uploaded package to latest 1.2.0 registry version
uploaded again version 1.3.0-preview1

Julia Bardi · Answer 13 · Mon Feb 13 2023 21:22:50 GMT+0800 (China Standard Time)

The API work is done, so closing the issue.
Planning to investigate gzip support separately.

Julia Bardi · Answer 14 · Mon Feb 13 2023 21:55:21 GMT+0800 (China Standard Time)

I tried uploading a tar.gz file (compressed an extracted zip package) and it was successful. There is also an integration test with application/gzip content type.

tar -czf verified-1.0.0.tar.gz verified-1.0.0/
curl -XPOST -H 'content-type: application/gzip' -H 'kbn-xsrf: true' http://localhost:5601/julia/api/fleet/epm/packages -u elastic:changeme --data-binary @verified-1.0.0.tar.gz

// response:
{"items":[{"id":"logs-verified.log-1.0.0","type":"ingest_pipeline"},{"id":"logs-verified.log","type":"index_template"},{"id":"logs-verified.log@package","type":"component_template"},{"id":"logs-verified.log@custom","type":"component_template"}],"response":[{"id":"logs-verified.log-1.0.0","type":"ingest_pipeline"},{"id":"logs-verified.log","type":"index_template"},{"id":"logs-verified.log@package","type":"component_template"},{"id":"logs-verified.log@custom","type":"component_template"}],"_meta":{"install_source":"upload"}}%

Support gzip in addition to zip (as of last testing, seems gzip has some issues)

@kpollich Could you let me know how to reproduce the issues mentioned in the description?

Another thing, I noticed that the upload package logic skips verification. Is this something we want to support in the future? If so, might be worth creating a follow up issue.

Kyle Pollich · Answer 15 · Mon Feb 13 2023 22:12:00 GMT+0800 (China Standard Time)

@kpollich Could you let me know how to reproduce the issues mentioned in the description?

It's been a very long time, so I'm not sure. If gzip functionality is working in recent tests, then I'd expect work we've done to archive parsing logic in recent sprints/releases has resolved whatever issues we had initially.

Another thing, I noticed that the upload package logic skips verification. Is this something we want to support in the future? If so, might be worth creating a follow up issue.

Yes, it'd be great to support verification of uploaded packages if possible. I agree that a follow up issue would be appropriate here. I suppose we'll have to think about how to provide a signature file along with an uploaded package, right?

Julia Bardi · Answer 16 · Mon Feb 13 2023 22:40:39 GMT+0800 (China Standard Time)

Yes, it'd be great to support verification of uploaded packages if possible. I agree that a follow up issue would be appropriate here. I suppose we'll have to think about how to provide a signature file along with an uploaded package, right?

Yes, we might have to change the API to support multiple files or create another API endpoint to verify an uploaded package. If we eventually add the elastic-package support for this, it would be easier for users to send the signature as well, even if there are more than one underlying API calls.

Kyle Pollich · Answer 17 · Tue Feb 14 2023 00:03:26 GMT+0800 (China Standard Time)

I think having the upload API support multiple files would be ideal, but not sure on how much work that'd actually be. A separate API for uploading signatures would be a fallback.

Another option could be to allow for a different archive directory structure that includes signature files at the root of the archive. We'd have to have some conditional logic that determines whether this is the case, though, because the current validation logic expects only a single top-level directory I believe.

Josh Dover · Answer 18 · Tue Feb 14 2023 19:20:32 GMT+0800 (China Standard Time)

Another option could be to allow for a different archive directory structure that includes signature files at the root of the archive. We'd have to have some conditional logic that determines whether this is the case, though, because the current validation logic expects only a single top-level directory I believe.

I don't think this would be feasible since the content of the signature itself depends on the content of the zip.

+1 to supporting multiple files on the same API call. This is supported by multipart content-type I believe

nima · Answer 19 · Wed Feb 22 2023 14:10:15 GMT+0800 (China Standard Time)

@kpollich & @juliaElastic is there a docs issue for this feature?

Julia Bardi · Answer 20 · Wed Feb 22 2023 17:44:36 GMT+0800 (China Standard Time)

@nimarezainia Not yet, perhaps we could add a section in the docs about how to upload packages around here: https://www.elastic.co/guide/en/fleet/master/air-gapped.html#air-gapped-diy-epr
@kpollich WDYT?

nima · Answer 21 · Mon Feb 27 2023 08:54:09 GMT+0800 (China Standard Time)

Thanks @juliaElastic. if there was a docs issue with enough details for the documentation to follow that would be great. Regarding the location where that content resides I would leave it to their expertise.

(fyi @dedemorton & @karenzone )

DeDe Morton · Answer 22 · Tue Feb 28 2023 02:13:29 GMT+0800 (China Standard Time)

@juliaElastic If this is an alternative to users hosting their own package registry, then, yes, it should be documented in the topic about air gapped environments. To get this in quickly, I would suggest opening a PR with the doc changes and tagging @elastic/ingest-docs for review and technical edits.