[Feature] Can you consider supporting downloading multiple images and compressing them into A.AR
comqx opened this issue · comments
regctl image export xxx/xxx/kibana:7.9.3 xxx/xxx/logstash:7.10.3 xxx/xxx/elasticserach:7.10.3 elk.tar
Can you consider supporting downloading multiple images and compressing them into elk.tar
Since the filename is optional in that command (it can also output the tar to stdout), I'm not sure there's a way to add more images to the command without being a breaking change. There's no guaranteed way to determine if the last arg is a filename or an image when the number of args is variable.
Is this for exporting to later import to another registry, or is this to import into a docker engine? With the former, there's options to leverage the OCI Layout.
Since the filename is optional in that command (it can also output the tar to stdout), I'm not sure there's a way to add more images to the command without being a breaking change. There's no guaranteed way to determine if the last arg is a filename or an image when the number of args is variable.
Is this for exporting to later import to another registry, or is this to import into a docker engine? With the former, there's options to leverage the OCI Layout.
regctl image export elk.tar xxx/xxx/kibana:7.9.3 xxx/xxx/logstash:7.10.3 xxx/xxx/elasticserach:7.10.3
Is that gonna solve that?
likeness:
tar cf a.tar xx01 xx02 xx03 xx04
regctl image export elk.tar xxx/xxx/kibana:7.9.3 xxx/xxx/logstash:7.10.3 xxx/xxx/elasticserach:7.10.3 Is that gonna solve that?
That's reordering the parameters (breaking every existing user's script) and requiring the tar as an argument (breaking stdout use cases).
likeness: tar cf a.tar xx01 xx02 xx03 xx04
That's a different scenario. a.tar is the argument to -f on the tar command. If the -f is not included, it knows to output to stdout. Since there's no flag for the elk.tar parameter, there's no guaranteed way to know if it's included or not (it's possible to infer, but that's error prone, i.e. if someone names their tar file like an image ref, or includes a tag with an extension).
regctl image export elk.tar xxx/xxx/kibana:7.9.3 xxx/xxx/logstash:7.10.3 xxx/xxx/elasticserach:7.10.3 Is that gonna solve that?
That's reordering the parameters (breaking every existing user's script) and requiring the tar as an argument (breaking stdout use cases).
likeness: tar cf a.tar xx01 xx02 xx03 xx04
That's a different scenario.
a.taris the argument to-fon the tar command. If the-fis not included, it knows to output to stdout. Since there's no flag for theelk.tarparameter, there's no guaranteed way to know if it's included or not (it's possible to infer, but that's error prone, i.e. if someone names their tar file like an image ref, or includes a tag with an extension).
The simplest solution is:
regctl image export xxx/xxx/kibana:7.9.3,xxx/xxx/logstash:7.10.3,xxx/xxx/elasticserach:7.10.3 elk.tar
","segmentation
Is this for exporting to later import to another registry, or is this to import into a docker engine? I think there are better ways to do the former that won't cause so many issues with the implementation.
Is this for exporting to later import to another registry, or is this to import into a docker engine? I think there are better ways to do the former that won't cause so many issues with the implementation.
The purpose is to export the image, and then take the tar package to import the private warehouse in a non-network environment. By using regctl, you can avoid pulling all the images locally and exporting them as a save.
The purpose is to export the image, and then take the tar package to import the private warehouse in a non-network environment. By using regctl, you can avoid pulling all the images locally and exporting them as a save.
Is "warehouse" a registry or something else? How is the content being imported into the warehouse?
Is "warehouse" a registry or something else? How is the content being imported into the warehouse?
is images registry
I think you'll be much better off copying content to an OCI Layout using the ocidir://path:tag reference. That will consolidate the content into a single directory with tags of your choice, blobs will be deduplicated, and if you reuse the same path, the pull/push will be faster since it knows the blobs it has already seen. If it needs to get consolidated to a single file, that directory can be packaged in a tar, but there's not much of a value add to compressing it since the layers will already be compressed. Better than tar'ing into a single file would be using a tool like rsync that only sends changed files between the systems.
This would look like a small script:
regctl image copy xxx/xxx/kibana:7.9.3 ocidir://elk:kibana
regctl image copy xxx/xxx/logstash:7.10.3 ocidir://elk.logstash
regctl image copy xxx/xxx/elasticserach:7.10.3 ocidir://elk:elasticsearch
tar -cf elk.tar elk # or preferably rsyncWhile this is a script with multiple commands, regctl is designed to be included in scripts. And importantly, with this you can differentiate between the logstash and elasticsearch images even through they have the same tag from the source, because you can specify any tag you want in the OCI Layout.
I think you'll be much better off copying content to an OCI Layout using the
ocidir://path:tagreference. That will consolidate the content into a single directory with tags of your choice, blobs will be deduplicated, and if you reuse the same path, the pull/push will be faster since it knows the blobs it has already seen. If it needs to get consolidated to a single file, that directory can be packaged in a tar, but there's not much of a value add to compressing it since the layers will already be compressed. Better than tar'ing into a single file would be using a tool like rsync that only sends changed files between the systems.This would look like a small script:
regctl image copy xxx/xxx/kibana:7.9.3 ocidir://elk:kibana regctl image copy xxx/xxx/logstash:7.10.3 ocidir://elk.logstash regctl image copy xxx/xxx/elasticserach:7.10.3 ocidir://elk:elasticsearch tar -cf elk.tar elk # or preferably rsyncWhile this is a script with multiple commands, regctl is designed to be included in scripts. And importantly, with this you can differentiate between the logstash and elasticsearch images even through they have the same tag from the source, because you can specify any tag you want in the OCI Layout.
This also works in shell scripts. An operational behavior that feels like it conforms to a command.
I have now split the project into api mode. You do not want to call through shell scripts in the project
While this is a script with multiple commands, regctl is designed to be included in scripts. And importantly, with this you can differentiate between the logstash and elasticsearch images even through they have the same tag from the source, because you can specify any tag you want in the OCI Layout.
This also works in shell scripts. An operational behavior that feels like it conforms to a command. I have now split the project into api mode. You do not want to call through shell scripts in the project
I'm not sure I understand. The image copy commands can be called directly from the Go API. How does this work for multiple images with the same tag?
While this is a script with multiple commands, regctl is designed to be included in scripts. And importantly, with this you can differentiate between the logstash and elasticsearch images even through they have the same tag from the source, because you can specify any tag you want in the OCI Layout.
This also works in shell scripts. An operational behavior that feels like it conforms to a command. I have now split the project into api mode. You do not want to call through shell scripts in the project
I'm not sure I understand. The image copy commands can be called directly from the Go API. How does this work for multiple images with the same tag?
{
"dir": "/temp/images-packages/",
"images": [
{
"origin": "mysql:3.7",
"newImages": "hub.private.com/service/mysql:3.7"
},
{
"origin": "mongo:4.0.28",
"newImages": "hub.private.com/service/mongo:4.0.28"
}
]
}
Finally, you can compress a package of packages.tar under /temp/images-packages/.
Then use the following command to export
regctl images import hub.private.com/service/mysql:3.7,hub.private.com/service/mongo:4.0.28 packages.tar
I'm asking about how this works when the tag is the same:
$ regctl image export -p local ghcr.io/regclient/regctl:v0.5.4,ghcr.io/regclient/regsync:v0.5.4,ghcr.io/regclient/regbot:v0.5.4 export.tar
$ regctl image import localhost:5000/import/regctl:v0.5.4,localhost:5000/import/regsync:v0.5.4,localhost:5000/import/regbot:v0.5.4 export.tar
$ regctl image digest -p local ghcr.io/regclient/regctl:v0.5.4
sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d
$ regctl image digest -p local ghcr.io/regclient/regsync:v0.5.4
sha256:ae30f9a095ee9433a6ec9b4c5728a946f0a5569e566cc91e8e02bf588a739f0e
$ regctl image digest -p local ghcr.io/regclient/regbot:v0.5.4
sha256:715075f3727bcd4e5f23919a3e6db374c437b21ea397dea57d14252ca5afbeca
$ regctl image digest -p local localhost:5000/import/regctl:v0.5.4
sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d
$ regctl image digest -p local localhost:5000/import/regsync:v0.5.4
sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d
$ regctl image digest -p local localhost:5000/import/regbot:v0.5.4
sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913dAll 3 imported images were regctl since the tag is used to pick the image.
I'm asking about how this works when the tag is the same:
$ regctl image export -p local ghcr.io/regclient/regctl:v0.5.4,ghcr.io/regclient/regsync:v0.5.4,ghcr.io/regclient/regbot:v0.5.4 export.tar $ regctl image import localhost:5000/import/regctl:v0.5.4,localhost:5000/import/regsync:v0.5.4,localhost:5000/import/regbot:v0.5.4 export.tar $ regctl image digest -p local ghcr.io/regclient/regctl:v0.5.4 sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d $ regctl image digest -p local ghcr.io/regclient/regsync:v0.5.4 sha256:ae30f9a095ee9433a6ec9b4c5728a946f0a5569e566cc91e8e02bf588a739f0e $ regctl image digest -p local ghcr.io/regclient/regbot:v0.5.4 sha256:715075f3727bcd4e5f23919a3e6db374c437b21ea397dea57d14252ca5afbeca $ regctl image digest -p local localhost:5000/import/regctl:v0.5.4 sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d $ regctl image digest -p local localhost:5000/import/regsync:v0.5.4 sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913d $ regctl image digest -p local localhost:5000/import/regbot:v0.5.4 sha256:813b63d96e135da0370daae11a87c086f078e5448dbedde5e61b7367b81e913dAll 3 imported images were regctl since the tag is used to pick the image.
This will not happen, here is the package I exported, and the effect after unzipped. The index manifest is merged, so the above situation does not exist
This will not happen
I'm not sure how better to say this did happen, it's not made up output. The commands I showed above are using a binary built from the linked PR. Can you run the same commands on your machine to compare? Note that the tags must be identical to see the issue.
This problem isn't in the generation of the index, it's in importing content contained in the index.
I'm not sure how better to say this did happen, it's not made up output. The commands I showed above are using a binary built from the linked PR. Can you run the same commands on your machine to compare? Note that the tags must be identical to see the issue.
This problem isn't in the generation of the index, it's in importing content contained in the index.
I'm sorry, I misunderstood. I have now submitted the latest code.
Based on feedback in the PR, I don't think this is currently feasible without introducing a lot of other issues. Given that there is a usable solution based on copying an image to an OCI Layout, I'm closing this as not planned.