Harbor replication not working
Jeansen opened this issue · comments
I run helm-charts-oci-proxy standalone, e.g. with this command:
docker run --rm --name helm-charts-oci-proxy -p 9000:9000 -e USE_TLS=true -v /data/cert/proxy-ng.cert:/certs/registry.pem:ro -v /data/cert/proxy-ng.key:/certs/registry-key.pem:ro 8gears.container-registry.com/library/helm-charts-oci-proxy
I did a test pull like this:
helm pull oci://proxy-ng:9000/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
And this works. The logs show
proxy-2024/06/10 20:34:56 listening HTTP over TLS serving on port 9000
proxy-2024/06/10 20:35:00 download index: https://open-telemetry.github.io/opentelemetry-helm-charts/index.yaml
proxy-2024/06/10 20:35:00 downloading : https://open-telemetry.github.io/opentelemetry-helm-charts/index.yaml
proxy-2024/06/10 20:35:01 searching index for opentelemetry-operator with reference
proxy-2024/06/10 20:35:01 downloading : https://github.com/open-telemetry/opentelemetry-helm-charts/releases/download/opentelemetry-operator-0.62.0/opentelemetry-operator-0.62.0.tgz
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/tags/list
proxy-2024/06/10 20:35:01 HEAD - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/manifests/0.62.0
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/manifests/sha256:6e610e6d75e2e9838de76cd3930751417b41df47d5c8ec989b3ee74165c751a8
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
proxy-2024/06/10 20:35:01 GET - /v2/open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/blobs/sha256:52fcda92afdc2d221213d45e2449a7f8123df0358ed898760fa229d10f1450e1
I followed along for the replication with Harbor, created an endpoint and a replication rule. I even removed the filter so everything would be replicated.
When I run the replication, I see the following logs:
proxy-2024/06/10 20:38:13 GET - /v2/
proxy-2024/06/10 20:38:13 download index: https://open-telemetry.github.io/index.yaml
proxy-2024/06/10 20:38:13 downloading : https://open-telemetry.github.io/index.yaml
proxy-2024/06/10 20:38:13 GET - /open-telemetry.github.io/v2/_catalog?n=1000
But...nothing will be replicated. Replication is a success, but 0 items have been replicated. I also cannot find anything specific in the Harbor logs ....
I am aware of the default settings, e.g. that manifests are cleaned up after 60 seconds. So, I did a replication directly after a test pull with helm. But also raising the timeouts did not help at all.
How does your replication look like? Can you share screenshots of replication and registry?
Full URL is https://proxy-ng:9000/open-telemetry.github.io
Source filter was set to open-telemetry.github.io/**, then * and finaly I left it empty. Nothing worked so far.
try this:
Endpoint:
- Endpoint URL: https://proxy-ng:9000
Replication:
- Source Registry: Name of the chartproxy Endpoint
To get all charts
Source resource filter:
- Name: open-telemetry.github.io/opentelemetry-helm-charts/**
- Tag: matching **
To get specific chart
Source resource filter:
- Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
- Tag: matching **
@HaveFun83 Thanks, but did not help. When I do it your way, I also do not see any downloads in the log, only:
proxy-2024/06/24 20:05:37 GET - /v2/
proxy-2024/06/24 20:05:37 GET - /v2/_catalog?n=1000
proxy-2024/06/24 20:06:22 cleanup cycle
proxy-2024/06/24 20:06:47 GET - /v2/
proxy-2024/06/24 20:06:47 GET - /v2/_catalog?n=1000
proxy-2024/06/24 20:06:52 GET - /v2/
proxy-2024/06/24 20:06:52 GET - /v2/
With the previous settings, at least something is being download on the proxy side ( see the logs excerpt below) but also then, there's nothing in Harbor:
proxy-2024/06/24 20:09:40 GET - /open-telemetry.github.io/v2/
proxy-2024/06/24 20:09:59 GET - /v2/
proxy-2024/06/24 20:09:59 download index: https://open-telemetry.github.io/index.yaml
proxy-2024/06/24 20:09:59 downloading : https://open-telemetry.github.io/index.yaml
proxy-2024/06/24 20:09:59 GET - /open-telemetry.github.io/v2/_catalog?n=1000
proxy-2024/06/24 20:10:22 cleanup cycle
proxy-2024/06/24 20:11:22 cleanup cycle
proxy-2024/06/24 20:11:52 GET - /v2/
proxy-2024/06/24 20:11:52 GET - /open-telemetry.github.io/v2/
proxy-2024/06/24 20:12:22 cleanup cycle
can you double check your config according to the logs your replication still point to the wrong path
open-telemetry.github.io instead of open-telemetry.github.io/opentelemetry-helm-charts
Also don't put any path behind your harbor endpoint url configuration
just plain https://proxy-ng:9000
I tried the following replication
Source registry: <your chartproxy endpoint>
Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
Tag: 0.63.0
Resource: image
Destination
Namespace: <your harbor project>
Flattening: Flatten All Levels
and it works without any problems
@HaveFun83 Oh..yay........ your are right.
With open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator it now works, before it was open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator/**.
So, I tried around a bit more. What works for me, is the one above, that is I have the following set:
Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-operator
Also, this works fine:
Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/opentelemetry-{operator,collector}
After running the replication, I have both charts with all their versions replicated in Harbor.
But this does not work. Nothing will be replicated:
Source resource filter
Name: open-telemetry.github.io/opentelemetry-helm-charts/**
But replication is indicated as a success.
@HaveFun83 Does this work for you?
@HaveFun83 I hope you are not too busy but maybe you could have a look at my last answer? Thank you :-)