helm pull/push randomly hangs with GCP Artifact Registry [255248281]

Assigned

Bug

Status Update

No update yet.

Description

dy...@variphy.com

created issue #1

Oct 24, 2022 07:08PM

Hello,

This is a reproduction of an issue filed in the Helm repository (in case the issue is actually on the GCP side): https://github.com/helm/helm/issues/11415

Problem you have encountered:

I'm seeing an issue on version 3.10.0 of helm where push/pull to/from GCP Artifact Registry seems to randomly hang. For example, maybe 2 out of 3 times, a pull will work, but then it seems to hang.

When things work, it looks like this:

$ helm pull oci://us-west1-docker.pkg.dev/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE --version CHART_VERSION_HERE --debug
DEBU[0000] resolving                                     host=us-west1-docker.pkg.dev
DEBU[0000] do request                                    host=us-west1-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=HEAD url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] fetch response received                       host=us-west1-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=102 response.header.content-type=application/json response.header.date="Wed, 05 Oct 2022 23:01:05 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://us-west1-docker.pkg.dev/v2/token\",service=\"us-west1-docker.pkg.dev\",scope=\"repository:PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE:pull\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://us-west1-docker.pkg.dev/v2/token\",service=\"us-west1-docker.pkg.dev\",scope=\"repository:PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE:pull\"" host=us-west1-docker.pkg.dev
DEBU[0000] do request                                    host=us-west1-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=HEAD url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] fetch response received                       host=us-west1-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=354 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Wed, 05 Oct 2022 23:01:05 GMT" response.header.docker-content-digest="sha256:DIGEST_HERE" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] resolved                                      desc.digest="sha256:DIGEST_HERE" host=us-west1-docker.pkg.dev
DEBU[0000] do request                                    digest="sha256:DIGEST_HERE" request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=GET url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/sha256:DIGEST_HERE"
DEBU[0000] fetch response received                       digest="sha256:DIGEST_HERE" response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=354 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Wed, 05 Oct 2022 23:01:05 GMT" response.header.docker-content-digest="sha256:DIGEST_HERE" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/sha256:DIGEST_HERE"
DEBU[0000] do request                                    digest="sha256:DIGEST_HERE" request.header.accept="application/vnd.cncf.helm.chart.content.v1.tar+gzip, */*" request.header.user-agent=Helm/3.10.0 request.method=GET url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/blobs/sha256:DIGEST_HERE"
DEBU[0000] do request                                    digest="sha256:DIGEST_HERE" request.header.accept="application/vnd.cncf.helm.config.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=GET url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/blobs/sha256:DIGEST_HERE"
DEBU[0000] fetch response received                       digest="sha256:DIGEST_HERE" response.header.accept-ranges=bytes response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control="private, max-age=0" response.header.content-length=24086 response.header.content-type=application/octet-stream response.header.date="Wed, 05 Oct 2022 23:01:06 GMT" response.header.expires="Wed, 05 Oct 2022 23:01:06 GMT" response.header.server=UploadServer response.header.x-goog-hash="HASH_HERE" response.header.x-guploader-uploadid=ID_HERE response.status="200 OK" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/blobs/sha256:DIGEST_HERE"
DEBU[0000] fetch response received                       digest="sha256:DIGEST_HERE" response.header.accept-ranges=bytes response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=304 response.header.content-type=application/octet-stream response.header.date="Wed, 05 Oct 2022 23:01:06 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/blobs/sha256:DIGEST_HERE"
DEBU[0000] encountered unknown type application/vnd.cncf.helm.config.v1+json; children may not be fetched 
DEBU[0000] encountered unknown type application/vnd.cncf.helm.chart.content.v1.tar+gzip; children may not be fetched 
Pulled: us-west1-docker.pkg.dev/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE:CHART_VERSION_HERE
Digest: sha256:DIGEST_HERE

But then when it hangs, it looks like this:

$ helm pull oci://us-west1-docker.pkg.dev/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE --version CHART_VERSION_HERE --debug
DEBU[0000] resolving                                     host=us-west1-docker.pkg.dev
DEBU[0000] do request                                    host=us-west1-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=HEAD url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] fetch response received                       host=us-west1-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=102 response.header.content-type=application/json response.header.date="Wed, 05 Oct 2022 23:01:07 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://us-west1-docker.pkg.dev/v2/token\",service=\"us-west1-docker.pkg.dev\",scope=\"repository:PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE:pull\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://us-west1-docker.pkg.dev/v2/token\",service=\"us-west1-docker.pkg.dev\",scope=\"repository:PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE:pull\"" host=us-west1-docker.pkg.dev
DEBU[0000] do request                                    host=us-west1-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.0 request.method=HEAD url="https://us-west1-docker.pkg.dev/v2/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE/manifests/CHART_VERSION_HERE"

What you expected to happen: I expect pulls and push with Helm to not hang.

Steps to reproduce: Attempt to pull a chart from a GCP Artifact Registry Helm repository and note that it can hang. E.g. attempting to do something like this a few times usually results in Helm hanging: helm pull oci://us-west1-docker.pkg.dev/PROJECT_NAME_HERE/REPO_NAME_HERE/CHART_NAME_HERE --version CHART_VERSION_HERE --debug

Other information (workarounds you have tried, documentation consulted, etc): I have been able to work around this by leveraging the timeout command in build scripts. e.g.

/bin/bash -c \
  "until 
  timeout 5 \
    helm pull oci://us-west1-docker.pkg.dev/$PROJECT_NAME/$REPO_NAME/$CHART_NAME --version $CHART_VERSION; 
  do 
    echo 'helm hung; retrying...'; 
  done"

for example:

$ /bin/bash -c   "until 
  timeout 5 \
    helm pull oci://us-west1-docker.pkg.dev/$PROJECT_NAME/$REPO_NAME/$CHART_NAME --version $CHART_VERSION; 
  do 
    echo 'helm hung; retrying...'; 
  done"
helm hung; retrying...
Pulled: us-west1-docker.pkg.dev/PROJECT_NAME/REPO_NAME/CHART_NAME:CHART_VERSION
Digest: sha256:DIGEST_HERE

Comments

go...@google.com <go...@google.com> Oct 25, 2022 02:16AM

Assigned to go...@google.com.

go...@google.com <go...@google.com> #2Oct 25, 2022 03:17AM

I have forwarded this request to the engineering team. We will update this issue with any progress updates and a resolution.

Best Regards,
Josh Moyer
Google Cloud Platform Support

go...@google.com <go...@google.com> Nov 1, 2022 10:10PM

Reassigned to gc...@google.com.

cm...@gmail.com <cm...@gmail.com> #3Nov 2, 2022 01:05PM

This is not only useful for IP addresses, but also for many other resources. I understand that names are currently used as identifiers, so this request is probably not trivial to implement. Maybe distinguishing between a (numeric, automatically generated) identifier and a (textual) label is the way to go?

cm...@gmail.com <cm...@gmail.com> #4Nov 4, 2022 04:48PM

Is it any hope? We have migrated our IP address to the server with different role, and now the name of this IP address resource doesn't match its role at all. It seems to be trivial enough to momentary reserve static IP address of the old named resource, drop resource, and immediately recreate it with the new name and the old IP address.

Message last modified on Nov 4, 2022 05:00PM

Issue 255248281