IssueTracker

Problem you have encountered:

Since we switched to Google Artifact Registry from Google Container Registry we have observed significant degradation with images tags listing  e.t.c. either from the UI or with gcloud commands.

For example the following command to list all tags of a specific repo/package (gcloud artifacts docker tags list us-docker.pkg.dev/staging-artifacts-786a/gcr.io/ats) returned after 5m30s.

We think by correlation that this also affects our CD (flux). Image automation from time to time scans the repo and returns the same tags and does not track newer pushes from time to time. Today it occurred again, flux reports that scanned and did not find any new images while there were uploaded. We managed to mitigate the problem by deleting some old images.

We  added cleanup policies to have fewer images that seems to improve the situation but in our eyes the repo size should not have such big impact (5m30s to fetch tags).

What you expected to happen:

expect same performance as GCR

Steps to reproduce:
we replicated the artifacts to another project (sre-artifacts-20e4) using GCR (same location as GAR)

 just made some rough comparisons and wanted to share with you the following findings:
 
GAR
time -p gcloud artifacts docker tags list us-docker.pkg.dev/staging-artifacts-786a/gcr.io/ats
real 114.64
user 8.81
sys 0.40

4581 distinct tags
 
GCR
time -p gcloud container images list-tags gcr.io/sre-artifacts-20e4/ats
real 2.19
user 1.60
sys 0.14

1432 distinct tags

Even though the data volume is not the same, it's clear the wall-clock times differ between GAR and GCR significantly.

We want to know whether its expected behavior.