Status Update
No update yet.
Comments
go...@google.com
Hello,
Thank you for reaching out. An internal bug is already filed for this case. Please keep in mind that this bug has to be analyzed and considered by the product team and I can't provide you with an ETA for it to be delivered. However, you can keep track of the status by following this thread.
Thanks.
tk...@redhat.com
ja...@cwik.org
I'm seeing the issue too. You can see the GCS header signing is using a different value than my request:
GET /bucket?list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: localhost:8080
User-Agent: aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.22.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 603b64d7-186a-431a-a645-42cf8af8b9c7
Amz-Sdk-Request: attempt=1; max=1
Authorization: AWS4-HMAC-SHA256 Credential=FOOBAR/20250314/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;am
z-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=926d520043cae9e9cbde3c33ffd9c855b2f9de98c2362f4d6b38
655f3ebc3a59
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250314T213902Z
HTTP/1.1 403 Forbidden
x-guploader-uploadid: AKDAyIuhrxVxHD8AdheGgMvaNwYwE0uUpfIr231GR5nocVuBJfiF9aqjPg0hPyHkqGz7pq4v5BP1v6Y
content-type: application/xml; charset=UTF-8
content-length: 943
date: Fri, 14 Mar 2025 21:39:02 GMT
expires: Fri, 14 Mar 2025 21:39:02 GMT
cache-control: private, max-age=0
server: envoy
x-envoy-upstream-service-time: 25
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>Access denied.</Message><Details>The request sig
nature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Details><StringToSign>AWS4-HMAC-SHA256
20250314T213902Z
20250314/us-east-1/s3/aws4_request
e94cc47c47443457e77f0227d51a9302f4daeb5836f26afb1cb84b26d613a4b8</StringToSign><CanonicalRequest>GET
/ddn-test-flat
list-type=2&max-keys=1000&prefix=
accept-encoding:identity,gzip(gfe)
amz-sdk-invocation-id:603b64d7-186a-431a-a645-42cf8af8b9c7
amz-sdk-request:attempt=1; max=1
host:localhost:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20250314T213902Z
accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest></Error>
I sent:
Accept-Encoding: identity
The server is signing:
accept-encoding:identity,gzip(gfe)
Not only is this a signing bug, but it appears that GCS is overriding the requested encoding and could return compressed content the client is not expecting.
ja...@cwik.org
I have identified a workaround. If you disable the middleware "DisableAcceptEncodingGzip", the issue appears to be bypassed. Not sure if this will cause other issues though, since this filter sounds like it was implemented to prevent auto-decompression of gzipped objects.
e.g.
var apiOptions []func(*middleware.Stack) error
if isGCS {
apiOptions = append(apiOptions, func(stack *middleware.Stack) error {
_, err := stack.Finalize.Remove("DisableAcceptEncodingGzip")
return err
})
}
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.APIOptions = apiOptions
})
Description
The example found athttps://cloud.google.com/storage/docs/aws-simple-migration does not work when implemented using the current v2 version of the aws s3 sdk.
Also documented here:https://github.com/minio/minio-java/issues/1067
We will have to use S3 instead until there is a fix/workaround.
Problem you have encountered:
The service account has the
Storage Adminrole. Error encountered:The AWS SDK sends a slightly different header from what GCP expects which breaks the API.
The "Authorization" header has an additional "accept-encoding" which the API does not handle:
"SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date"
What you expected to happen:
Interoperability should work with the v2 AWS golang API.
Steps to reproduce:
See attached example.
Other information (workarounds you have tried, documentation consulted, etc):