Status Update
Comments
ba...@google.com <ba...@google.com>
ni...@gmail.com <ni...@gmail.com> #2
Here is the problem summarized from the link above:
When you issue a list with versions
GET /?delimiter=&encoding-type=url&max-keys=1000&prefix=pacs-data%2Farchive%2F2022%2F8%2F&versions= HTTP/1.1
It gives this list response
<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
<Name>nd-srv</Name>
<Prefix>pacs-data/archive/2022/8/</Prefix>
<Marker>
</Marker>
<GenerationMarker>
</GenerationMarker>
<NextMarker>pacs-data/archive/2022/8/1/10/11992158/8EC0C0C4/77407295</NextMarker>
<NextGenerationMarker>1683228165974407</NextGenerationMarker>
<MaxKeys>1000</MaxKeys>
<IsTruncated>true</IsTruncated>
<Encoding-Type>url</Encoding-Type>
<Version>
<Key>pacs-data/archive/2022/8/1/10/08E75EBF/F9A5759E/271EE0F6</Key>
<Generation>1683224794134204</Generation>
<MetaGeneration>1</MetaGeneration>
<IsLatest>true</IsLatest>
<LastModified>2023-05-04T18:26:34.136Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"ea549ded855170e0670a00e5d2e13a0a"</ETag>
<Size>54366</Size>
</Version>
...snip lots of similar blocks...
<Version>
<Key>pacs-data/archive/2022/8/1/10/11992158/8EC0C0C4/77407295</Key>
<Generation>1683228165974407</Generation>
<MetaGeneration>1</MetaGeneration>
<IsLatest>true</IsLatest>
<LastModified>2023-05-04T19:22:45.975Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"361b28d02accd86fef5cf92791795699"</ETag>
<Size>243496</Size>
</Version>
</ListBucketResult>
This list response does not match the response expected from an S3 compatible API.
According to
<?xml version="1.0" encoding="UTF-8"?>
<ListVersionsResult>
<IsTruncated>boolean</IsTruncated>
<KeyMarker>string</KeyMarker>
<VersionIdMarker>string</VersionIdMarker>
<NextKeyMarker>string</NextKeyMarker>
<NextVersionIdMarker>string</NextVersionIdMarker>
<Version>
<ChecksumAlgorithm>string</ChecksumAlgorithm>
...
<ETag>string</ETag>
<IsLatest>boolean</IsLatest>
<Key>string</Key>
<LastModified>timestamp</LastModified>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
<RestoreStatus>
<IsRestoreInProgress>boolean</IsRestoreInProgress>
<RestoreExpiryDate>timestamp</RestoreExpiryDate>
</RestoreStatus>
<Size>long</Size>
<StorageClass>string</StorageClass>
<VersionId>string</VersionId>
</Version>
...
<DeleteMarker>
<IsLatest>boolean</IsLatest>
<Key>string</Key>
<LastModified>timestamp</LastModified>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
<VersionId>string</VersionId>
</DeleteMarker>
...
<Name>string</Name>
<Prefix>string</Prefix>
<Delimiter>string</Delimiter>
<MaxKeys>integer</MaxKeys>
<CommonPrefixes>
<Prefix>string</Prefix>
</CommonPrefixes>
...
<EncodingType>string</EncodingType>
</ListVersionsResult>
In particular note that the response should be a ListVersionsResult
but gcs has given us a ListBucketResult
Also note that NextKeyMarker
and NextVersionIdMarker
are missing which means that a program using the S3 API as per the AWS docs won't be able to get the next page of data.
This is as documented in
ku...@google.com <ku...@google.com> #3
Hello,
Thank you for reaching out to us with your request.
We have duly noted your feedback and this has been forwarded to the engineering team so that they may evaluate it and validate it thoroughly. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issue, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.
We appreciate your continued trust and support in improving our Google Cloud Platform products. In case you want to report a new issue, Please do not hesitate to create a new issue on the
Once again, we sincerely appreciate your valuable feedback; Thank you for your understanding and collaboration.
ku...@google.com <ku...@google.com> #4
Hello,
Thank you for your patience.
I’ve got an update from the Engineering team: This feature is currently allow-listed due to the potential to disrupt customers who are currently reliant on the existing implementation. The team is in the process of initiating the MSA (Mandatory Service Announcement) procedure to modify the default behavior. Meanwhile, the team has asked for your project details, this will enable them to check more on the issue.
Please note that I’ve created a Confidential issue with the same issue title as here for you to provide your project details, for which you must have received an email notification. Kindly provide the requested details over the confidential issue only and do not provide any sensitive information here on this public issue.
Awaiting your response.
ku...@google.com <ku...@google.com> #5
Hello,
Thank you for providing the requested details in the confidential issue.
The details have been forwarded to the engineering team for further investigation. Please stay tuned to this issue, any further communication will be made here. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issue, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.
ku...@google.com <ku...@google.com>
ku...@google.com <ku...@google.com> #6
Hello,
Thank you for your patience.
I’m glad to inform you that by now our product engineering team has taken the necessary steps to address and resolve the reported issue. Can you please verify the list object response for your submitted project to ensure that it is now working as expected for you?
Awaiting your response.
ni...@gmail.com <ni...@gmail.com> #7
I tried this and it looks exactly the same as it used to.
2023/12/27 16:40:44 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=10&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7581.595bf074b.fix-gphotos-nil-pointer
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231227T164044Z
Accept-Encoding: gzip
with result
<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
<Name>rclone-test1</Name>
<Prefix>
</Prefix>
<Marker>
</Marker>
<GenerationMarker>
</GenerationMarker>
<NextMarker>file1.txt</NextMarker>
<NextGenerationMarker>1588355689123026</NextGenerationMarker>
<MaxKeys>10</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>true</IsTruncated>
<Encoding-Type>url</Encoding-Type>
So ListBucketResult
is wrong and there is no NextKeyMarker
even though <IsTruncated>true</IsTruncated>
So I don't know whether the change hasn't been rolled out or what?
cb...@gmail.com <cb...@gmail.com> #8
cr...@animati.com.br <cr...@animati.com.br> #9
I think Google made the adjustments exclusively for my project. Can you provide your project name and id?
ni...@gmail.com <ni...@gmail.com> #10 Restricted
cr...@animati.com.br <cr...@animati.com.br> #11 Restricted
ku...@google.com <ku...@google.com> #12
Hello,
Thank you for your responses.
And
Thank you.
cr...@animati.com.br <cr...@animati.com.br> #13
and with this problem solved. It looks like another one has appeared which depends on the nick analysis.
We have more than 50 projects that depend on this modification. Will it apply to everyone?
ku...@google.com <ku...@google.com> #14
Hello,
Thanks for your patience.
The product engineering team is currently in the process of initiating the MSA to modify the default behavior. They are actively working on it, with a sync coming up, planning to enable this feature for everyone and remove the allowlist. Following the completion of the MSA process, it will be applicable to all your projects. Presently, it is limited to the projects where you specifically requested this feature to be enabled.
2********644
) has been added to this feature allowlist. Can you please check and verify the same from your end?
Awaiting your response.
ni...@gmail.com <ni...@gmail.com> #15
Unfortunately this isn't working for me yet.
I try a query like this
2024/01/16 17:58:55 DEBUG : HTTP REQUEST (req 0xc000def560)
2024/01/16 17:58:55 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=5&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7619.dd4590044.fix-7567-stats-race
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240116T175855Z
Accept-Encoding: gzip
And I get a plain ListBucketResult
response with IsTruncated set and no NextKeyMarker
So I would say this is not working on my account.
For reference I tested this with rclone doing this on a versioned bucket, setting the list page to 5 in order to trigger the problem.
rclone lsf --s3-list-chunk 5 --s3-versions gcss3:rclone-test1 -vv --dump bodies
Rclone gives this error
Failed to lsf with 2 errors: last error was: error in ListJSON: s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker
ku...@google.com <ku...@google.com> #16
I’m sorry to hear that this is not yet working for you. I’ve let the product engineering team know about your issue. I'll update you once I get an update from them.
ku...@google.com <ku...@google.com> #17
I have an update for you. The reason it wasn't working last time was that the fix applied to your project had not been fully rolled out to production at that time. Apologies for any inconvenience caused. However, it is expected that it should be working for you by now, for your project(2********644
). Can you please check and verify the same from your end?
Awaiting your response.
ni...@gmail.com <ni...@gmail.com> #18
This is now giving the correct ListVersionResult
rather than the ListBucketResult
it was previously so that is good, however this is not yet in spec with the AWS documentation because the NextKeyMarker
is missing even though IsTruncated
is true
.
I'm testing with
rclone lsf --s3-list-chunk 5 --s3-versions gcss3:rclone-test1 -vv --dump bodies
The request looks like this
2024/01/25 16:39:40 DEBUG : HTTP REQUEST (req 0xc0004df440)
2024/01/25 16:39:40 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=5&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7672.6e4dd2ab9
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240125T163940Z
Accept-Encoding: gzip
And the response looks like this
2024/01/25 16:39:41 DEBUG : HTTP RESPONSE (req 0xc0004df440)
2024/01/25 16:39:41 DEBUG : HTTP/2.0 200 OK
Content-Length: 1218
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Thu, 25 Jan 2024 16:39:41 GMT
Expires: Thu, 25 Jan 2024 16:39:41 GMT
Server: UploadServer
X-Goog-Metageneration: 2
X-Guploader-Uploadid: ABPtcPrHXcr3qpD4eDY6Tq-v37hteJUdC8qWH_M7mCpGC9fOyWeON81oP19MeKFzkanbIBLXv1r_uv67gQ
With the XML body prettified as
<?xml version='1.0' encoding='UTF-8'?>
<ListVersionsResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
<Name>rclone-test1</Name>
<Prefix>
</Prefix>
<Marker>
</Marker>
<GenerationMarker>
</GenerationMarker>
<VersionIdMarker>
</VersionIdMarker>
<KeyMarker>
</KeyMarker>
<NextMarker>dir/</NextMarker>
<MaxKeys>5</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>true</IsTruncated>
<Encoding-Type>url</Encoding-Type>
<Version>
<Key>bajipin0waj</Key>
<Generation>1648193671637937</Generation>
<MetaGeneration>1</MetaGeneration>
<VersionId>1648193671637937</VersionId>
<IsLatest>true</IsLatest>
<LastModified>2022-03-25T07:34:31.674Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"1aaef37ea4d768a633040174c5949d7d"</ETag>
<Size>38</Size>
</Version>
<Version>
<Key>dahof</Key>
<Generation>1648193671637965</Generation>
<MetaGeneration>1</MetaGeneration>
<VersionId>1648193671637965</VersionId>
<IsLatest>true</IsLatest>
<LastModified>2022-03-25T07:34:31.674Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"50579bafba71b03b5eda2fa69332994e"</ETag>
<Size>57</Size>
</Version>
<CommonPrefixes>
<Prefix>B/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>dir-1/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>dir/</Prefix>
</CommonPrefixes>
</ListVersionsResult>
Here we can see <IsTruncated>true</IsTruncated>
but we see <NextMarker>dir/</NextMarker>
whereas we should be seeing
<NextKeyMarker>string</NextKeyMarker>
<NextVersionIdMarker>string</NextVersionIdMarker>
ku...@google.com <ku...@google.com> #19
Thanks for your patience.
I have an update for you. The engineering team has addressed your last response, and they have implemented a change to make the NextKeyMarker
tag available in the List response as well. This change has been fully rolled out and moved to production. I hope it should be working for you by now. Could you please check and verify the same from your end?
Awaiting your response.
ku...@google.com <ku...@google.com> #20
We haven't heard back from you for some time now. Have you had a chance to review the recent changes made by the engineering team? Are they working fine for you as expected, or do you notice any other issues or discrepancies? Additionally, if you have any other concerns, please feel free to share them.
Awaiting your feedback.
ni...@gmail.com <ni...@gmail.com> #21
Sorry for the delay - I think this is all working as it should be now.
Thank you for the fix.
-- Nick
ku...@google.com <ku...@google.com> #22
Hello Nick,
Thank you for replying back!
I am glad to hear that the issue is resolved for you, and everything is now working as it should. However, as the MSA is yet to be announced by the product engineering team, I’ll keep this thread open; any communication regarding the same will be updated here for all customers. If you encounter any further issues or have any additional concerns or feedback related to the current issue, please feel free to let us know.
If you have any other issues or want to report a new issue, please don't hesitate to
Thank you for your continued trust and support to improve Google Cloud Platform products.
Description
All the problem is described in the link bellow: