IssueTracker

I tried this and it looks exactly the same as it used to.

2023/12/27 16:40:44 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=10&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7581.595bf074b.fix-gphotos-nil-pointer
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231227T164044Z
Accept-Encoding: gzip

with result

<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
  <Name>rclone-test1</Name>
  <Prefix>
  </Prefix>
  <Marker>
  </Marker>
  <GenerationMarker>
  </GenerationMarker>
  <NextMarker>file1.txt</NextMarker>
  <NextGenerationMarker>1588355689123026</NextGenerationMarker>
  <MaxKeys>10</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>true</IsTruncated>
  <Encoding-Type>url</Encoding-Type>

So ListBucketResult is wrong and there is no NextKeyMarker even though <IsTruncated>true</IsTruncated>

So I don't know whether the change hasn't been rolled out or what?

Hello,

Thank you for your responses.

Cristiano@animati.com.br - as you mentioned in comment #9, it seems that the issue is now fixed for you, and things are working fine. Can you please confirm the same?.

And Nickcw@gmail.com - for your case, I'm checking with the product engineering team on what we can do for you; until then, please hold on. Any further updates will be communicated here. As mentioned earlier in comment #4, It is not recommended to provide project identifiers in public issue trackers. Please do not provide any sensitive information on this public issue. Please make a note of this for any future comments on any public issue tracker. And for now I have Restricted the comments which contain sensitive information by taking note of the project identifiers which you have shared in your comments.

Thank you.

Hello,

Thanks for your patience.

Cristiano@animati.com.br - Thanks for your reply. Okay, This has now been enabled for Nickcw@gmail.com’s project as well, so that he can confirm from his end.

The product engineering team is currently in the process of initiating the MSA to modify the default behavior. They are actively working on it, with a sync coming up, planning to enable this feature for everyone and remove the allowlist. Following the completion of the MSA process, it will be applicable to all your projects. Presently, it is limited to the projects where you specifically requested this feature to be enabled.

Nickcw@gmail.com - for your case, I’m glad to inform you that the product engineering team has now enabled this feature for you as well, your project (2********644) has been added to this feature allowlist. Can you please check and verify the same from your end?

Awaiting your response.

Unfortunately this isn't working for me yet.

I try a query like this

2024/01/16 17:58:55 DEBUG : HTTP REQUEST (req 0xc000def560)
2024/01/16 17:58:55 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=5&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7619.dd4590044.fix-7567-stats-race
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240116T175855Z
Accept-Encoding: gzip

And I get a plain ListBucketResult response with IsTruncated set and no NextKeyMarker

So I would say this is not working on my account.

For reference I tested this with rclone doing this on a versioned bucket, setting the list page to 5 in order to trigger the problem.

rclone lsf --s3-list-chunk 5 --s3-versions gcss3:rclone-test1 -vv --dump bodies

Rclone gives this error

Failed to lsf with 2 errors: last error was: error in ListJSON: s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker

Hello Nickcw@gmail.com,

I’m sorry to hear that this is not yet working for you. I’ve let the product engineering team know about your issue. I'll update you once I get an update from them.

Hello Nickcw@gmail.com,

I have an update for you. The reason it wasn't working last time was that the fix applied to your project had not been fully rolled out to production at that time. Apologies for any inconvenience caused. However, it is expected that it should be working for you by now, for your project(2********644). Can you please check and verify the same from your end?

Awaiting your response.

This is now giving the correct ListVersionResult rather than the ListBucketResult it was previously so that is good, however this is not yet in spec with the AWS documentation because the NextKeyMarker is missing even though IsTruncated is true.

I'm testing with

rclone lsf --s3-list-chunk 5 --s3-versions gcss3:rclone-test1 -vv --dump bodies

The request looks like this

2024/01/25 16:39:40 DEBUG : HTTP REQUEST (req 0xc0004df440)
2024/01/25 16:39:40 DEBUG : GET /?delimiter=%2F&encoding-type=url&max-keys=5&prefix=&versions= HTTP/1.1
Host: rclone-test1.storage.googleapis.com
User-Agent: rclone/v1.66.0-beta.7672.6e4dd2ab9
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240125T163940Z
Accept-Encoding: gzip

And the response looks like this

2024/01/25 16:39:41 DEBUG : HTTP RESPONSE (req 0xc0004df440)
2024/01/25 16:39:41 DEBUG : HTTP/2.0 200 OK
Content-Length: 1218
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Thu, 25 Jan 2024 16:39:41 GMT
Expires: Thu, 25 Jan 2024 16:39:41 GMT
Server: UploadServer
X-Goog-Metageneration: 2
X-Guploader-Uploadid: ABPtcPrHXcr3qpD4eDY6Tq-v37hteJUdC8qWH_M7mCpGC9fOyWeON81oP19MeKFzkanbIBLXv1r_uv67gQ

With the XML body prettified as

<?xml version='1.0' encoding='UTF-8'?>
<ListVersionsResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
  <Name>rclone-test1</Name>
  <Prefix>
  </Prefix>
  <Marker>
  </Marker>
  <GenerationMarker>
  </GenerationMarker>
  <VersionIdMarker>
  </VersionIdMarker>
  <KeyMarker>
  </KeyMarker>
  <NextMarker>dir/</NextMarker>
  <MaxKeys>5</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>true</IsTruncated>
  <Encoding-Type>url</Encoding-Type>
  <Version>
    <Key>bajipin0waj</Key>
    <Generation>1648193671637937</Generation>
    <MetaGeneration>1</MetaGeneration>
    <VersionId>1648193671637937</VersionId>
    <IsLatest>true</IsLatest>
    <LastModified>2022-03-25T07:34:31.674Z</LastModified>
    <DeletedTime>
    </DeletedTime>
    <ETag>"1aaef37ea4d768a633040174c5949d7d"</ETag>
    <Size>38</Size>
  </Version>
  <Version>
    <Key>dahof</Key>
    <Generation>1648193671637965</Generation>
    <MetaGeneration>1</MetaGeneration>
    <VersionId>1648193671637965</VersionId>
    <IsLatest>true</IsLatest>
    <LastModified>2022-03-25T07:34:31.674Z</LastModified>
    <DeletedTime>
    </DeletedTime>
    <ETag>"50579bafba71b03b5eda2fa69332994e"</ETag>
    <Size>57</Size>
  </Version>
  <CommonPrefixes>
    <Prefix>B/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>dir-1/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>dir/</Prefix>
  </CommonPrefixes>
</ListVersionsResult>

Here we can see <IsTruncated>true</IsTruncated> but we see <NextMarker>dir/</NextMarker> whereas we should be seeing

   <NextKeyMarker>string</NextKeyMarker>
   <NextVersionIdMarker>string</NextVersionIdMarker>

Hello Nickcw@gmail.com,

Thanks for your patience.

I have an update for you. The engineering team has addressed your last response, and they have implemented a change to make the NextKeyMarker tag available in the List response as well. This change has been fully rolled out and moved to production. I hope it should be working for you by now. Could you please check and verify the same from your end?

Awaiting your response.

Hello Nickcw@gmail.com,

We haven't heard back from you for some time now. Have you had a chance to review the recent changes made by the engineering team? Are they working fine for you as expected, or do you notice any other issues or discrepancies? Additionally, if you have any other concerns, please feel free to share them.

Awaiting your feedback.

Sorry for the delay - I think this is all working as it should be now.

Thank you for the fix.

-- Nick

Hello Nick,

Thank you for replying back!

I am glad to hear that the issue is resolved for you, and everything is now working as it should. However, as the MSA is yet to be announced by the product engineering team, I’ll keep this thread open; any communication regarding the same will be updated here for all customers. If you encounter any further issues or have any additional concerns or feedback related to the current issue, please feel free to let us know.

If you have any other issues or want to report a new issue, please don't hesitate to create a new issue on the IssueTracker providing a detailed description of your issue and the steps to reproduce it.

Thank you for your continued trust and support to improve Google Cloud Platform products.