IssueTracker

When sending a request to a prediction endpoint it will exceed the size limit of 1.5 MB even though the request body is below that limit.

Problem you have encountered:

We have created a batcher which groups instances for the predictions so that the limit is honored. It uses "compact" JSON serialization (no newlines, no indents). We then string-join the separate "instance" JSON strings to form a request body.

We are sending a request which we know is below limit in bytesize (1572847 bytes). This is below 1.5MB (1572864 bytes). However, we still receive an error from the endpoint:

{
  "error": {
    "code": 400,
    "message": "The request size (1674849 bytes) exceeds 1.500MB limit.",
    "status": "FAILED_PRECONDITION"
  }
}

What you expected to happen:

We expected the request body to be accepted. It is valid JSON and it is below the bytesize limit.

The suspicion we have is that the API gateway for the endpoint parses JSON and then re-emits it, and then checks for bytesize. This is opaque and makes it hard for a developer to adhere to the limit in the first place. The bytesize should instead be checked against the actual HTTP request body "as received".

Steps to reproduce:

Generate a JSON request body using "compact" JSON serialization (no newlines, no indents), which would be just below the bytesize limit. Send it to the endpoint. You will likely observe the same error.

We have implemented a workaround by having to generate "pretty" JSON serialization when batching. In practice this leads to a substantial increase of JSON serializations we need to do, because indented/"pretty" serialization is hard to precompute for bytesize. This leads to some waste of compute.