IssueTracker

Hi Google Technical Support team,

I received "The datastore operation timed out, or the data was temporarily unavailable" for one of by database hosted on Google Cloud datastore NDB while doing 'get_by_id' operation to fetch an entity by key.

I already had exponential retries setup with 5 retries but the operation still failed and reported this error.

On analyzing the metrics explorer for, there are few things that we found:
1. There was no sudden increase in requests to this Database including get/put requests.
2. The Average number of requests to this database is low as there is a memcache layer in between to fetch data but it should not be more than 1-2 request/sec
3. Total number of entities in this database is 3336812 with average entity size ~ 200kb.
4. The application is running on Automated scaling on python 3.11 with F4_1G instance class and 10 workers. other configuration in app.yaml is set as :
  min_instances: 2
  min_idle_instances: 1
  max_instances: 10
  target_cpu_utilization: 0.8

 During the time of error we did see a new instance being spun up.
5. This error only occured once for this database.

What could be the possible reason for this error? can you please help me out or suggest something which could help us figure out the root cause of the problem.

I have attached logs reported screenshot for your reference. Looking forward to you speedy reply.

Thank you,