IssueTracker

Yesterday afternoon, 2021-03-26 around 13:00 CET, i noticed my application in Google App Engine Flexible environment was returning 502's on all requests. I thought this might be related to an outage, since one of the log entries had the error ZONE_RESOURCE_POOL_EXHAUSTED.

I thought something was wrong with my version (although i had not touched it for 3 days), so i wanted to deploy a new one. When trying to deploy the application, i got this message:

ERROR: (gcloud.app.deploy) Error Response: [8] The region europe-west1 does not have enough resources available to fulfull the request. Please try again later.

After checking the logs, i came across these messages in a timeframe of about 2 minutes:

~12:38 CET

IAM deploymentmanager.googleapis.com v2.deploymentmanager.deployments.update projects/XXXX/global/deployments/XXXXX XXXX@gae-api-prod.google.com.iam.gserviceaccount.com audit_log, method: "v2.deploymentmanager.deployments.update", principal_email: "XXXX@gae-api-prod.google.com.iam.gserviceaccount.com"

IAM compute.googleapis.com v1.compute.instanceTemplates.insert projects/XXXXX/global/instanceTemplates/XXXXX XXXX@cloudservices.gserviceaccount.com audit_log, method: "v1.compute.instanceTemplates.insert", principal_email: "XXXX@cloudservices.gserviceaccount.com"

IAM compute.googleapis.com beta.compute.regionInstanceGroupManagers.update projects/XXXXXX/regions/europe-west1/instanceGroupManagers/XXXXXX XXXXX@cloudservices.gserviceaccount.com audit_log, method: "beta.compute.regionInstanceGroupManagers.update", principal_email: "XXXXXX@cloudservices.gserviceaccount.com"

IAM compute.googleapis.com v1.compute.instanceTemplates.delete projects/XXXXX/global/instanceTemplates/XXXXXX XXXXX@cloudservices.gserviceaccount.com audit_log, method: "v1.compute.instanceTemplates.delete", principal_email: "XXXXX@cloudservices.gserviceaccount.com"

IAM compute.googleapis.com v1.compute.regionInstanceGroupManagers.recreateInstances projects/XXXXXX/regions/europe-west1/instanceGroupManagers/XXXXXX XXXXXX@gae-api-prod.google.com.iam.gserviceaccount.com audit_log, method: "v1.compute.regionInstanceGroupManagers.recreateInstances", principal_email: "XXXXXX@gae-api-prod.google.com.iam.gserviceaccount.com"

~12:39 CET, the first errors occur:

IAM compute.googleapis.com v1.compute.instances.update projects/XXXXXX/zones/europe-west1-b/instances/XXXXXX XXXXX@cloudservices.gserviceaccount.com Invalid value for field 'resource.networkInterfaces': ''. Updating 'networkInterface' is not supported.

IAM compute.googleapis.com v1.compute.instanceGroups.removeInstances projects/XXXXX/zones/europe-west1-b/instanceGroups/XXXXXX XXXXXX@cloudservices.gserviceaccount.com Resource projects/XXXXX/zones/europe-west1-b/instances/XXXX is not a member of projects/XXXXXX/zones/europe-west1-b/instanceGroups/XXXXX

~12:40 CET VM shutting down:

------------------------VM XXXX shutdown initiated-----------------------

This is the moment when the /readiness_check starts to fail, and shortly after, this message appears:

IAM compute.googleapis.com v1.compute.instances.insert projects/XXXXX/zones/europe-west1-b/instances/XXXXX XXXXXX@cloudservices.gserviceaccount.com ZONE_RESOURCE_POOL_EXHAUSTED

Then, there are about 20 log entries saying:

IAM compute.googleapis.com v1.compute.regionInstanceGroupManagers.recreateInstances

The instance was down from roughly 12:40 CET till 16:30 CET, and in this timeframe, i could not deploy any new versions either. Was there an outage, or are there any other underlying issues here?