Recently, my Google App Engine (GAE) platform started returning 403 Forbidden errors to all incoming requests even if the IP is allowed access. After investigating, I discovered that I had 1002 active firewall rules in place. Interestingly, as soon as I manually deleted some rules and the count dropped below 1000 rules, the platform resumed normal operation, and the 403 errors disappeared.

The problem raises two major concerns:

1. Why did GAE give blanket 403 errors for all incoming requests (even from IPs not blocked by the firewall) after the firewall rule count exceeded 1000?

2. Why did the API allow the rule count to exceed 1000, when in the past it consistently rejected any attempts to go beyond this limit with the following error message:

{'error': {'code': 400, 'message': 'Cannot add rule. Total rule count may not exceed 1000 rules', 'status': 'INVALID_ARGUMENT'}}

I expected the following:
1. The firewall API would not allow more than 1000 rules
2. If at all somehow more than 1000 rules are created, why in the world would GAE start giving 403 to every single incoming requests!?

I do not have a proven way to share how this can be reproduced, this has happened for the first time with me. Best guess is to try creating firewall rules via the API parallelly. In some edge case, when at 999, the API allows more than 1 rule in parallel to get created.

Additional Details:
Platform: Google App Engine Standard Environment
Firewall Rules: Mixture of IP blocks, both specific IPs and CIDR ranges (subnets)

This seems to be an edge case or a potential bug but is very concerning as all of my end users were unable to use the platform (and saw a 403) which reduces their trust on the application. Any insights or documentation around this behavior would be greatly appreciated.

Note that it was not the case that a deny all rule got created by us. We know this because we deleted only a few rules specific to an IP to bring total count below 1000 and platform was up again!

Issue 373794473

Links (2)

Issue metadata