IssueTracker

Description

Beginning around 2024-09-13 08:03 PST most of our requests to https://gmail.googleapis.com/gmail/v1/users/me/history began failing with with a response of 429. These errors were accompanied by an HTML body that has the text of "Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot." This went on to also point out that it was a particular IP address being blocked and wanted us to solve a CAPTCHA.

We (Shortwave) were affected and we believe that Streak and Polymail were as well (given the timing of their downtime).

Mitigation

We are using Kubernetes on Google Cloud to host our servers. We found that the errors were coming only from specific pods. We ended up cordoning off two nodes out of our node pool that were seeing these errors (and deleting the offending pods), afterwards our traffic returned to normal. Interestingly one node was in zone us-central1-b and the other in us-central1-c however we still had other nodes in both of those zones that were not getting any 429s.

Suspicion

My suspicion is that some other Google Cloud customer had some bad code (or was just nefarious) and making lots of requests that might have been breaking the Gmail TOS. We just happened to have two of our nodes on the same physical machines as this user. So when Gmail added the IP ban for those machines we happened to get caught up in it as well.

Ideally rather than banning a particular IP address completely there should be some kind of check against the OAuth client ID instead. Granted, if there is a DDOS occurring then it might be too expensive to make those checks.