Change theme
Help
Press space for more information.
Show links for this issue (Shortcut: i, l)
Copy issue ID
Previous Issue (Shortcut: k)
Next Issue (Shortcut: j)
Sign in to use full features.
Vote: I am impacted
Notification menu
Refresh (Shortcut: Shift+r)
Go home (Shortcut: u)
Pending code changes (auto-populated)
View issue level access limits(Press Alt + Right arrow for more information)
Request for new functionality
View staffing
Description
Since this morning 8:25 CET we have encountered a lot of issues on query jobs running on the EU multi region.
Our software launch thousands of various query jobs by hour across multiple projects. Jobs that usually take minutes to execute are kept in pending status for hours.
We tried switching projects and every time the first few jobs ran as expected then they are all created with a pending status and never processed.
This issue is inconsistent as things went back to normal between 12h and 14h but then they went wrong again.
Note : most of our queries are launched with INTERACTIVE priority but the issue exists also with BATCH priority.
We have seen that other users are also impacted (
What you expected to happen:
Jobs should run within minutes.
Steps to reproduce:
I don't have a step by step way to reproduce it but running 1000 jobs on a project seems to do the trick. Lately, about ten jobs is enough to overload a project.
Other information (workarounds you have tried, documentation consulted, etc):
- Using other GCP project
- No issue with jobs running in the US. We cannot use US as all our datasets are located in EU.