Setting per-user custom query quotas [119608422]

Assigned

Feature Request

Status Update

No update yet.

Description

vg...@google.com

created issue #1

Nov 15, 2018 03:18PM

It is possible to create custom quotas, which will be the same for all users in the one project [1]
It would be nice to be able to set different limits for different users within one project.

[1]:

https://cloud.google.com/bigquery/docs/custom-quotas#controlling_query_costs_using_bigquery_custom_quotas

Comments

bl...@google.com <bl...@google.com> Nov 15, 2018 03:19PM

Assigned to al...@google.com.

al...@google.com <al...@google.com> Nov 17, 2018 12:53AM

Reassigned to we...@google.com.

we...@google.com <we...@google.com> #2Nov 27, 2018 06:51PM

Can you describe how you would like to see this expressed? With potentially thousands of users, how would you like to view and manage their separate quota limits and see their usage?

We're concerned that this visibility problem is unscalable without large changes that will make more common, less-complicated use cases more difficult.

[Deleted User] <[Deleted User]> #3Dec 3, 2018 01:54PM

+1 for this. In terms of usability, I see it as an 'opt-in' thing rather than having to manage thousand of users - i.e, all users will be assumed to fall under the regular per project/per user quotas unless a quota has been specifically set against them.

sr...@egen.ai <sr...@egen.ai> #4Dec 13, 2018 01:27AM

+1 for this. We are working with a customer, who wants the feature to be able to assign a custom quotas for a "group" of users with IAM policy. Currently its only available per project but the users can bypass if they have access from multiple projects.

da...@gmail.com <da...@gmail.com> #5Feb 8, 2019 12:11PM

+1. This would be really useful for us too, as we have both users logged in to GCP and service account accessing BQ

Thanks

ja...@redbull.com <ja...@redbull.com> #6Jul 17, 2019 09:12PM

+1. To solve the UI/visibility/scalability problem, I would create usage quotas for groups, and then associate users to those groups.

ja...@redbull.com <ja...@redbull.com> #7Jul 17, 2019 09:15PM

as I create the groups, I'd have the option to spread the usage associated with the group equally across users, or treat it first-come-first-serve like similar to how the project quota is implemented.

sa...@google.com <sa...@google.com> #8Oct 7, 2019 10:45PM

+1
one may consider separating the end users and the service accounts . this will give us more granularity in those quotas such that the `QueryUsagePerUserPerDay` quota can be set for both groups separately (read: end users and service accounts)?

thanks

we...@google.com <we...@google.com> Oct 14, 2019 06:29PM

Reassigned to se...@google.com.

se...@google.com <se...@google.com> Dec 5, 2019 05:24PM

Reassigned to gc...@google.com.

bw...@google.com <bw...@google.com> Nov 10, 2020 12:42AM

Reassigned to vt...@google.com.

vt...@google.com <vt...@google.com> Nov 20, 2020 08:09PM

Reassigned to da...@google.com.

ku...@gmail.com <ku...@gmail.com> #9Jan 6, 2021 05:16PM

An...@betfred.com <An...@betfred.com> #10Apr 11, 2021 06:54AM

please implement this

bu...@alo-tech.com <bu...@alo-tech.com> #11May 17, 2021 08:00PM

+1 - > waiting for implementation

ef...@gmail.com <ef...@gmail.com> #12May 31, 2021 08:38AM

[Deleted User] <[Deleted User]> #13Jun 17, 2021 10:24AM

d....@gmail.com <d....@gmail.com> #14Oct 18, 2021 01:21PM

da...@moneysupermarketgroup.com <da...@moneysupermarketgroup.com> #15Nov 16, 2021 10:21AM

This would be incredibly useful.

As a cost control measure having a quota limit is a good thing. For investigation and data incident management we would like to assign a quota to a group containing a limited number of users rather than increase the quota for all users.

ja...@growbots.com <ja...@growbots.com> #16Apr 21, 2022 05:50PM

[Deleted User] <[Deleted User]> #17Apr 22, 2022 12:48PM

+1 Very much needed to avoid analysts running wrong queries

as...@myglamm.com <as...@myglamm.com> #18Apr 22, 2022 12:56PM

+1 this is required as a default feature, please! :)

ga...@gmail.com <ga...@gmail.com> #19May 27, 2022 04:00AM

Is this feature being worked on?

de...@petsathome.co.uk <de...@petsathome.co.uk> #20Jul 28, 2022 01:19PM

I have a few ways we could implement this:

We will make the assumption that we give 10TiB

Idea 1:

set a quota of TiB based on a group, and the options are either per user in group or total

where per user in group lets each user in the group get 10Tib

and then where as total is all users in the group have a collective 10TiB to use

Idea 2:

Set a dollar usage following the above idea, where you can set a group, or a specific user.

note, the comments made do not reflect that of my employer, only the author

an...@blablacar.com <an...@blablacar.com> #21Sep 30, 2022 12:29PM

The current limitation has been causing a lot of issues in our organisation. We have services accounts running costly queries every day so we had to raise the user level quota to allow those SAs to run their queries. That implies that normal (real) users benefit from the same quota and could risk running costly queries. Having different quotas per users would allow us to have higher quotas for service accounts vs normal users.
It would also be super useful to be able to raise the quota on a given day for a given user. For example a Data Analyst might need to run on an ad-hoc basis an expensive query. It should be possible for an administrator to raise the quota of only that user rather than for all users at once

[Deleted User] <[Deleted User]> #22Jan 4, 2023 09:04PM

ol...@openvpn.com <ol...@openvpn.com> #23Jan 4, 2023 10:41PM

We need this feature very much. Please implement.

vs...@techtarget.com <vs...@techtarget.com> #24Jan 12, 2023 10:16PM

+ infinity

ma...@priceline.com <ma...@priceline.com> #25Mar 16, 2023 03:12PM

lu...@ingka.ikea.com <lu...@ingka.ikea.com> #26Mar 22, 2023 04:54PM

For me, a simple solution which would solve some of the issues is creating a role which is not affected by quotas. That way we can keep the quota limit for the employees, but assign this special role to service accounts and admins so that they are not affected by the Query use per day per user limit

si...@google.com <si...@google.com> May 3, 2023 09:11AM

Reassigned to si...@google.com.

[Deleted User] <[Deleted User]> #27May 3, 2023 03:09PM

wi...@sybogames.com <wi...@sybogames.com> #28May 9, 2023 04:08PM

+1. We have some large tables, and so poorly managed queries could be quite expensive. We'd like to give access to more people in the business to be able to query our tables, but need to enforce that these less experienced users don't accidentally run up a huge bill. The ideal would be to have these users a group and provide total or per query limits/quotas to reduce risk of surprise bills.

si...@google.com <si...@google.com> Jun 8, 2023 11:02AM

Reassigned to gc...@google.com.

su...@news.com.au <su...@news.com.au> #29Jun 19, 2023 02:28AM

Can we get an update on this feature request please ?

st...@genmills.com <st...@genmills.com> #30Jun 26, 2023 09:22PM

jo...@teamtag.com <jo...@teamtag.com> #31Aug 22, 2023 04:55PM

Just got bit by this, we need to be able to set safe defaults for using this service.

We want to give our teams freedom to use the tool to find data but ensure that at the end of the day the workflow is architected; not some user who is running poorly optimized queries manually.

Ideally, we should be able to tie it to groups such as:

gcp-marketing - 1TB
gcp-dataengineering - 10TB

fa...@aetna.com <fa...@aetna.com> #32Sep 8, 2023 06:02PM

Is this feature being worked on.

da...@springbokagency.com <da...@springbokagency.com> #33Oct 2, 2023 12:00PM

I would like to know too.

de...@servier.com <de...@servier.com> #34Oct 23, 2023 02:33PM

feature needed also here. +1

mp...@mux.com <mp...@mux.com> #35Nov 1, 2023 11:27PM

Same

bw...@google.com <bw...@google.com> Nov 17, 2023 10:56PM

Reassigned to vy...@google.com.

vi...@gmail.com <vi...@gmail.com> #36Dec 19, 2023 09:31AM

+1 need this so that we can set different quotas for different service accounts and users.

However, current Query usage Per Day Per User is not working correctly always. I see many times some users are able to user more than 2 times the quota limit set.

E.g. the Query usage Per Day Per User is set to 20 TB. I have seen many users to be able to query around 43 TB per day.The quotas work sometimes but not always. Sometimes it will stop users to use more than 22TB - 30 TB, but I have seen multiple users to be able to query over 40TB per day on multiple days.

https://issuetracker.google.com/issues/257460447

ab...@gmail.com <ab...@gmail.com> #37Jan 26, 2024 03:23PM

Any updates on the progress, please?

ge...@hermes.com <ge...@hermes.com> #38Mar 11, 2024 08:27AM

A good example that we have, which makes this feature essential for us:
- All of our dashboards are linked to BQ, and we want our users to have a limit (in case something is not optimized properly)
- However the dataform service account will always use much, much more data to create the tables, then the people using the dashboards.
Would be a really great feature for us !
Thank you.

an...@real-digital.de <an...@real-digital.de> #39Apr 3, 2024 09:29AM

Hi Team,
Is there any update or timeline when this will be enabled ? We really need this feature in our workspace as well.

al...@gmail.com <al...@gmail.com> #40Jun 27, 2024 01:54PM

Really important feature, waiting for it!

eb...@google.com <eb...@google.com> #41Aug 6, 2024 06:31PM

Any updates on this feature?

vy...@google.com <vy...@google.com> Sep 19, 2024 06:04PM

Reassigned to lb...@google.com.

ol...@openvpn.com <ol...@openvpn.com> #42Sep 19, 2024 07:18PM

Any update ? We still need it badly.

eb...@google.com <eb...@google.com> #43Jan 13, 2025 11:42PM

Reassigned to ro...@google.com.

Any update on this FR? @rohitray

pm...@google.com <pm...@google.com> #44Jan 27, 2025 06:36PM

Hello @rohitray.

Dropping a friendly bump on this FR.

We have a customer working with a BigQuery quota that allows approximately 30TB of query usage per day per user, which applies uniformly to all users. However, they'd like more granular control over these quotas. Specifically, they want to assign custom query limits (e.g., Y for user X and B for user A).

The challenge arises because they have an automation user that updates refined tables or dashboards, and assigning a lower limit for all users could disrupt this automation. Ideally, they'd want the ability to exempt the automation user from limits while assigning specific quotas to individual users.

Any updates on the progress, please? Thank you.