Query scheduler (to pre-aggregate tables or intermediate results) [35905577]

Fixed

Feature Request

Status Update

No update yet.

Description

wo...@old.vente-exclusive.com

created issue #1

Mar 18, 2016 07:42PM

It would be great to have a simple scheduler in BigQuery to run certain queries on specific time schedules. This would allow to pre-calculate results and store them in a new table or append them to an existing table.

In our case it would allow us to extend pre-aggregated tables, avoiding the repeated calculation of aggregates. Or pre-joining tables.

I know all of this is possible using the API. But for business analysts it would be a great addition to the online interface of BigQuery.

Comments

ad...@google.com <ad...@google.com> #2Mar 18, 2016 08:25PM

Assigned to ad...@google.com.

Thanks for the suggestion. I've opened a feature request in our internal issue tracker.

You might also want to star the "cache view results" issue, since this sounds like a very similar problem:

https://code.google.com/p/google-bigquery/issues/detail?id=184

bv...@gmail.com <bv...@gmail.com> #3Jul 25, 2016 03:20PM

Hi, is there an option to see the scheduler feature in 2016-17?

Cache view results wouldn't work for us since there are streaming tables involved that are updated in real-time of several tens of Gb per day.

We would like to build our queries on compact daily-updated resulting tables.

[Deleted User] <[Deleted User]> #4Jun 1, 2017 01:25AM

This is one of the most needed features.

Companies seem to be set to resolve this with custom solutions at the moment. NY Times:

https://open.nytimes.com/faster-simpler-workflow-analytical-insights-ae6c7055e187

Ourselves we are using Pentaho Data INtegrator with BigQuery plugin and crontab with python code.

Would be great to be able to cater for simple logic like if/then. But that's probably a case for another ticket.

ad...@google.com <ad...@google.com> Feb 7, 2018 09:33PM

Reassigned to we...@google.com.

we...@google.com <we...@google.com> #5Jan 30, 2019 11:58PM

Marked as fixed.

Closing due to scheduled query system availability.T

Issue 35905577