Change theme
Help
Press space for more information.
Show links for this issue (Shortcut: i, l)
Copy issue ID
Previous Issue (Shortcut: k)
Next Issue (Shortcut: j)
Sign in to use full features.
Vote: I am impacted
Notification menu
Refresh (Shortcut: Shift+r)
Go home (Shortcut: u)
Pending code changes (auto-populated)
View issue level access limits(Press Alt + Right arrow for more information)
Request for new functionality
View staffing
Description
This will create a feature request which anybody can view and comment on.
Please describe your requested enhancement. Good feature requests will solve common problems or enable new use cases.
Dataflow job submitted by flex template[1] fails[2] for Google BigQuery partitioned tables which has partition filter requirements[3]
This feature request is a successor of a customer support ticket, where after consultation with the Dataflow engineering team, we have been confirmed that as of now the template [1] doesn't support partitioned tables with filter requirements.
Links:
[1]https://cloud.google.com/dataflow/docs/guides/templates/provided/bigquery-to-parquet
[2] io.grpc.StatusRuntimeException: INVALID_ARGUMENT: request failed: Query error: Cannot query over table '[project].[dataset].[partition_table]' without a filter over column(s) '[partition_column]' that can be used for partition elimination
[3]https://cloud.google.com/bigquery/docs/managing-partitioned-tables#require-filter
What you would like to accomplish:
BigQuery export to Parquet (via Storage API) template will work for partitioned tables with filter requirements.
How this might work: NA
If applicable, reasons why alternative solutions are not sufficient: NA
Other information (workarounds you have tried, documentation consulted, etc): NA