Assigned
Status Update
Comments
va...@google.com <va...@google.com>
je...@google.com <je...@google.com> #2
This feature request has been forwarded to the Data Fusion engineering team so that they may evaluate it. Note that there are no ETAs or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here.
ga...@gmail.com <ga...@gmail.com> #3
Struct support is being prioritized and can be tracked in open source by https://issues.cask.co/browse/CDAP-15349 .
Description
Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.
Problem you have encountered: When Reading data from big query tables , temp dataset should be created in same location where source and target dataset is. But in case if source is view dataflow is trying to create temp dataset in US location not at the same location where underline table is .
What you expected to happen: If underline table for a view is in europe-west 2 location temp dataset should also e created in europe west 2 location . But Dataflow is trying to create that in US location . Org policy is restricting that resulting in dataflow failure.
Steps to reproduce: 1.create a org policy which restricts that no table should be created out of europe west 2.
2. create a table in europe west 2 and create an auth view / view on top of that.
3. Read that data in dataflow using ReadfromBigQuery python library . It will create temp dataset in US location not in europ west 2 .
Other information (workarounds you have tried, documentation consulted, etc): created a temp dataset explicitly in europe west 2 and used it but by deafult it should be able to handle it
beam.io.ReadFromBigQuery(query=query, use_standard_sql=True, temp_dataset=bigquery.DatasetReference( projectId="PROJECT_ID", datasetId="DATASET")