Add cross-region support to bq cp [152689624]

Assigned

Feature Request

Status Update

No update yet.

Description

vl...@google.com

created issue #1

Mar 29, 2020 03:59AM

Using the Transfer Service for copying datasets as we recommend (

https://cloud.google.com/bigquery/docs/copying-datasets) is not very user-friendly. The params field is tricky, and if the user forgets to set a schedule end time, they will encounter egress charges as the copy is repeated every day. Plus, it requires the user to go through an OAuth2 workflow.

The semantics of a one-time copy and a repeating transfer are different. We should improve the user experience here and add cross-region support to bq cp. It can use transfer service under the hood, of course.

See:

https://stackoverflow.com/questions/60909980/how-can-i-copy-a-bigquery-dataset-from-one-region-to-another/60909981#60909981

Comments

bl...@google.com <bl...@google.com> Mar 29, 2020 04:15AM

Assigned to gi...@google.com.

gi...@google.com <gi...@google.com> Mar 30, 2020 05:21PM

Reassigned to sh...@google.com.

sh...@google.com <sh...@google.com> Aug 6, 2020 10:24PM

Reassigned to mi...@google.com.

bw...@google.com <bw...@google.com> Sep 21, 2021 10:00PM

Accepted by bw...@google.com.

st...@google.com <st...@google.com> #2Nov 15, 2021 04:44PM

em...@gtempaccount.com <em...@gtempaccount.com> #3Nov 18, 2021 04:44PM

Definetly agree to develop cp function at least to make it easier cross-region copy task and it can be very usefull to avoid lots of configratin issues too.

Message last modified on Nov 18, 2021 04:44PM

bw...@google.com <bw...@google.com> Nov 18, 2021 05:15PM

Assigned to dy...@google.com.

g....@luisaviaroma.com <g....@luisaviaroma.com> #4Nov 29, 2021 01:46PM

I would love to see this feature implemented. It would remove the need for extra steps in pipelines and help unify data from different parts of the business.

el...@sada.com <el...@sada.com> #5Jan 13, 2022 04:26PM

+1

Is there an ETA for this feature request?

Thank you

Elena.

bw...@google.com <bw...@google.com> #6Jan 13, 2022 05:15PM

This is not currently on the roadmap. It is still under consideration.

dy...@google.com <dy...@google.com> Mar 26, 2022 06:53PM

Accepted by dy...@google.com.

rd...@cnet.com <rd...@cnet.com> #7Jun 22, 2022 08:12PM

Any ETA on this feature request?

be...@soobeng.my <be...@soobeng.my> #8Apr 1, 2023 01:47AM

How is that roadmap looking?

dy...@google.com <dy...@google.com> #9May 9, 2023 04:49AM

I appreciate this Feature Request. A BQ user should be able to copy a table cross region w/out enabling, or even learning about the Transfer Service.

The following (for example) should just work:

If mydataset1 is in region1 and mydataset2 is in region2, a cross-region table copy would result.

CREATE OR REPLACE TABLE mydataset1.mytable1 
COPY mydataset2.mytable2

bq cp mydataset1.mytable1 mydataset2.mytable2

Current ETA is EOQ3 2023.

Cross region Network Traffic Egress charges will be incurred upon successful table copy completion.

The Egress charges will mirror that of GCS. See the "Network egress within Google Cloud" section here: https://cloud.google.com/storage/pricing#network-buckets

There will be no operations or retrieval costs to the cross-region copy job, but there will be Egress charges.

Message last modified on May 9, 2023 04:52AM

dy...@google.com <dy...@google.com> #10Sep 6, 2023 05:03PM

Assigned to vi...@google.com.

Hi All - while we are waiting for cross region copy Preview [ to be announced soon ] , the cross region dataset replication preview may serve some of the use cases above https://cloud.google.com/bigquery/docs/data-replication

ro...@hsbc.com <ro...@hsbc.com> #11Sep 6, 2023 05:28PM

Will this new feature support CMEK?

dy...@google.com <dy...@google.com> #12Sep 6, 2023 06:51PM

Yes. Cross region table copy Preview will support setting a destination CMEK that is different from the source CMEK.

bw...@google.com <bw...@google.com> #13Nov 17, 2023 05:59PM

Cross-region table copy is now in preview: https://cloud.google.com/bigquery/docs/managing-tables#copy_tables_across_regions.