What you would like to accomplish:
When Exporting data from Bigquery, it would be helpful to have an option to split the export into multiple files, depending on the desired file size.

I know there is a compressed option, but when needing to upload the data into external destinations, there is often requirements for file type and size.

How this might work:
When given the option to export BQ data as a CSV, have a File Size Limit option where we can input a desired file size. The resulting export will be all of the data into multiple files where the File Size Limit is the max file size of those files.

Problem you have encountered:
Uploading BQ CSV export to an external destination and hitting a file size limit. We have to manually split up the file into multiple section to get it through.

I've found multiple stackoverflow postings on this topic with no easy answer:

https://stackoverflow.com/questions/28334241/exporting-data-to-gcs-from-bigquery-split-file-size-control
https://stackoverflow.com/questions/39972165/google-cloud-storage-split-files-by-value-in-the-file
https://stackoverflow.com/questions/45684531/how-do-i-split-a-large-csv-gz-file-in-google-cloud-storage
https://stackoverflow.com/questions/68006825/splitting-large-files-by-rows-in-gcp

Issue 254661555

Links (5)

Issue metadata