This will create a public issue which anybody can view and comment on.
Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.
Problem you have encountered:
BigQuery table creation from a csv file fails where the data type after the first 500 rows doesn't match the data type in the first 500 rows (specified as a limitation in https://cloud.google.com/bigquery/docs/schema-detect)
"400 Error while reading data, error message: Could not parse <example_string> as INT64 for field <field_name> (position 1) starting at location xxxxxxx with message 'Unable to parse' File: <csv_name>"
What you expected to happen:
Column detected as string instead of integer and the table creates successfully
Steps to reproduce:
Try to load data into BigQuery using schema auto detection, with all values for a column being integers for the first 500 rows, with a value for the column being a string in the 501st row
Other information (workarounds you have tried, documentation consulted, etc):
Workarounds tried;
Creating the schema manually for each csv - In our context this is not feasible as we have 1000's of different csv files each with a different schema and are looking to load them into BigQuery automatically.
Possible solutions could be;
1) Allowing the user to specify the number of rows to use for schema auto detection 2) Allowing the user to instruct schema auto detection to define all columns as strings
Description
Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.
Problem you have encountered:
BigQuery table creation from a csv file fails where the data type after the first 500 rows doesn't match the data type in the first 500 rows (specified as a limitation in
"400 Error while reading data, error message: Could not parse <example_string> as INT64 for field <field_name> (position 1) starting at location xxxxxxx with message 'Unable to parse' File: <csv_name>"
What you expected to happen:
Column detected as string instead of integer and the table creates successfully
Steps to reproduce:
Try to load data into BigQuery using schema auto detection, with all values for a column being integers for the first 500 rows, with a value for the column being a string in the 501st row
Other information (workarounds you have tried, documentation consulted, etc):
Workarounds tried;
Creating the schema manually for each csv -
In our context this is not feasible as we have 1000's of different csv files each with a different schema and are looking to load them into BigQuery automatically.
Possible solutions could be;
1) Allowing the user to specify the number of rows to use for schema auto detection
2) Allowing the user to instruct schema auto detection to define all columns as strings
Similar Issues;