IssueTracker

+1

This is not a problem with the Python SDK. It's a problem with the load job backend for Parquet files.

I can reproduce this issue with the BQ CLI (parquet file attached).

bq load --source_format=PARQUET swast-scratch:my_dataset.j
son_from_parquet json.parquet name:STRING,metadata:JSON

Output:

Upload complete.
Waiting on bqjob_r3cb1283b5b4dd24a_0000018508419771_1 ... (0s) Current status: R                                                                                Waiting on bqjob_r3cb1283b5b4dd24a_0000018508419771_1 ... (0s) Current status: DONE
BigQuery error in load operation: Error processing job 'swast-
scratch:bqjob_r3cb1283b5b4dd24a_0000018508419771_1': Unsupported field type:
JSON
Failure details:
- Unsupported field type: JSON

The Python SDK uses Parquet by default because both pandas DataFrame and Parquet are columnar formats.

Customers can workaround this issue by using CSV as a serialization format. (Feature added here: https://github.com/googleapis/python-bigquery/pull/399)

Are you saying to use SourceFormat.CSV [1], like in the scenario below ?

[1] job_config = bigquery.LoadJobConfig( schema=table_schema, source_format=SourceFormat.CSV )

My customer currently has this scenario:

1. create bigquery table with a json column.
2. insert into the table with cli

bq query "insert into ts-storage-dba-poc.vdang_demo_dataset.json_table6 values (JSON '{\"first\": \"vivian\", \"last\": \"dang\"}')"

3. Now if we run the following python api code, which read data into dataframe then load it backup, it failed with error message google.api_core.exceptions.BadRequest: 400 Unsupported field type: JSON

import pandas as pd
from google.cloud import bigquery

client = bigquery.Client(project='myproject')
job = client.query("select * from vdang_demo_dataset.json_table6")
df = job.to_dataframe()
print(client.load_table_from_dataframe(df, "vdang_demo_dataset.json_table6").result())

Yes, even omitting the schema should work as the Python SDK will fetch it if needed.

import pandas as pd
from google.cloud import bigquery

client = bigquery.Client(project='myproject')
job = client.query("select * from vdang_demo_dataset.json_table6")
df = job.to_dataframe()
print(client.load_table_from_dataframe(
    df, "vdang_demo_dataset.json_table6",
    job_config=bigquery.LoadJobConfig(source_format="CSV")).result())

Thanks for confirming!

Is there a documentation or update planned yet to mention the JSON interoperability ?