Status Update
No update yet.
Comments
sa...@doit.com
sa...@doit.com
+1
gi...@google.com
I have performed some tests, and the option_name = 'storage_billing_model' is apparently only populated for datasets which are updated, as per sample command:
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" \
-L -X PUT https://bigquery.googleapis.com/bigquery/v2/projects/<project_name>/datasets/<dataset_name> -d '{"datasetReference": {"projectId": "project_name", "datasetId": "dataset_name"}, "storageBillingModel": "LOGICAL"}'
You can observe the output (bq_output.png) for a sample query on my project that does work, but only for datasets I updated the billing storage model:
-
As per documentation [1], when you create a dataset, the storage used by that dataset is billed to you using logical bytes as the default unit of consumption. I believe since you are not seeing this on the output for your queries, those datasets were not updated and are currently using the default option.
-
My recommendation as of now is for you to review and update datasets with the intended storage_billing_model as the info will be proper populated and showed to your queries on
INFORMATION_SCHEMA.SCHEMATA_OPTIONS.
References:
[1]
sa...@doit.com
However, if I go back and look at datasets that were changed from logical to physical on 8/17/2023 the row doesn't exist inside of SCHEMATA_OPTIONS for it which would imply it's on logical storage. The issue is that I checked the billing data with a label/tag on the dataset and see an active physical storage SKU with a cost associated with it. There is not an active logical storage SKU on it with a cost since the day we switched it over to physical storage (8/17/2023). I do not have any other datasets still available that were changed from logical to physical billing prior to that to confirm more details, unfortunately.
This confirms it is still a bug for older datasets that were created with or moved to physical storage from sometime before 8/21/2023 when I filed this ticket.
Description
Ran a test across over 50 internal and customer projects, most of which have a mix of logical and physical storage datasets, and not a single one of them had this value populated. Creating a new project and dataset and then setting the value doesn't seem to populate it either. Created one, set it to physical storage, and waited over 72 hours and it still hasn't populated either.
This feature needs to be implemented to match the documentation.
[1]