Big Query Storage API Logs [159125960]

In Progress

Feature Request

Status Update

No update yet.

Description

sa...@google.com

created issue #1

Jun 16, 2020 04:15PM

Problem you have encountered:

We are unable to find a way to distinguish calls to BigQuery via Storage API and REST API in the logs. This distinction is required as the pricing model is very different.

we are aware we can see the usage of the BigQuery Storage API in the APIs & Services menu but having the logs in Satckdriver it much suitable

What you expected to happen:

being able to See log of the call made to the BigQuery Storage API same way we can see the log of the call made to to the REST API

Steps to reproduce:

1) Create a BigQuery table
2) Load data to the table
3) Use the quickstart[1] to query the table
4) Check the Stackdriver logs

[1]: ttps://

cloud.google.com/bigquery/docs/reference/storage/libraries#client-libraries-install-python

Other information (workarounds you have tried, documentation consulted, etc):

Comments

bl...@google.com <bl...@google.com> Jun 16, 2020 04:16PM

Assigned to la...@google.com.

sa...@google.com <sa...@google.com> #2Jun 16, 2020 04:21PM

Reassigned to gc...@google.com.

st...@google.com <st...@google.com> #3Sep 30, 2021 02:58PM

er...@condenast.com <er...@condenast.com> #4Nov 25, 2021 11:16AM

Is there still no update or ETA or progress you could relate on this?

ra...@google.com <ra...@google.com> #5Nov 25, 2021 10:10PM

Hello,

The product team is still analyzing this, please consider that once they have new information they will post it here. However, there is no ETA for the release.

sa...@google.com <sa...@google.com> #6Jul 8, 2022 04:48PM

Hi team do we have an update or timelines on this feature request?

st...@google.com <st...@google.com> #7Sep 12, 2022 12:49AM

Hi team,

do we have any update? customer would like to observe the payload size.
thanks

mi...@strive.ai <mi...@strive.ai> #8Feb 8, 2023 02:59PM

+1 for this. I see random AppendRows failures but can't figure out what the root cause is given it's behind a BQ Subscription with pub/sub. Would like to see logs in addition to metrics if possible? Any advice on where to see logs for BQ Subscriptions?

ds...@google.com <ds...@google.com> Jul 6, 2023 10:02AM

Reassigned to gc...@google.com.

mb...@uma-health.com <mb...@uma-health.com> #9Jul 10, 2023 06:52PM

Hello Team,

I am also looking forward to a solution to this feature request. Any news?

bw...@google.com <bw...@google.com> Nov 18, 2023 12:52AM

Reassigned to ga...@google.com.

ga...@google.com <ga...@google.com> #10Nov 18, 2023 12:55AM

This ask is currently in our backlog, and we will consider this for 2024 plans.

an...@iheartmedia.com <an...@iheartmedia.com> #11Dec 21, 2023 10:33PM

mega bump on this, really hard to debug issues without it

ga...@google.com <ga...@google.com> Mar 21, 2024 04:10PM

Reassigned to ni...@google.com.

ni...@google.com <ni...@google.com> Mar 21, 2024 06:26PM

Accepted by ni...@google.com.

em...@google.com <em...@google.com> #12Jan 10, 2025 10:33AM

Hello team

Any update on this please?

Thank you,
Emna

ni...@google.com <ni...@google.com> #13Jan 15, 2025 06:43AM

Comment has been deleted.

Message last modified on Jan 15, 2025 06:45AM

ni...@google.com <ni...@google.com> #14Jan 15, 2025 06:47AM

A Storage Read API specific BigQuery Information_Schema table is under development which should address these needs around spend reporting and error troubleshooting.

I will share an ETA when possible.