GKE Usage Metering starts failing ~12 billion rows. [391590632]

Assigned

Customer Issue

Status Update

No update yet.

Description

bo...@google.com

created issue #1

Jan 22, 2025 03:14PM

BigQuery only supports operations of 6 hours long. The SQL template from GKE Usage metering [1] w/ consumption takes longer than this. It appears to be due to a join explosion that starts to exceed this length after ~12 billion rows.

We need to optimize the template to avoid this limitation.

Additionally, it should be possible to iteratively update the GKE Usage metering consumption table instead of regenerating the whole thing. An iterative update would avoid increasing query times.

[1]:

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-usage-metering

Comments

ba...@google.com <ba...@google.com> Jan 23, 2025 04:09AM

Reassigned to on...@google.com.

on...@google.com <on...@google.com> Jan 23, 2025 11:52AM

Status: New

ka...@google.com <ka...@google.com> #2Feb 17, 2025 10:53AM

Assigned to gc...@google.com.

I had the same problem!
I could solve it by putting the .aidl file in an aidl directory.
Look at the attached screenshot to see the project structure.

Issue 391590632

Description

Issue summary

Comments

ba...@google.com <ba...@google.com> Jan 23, 2025 04:09AM

on...@google.com <on...@google.com> Jan 23, 2025 11:52AM

ka...@google.com <ka...@google.com> #2Feb 17, 2025 10:53AM

Add comment

Issue metadata