Assigned
Status Update
No update yet.
Default access
View
Expanded Access
--
--
--
Issue 364957696
Cloud Function triggered using a Eventarc trigger firing up several times for the same Big Query event on Insert or Update.
Feature Request
No update yet.
Description
Problem you have encountered: Multiple events for one Big Query operation.
Setup context: We have developed a cloud function that is triggered using an Eventarc trigger. The Eventarc trigger is based on Bigquery table. The event method is "google.cloud.bigquery.v2.JobService.InsertJob".
Desired behavior: We need a solution in which the cloud function itself is called just once per insert instead of we adding any filter logic within the cloud function source code.
Observed behavior: The cloud function is invoked multiple times in case of even one record insert or update operation.
Other information (workarounds you have tried, documentation consulted, etc):
As per [1], the expected behavior is that there will be several audit logs for a given BigQuery action so you need to make sure to look for a unique set of attributes that clearly identifies your action. This has also been raised in a few Stack Overflow inquiries such as [2].
In the case of inserting rows, this is the combination:
The method is google.cloud.bigquery.v2.JobService.InsertJob
The name of the table being inserted to is the protoPayload.resourceName
The dataset id is available as resource.labels.dataset_id
The number of inserted rows is protoPayload.metadata.tableDataChanged.insertedRowsCount
The current functionality of Big Query is that at user-level some operations like insert, delete or update look like atomic operations but internally big query performs more operations and writes more than 1 log per each insert or update query. Eventarc will trigger the function no matter what operation is performed on the specified table (these can be read, write or any other kind of operation), because it only listens for logs generated by bigquery related to the selected table and api method (google.cloud.bigquery.v2.JobService.InsertJob). Eventarc is currently not capable of performing more specific filters.
References:
[1]
[2]