Assigned
Status Update
No update yet.
Comments
va...@google.com
ja...@google.com
This feature request has been forwarded to the Data Fusion engineering team so that they may evaluate it. Note that there are no ETAs or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here.
Description
It is possible to tune the autoscaling behavior using worker_utilization_hint but, if the backlog time stays above 15 seconds, the pipeline is considered 'backlogged' and the number of nodes start to increase, even if the CPU usage is less than 30%.
Putting a cap on the max number of nodes is not a good solution, if the pipeline sometimes needs to process huge amount of data for many hours.
I'd like to have a "backlog-threshold-hint" parameter that controls how long the backlog time must be before the pipeline is considered 'backlogged'.
In other words, if the autoscaling algorithm considers both CPU and backlog, I'd like to tune how both metrics influence the number of nodes.