IssueTracker

We extract the pipeline graph and the metrics for all pipelines at Spotify in order to track various properties of our "fleet". One thing we used to be able to do and would like to do even with Runner V2 is to use these metrics to identify specific steps that have large amounts of shuffle so that we can potentially provide recommendations for optimization and cost control. At present on Runner V2 we can only identify that an entire pipeline has high shuffle/cost but cannot say, for example, "step X in pipeline Y joins a huge and a tiny dataset, use a side input"

Hi Kellen,

Sorry for the long silence. Please note that even with legacy runner ShuffleBytesRead and ShuffleBytesWritten counter are not perfectly tracking costs (they might not account for caching) and implemented only for java. Runner V2 use different model to report the counters and we prefer customers to rely on global billing counters which are guaranteed to be accurate.

We probably should be able to provide you shuffle read bytes coutner associated with the step, but its accuracy will also be limited (for instance no accounting for re-iterations). And there is no easy way to provide per step shuffle write counter.

Would such solution work for you? Alternatively you could add your own custom counters which can be presented in UI and associated wit the step, but if I understand correctly that was not an ideal option from your perspective.

Thanks,
Alex