IssueTracker

Thank you for opening this feature request. I've forwarded this request to our engineering team for evaluation. Any progress updates or a resolution will be posted on this thread.

Sincerely,

are there any updates on this?

Thanks!

Not an update, but a related link: BEAM support for sketch-based statistics

https://issues.apache.org/jira/browse/BEAM-2728

Update: Andrea and I are actively working on this.

It would be great if you open sourced the solution. We want to use this with Apache Spark also.

Ditto.

Ditto.

Is there any update on this item? It was asked over two years ago, and the only update has been "we're actively working on this." Is there an ETA when this could be ready?

Thanks for asking, David! We don't have an ETA that we can share yet, but I can come back in a month and share our progress made on this project.

Update: A BigQuery-compatible implementation of HyperLogLog++ has been open-sourced to https://github.com/google/zetasketch, and a design doc (https://docs.google.com/document/d/1U5aXdC9lDSOqT6FPHRulp-EutYiQ9KeHpgu-19CIfEI) about integrating it into Apache Beam has been sent out to dev@beam.apache.org.

Re #6: Thank you for your interest in ZetaSketch (https://github.com/google/zetasketch)! It would be great if the algorithm will be supported in Apache Spark. Also note that Apache Beam will support it and you can run Beam on Spark as well.

Oh wow, this is fantastic -- nice job Google team!!

Update: Zetasketch's HLL++ algorithm is supported in Beam now!
(HllCount: https://github.com/apache/beam/blob/master/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java)

I will follow up with the release manager to make sure this feature becomes available in Beam 2.16.

Hello,
My customer (Moonactive) could not use HLL between Dataflow and BigQuery before. Does it mean it should work now?
Thanks

Hi Oren! Yes! Once Beam 2.16.0 is out they will be able to use the Beam HllCount transforms to use HLL between Dataflow and BigQuery.