Status Update
No update yet.
Comments
va...@google.com
nr...@google.com
Hello,
To troubleshoot the issue further, I have created a private ticket to provide some information about the issue (for which you should have received a notification). Please provide requested information there. Don't put any personal information, including project identifiers in this public ticket.
nr...@google.com
Hello,
Thank you for reaching out to us with your request.
We have duly noted your feedback and will thoroughly validate it. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issue, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.
We appreciate your continued trust and support in improving our Google Cloud Platform products. In case you want to report a new issue, Please do not hesitate to create a new issue on the
Once again, we sincerely appreciate your valuable feedback; Thank you for your understanding and collaboration.
Description
We have a Python Beam dataflow pipeline that writes events to pubsub lite topic
PS lite Topic has a Custom reservation with more than enough capacity units for publish and subscribe throughput. Topic also have 10 partitions.
Problem you have encountered:
Pipeline step reports following errors while publishing. My assumption is that its either publishing some records but then fails on some.. or publishing all but still reports this timeout error. Either ways it's an issue:
"Error message from worker: generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.util.concurrent.TimeoutException: Waited 1 minutes (plus 186472 nanoseconds delay) for ListFuture@47622653[status=PENDING, info=[futures=[com.google.api.core.ApiFutureToListenableFuture@9d2a6f6, com.google.api.core.ApiFutureToListenableFuture@3a21b389, com.google.api.core.ApiFutureToListenableFuture@38e5d00c, com.google.api.core.ApiFutureToListenableFuture@6ee339fd, com.google.api.core.ApiFutureToListenableFuture@721ac7c1,
com.google.api.core.ApiFutureToListenableFuture@7d3bd192, com.google.api.core.ApiFutureToListenableFuture@6b5ee761, com.google.api.core.ApiFutureToListenableFuture@3e30444b, com.google.api.core.ApiFutureToListenableFuture@51786ceb, com.google.api.core.ApiFutureToListenableFuture@72ec8017, com.google.api.core.ApiFutureToListenableFuture@6e1514c4]]]
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
org.apache.beam.sdk.io.gcp.pubsublite.internal.PubsubLiteSink$DoFnInvoker.invokeFinishBundle(Unknown Source)
org.apache.beam.fn.harness.FnApiDoFnRunner.finishBundle(FnApiDoFnRunner.java:1772)
org.apache.beam.fn.harness.data.PTransformFunctionRegistry.lambda$register$0(PTransformFunctionRegistry.java:116)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:560)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException: Waited 1 minutes (plus 186472 nanoseconds delay) for ListFuture@47622653[status=PENDING, info=[futures=[com.google.api.core.ApiFutureToListenableFuture@9d2a6f6, com.google.api.core.ApiFutureToListenableFuture@3a21b389, com.google.api.core.ApiFutureToListenableFuture@38e5d00c, com.google.api.core.ApiFutureToListenableFuture@6ee339fd, com.google.api.core.ApiFutureToListenableFuture@721ac7c1, com.google.api.core.ApiFutureToListenableFuture@45108ff0,
com.google.api.core.ApiFutureToListenableFuture@affe761, com.google.api.core.ApiFutureToListenableFuture@4c34b434, com.google.api.core.ApiFutureToListenableFuture@512265b2, com.google.api.core.ApiFutureToListenableFuture@121fd235]]]
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:531)
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:119)
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:75)
org.apache.beam.sdk.io.gcp.pubsublite.internal.PubsubLiteSink$RunState.waitForDone(PubsubLiteSink.java:65)
org.apache.beam.sdk.io.gcp.pubsublite.internal.PubsubLiteSink.finishBundle(PubsubLiteSink.java:83)
What you expected to happen:
Functionally - no failures in publishing
Also, Report No errors if publishing went fine or report some as warning
Steps to reproduce:
Create a beam dataflow piepeline (any version I used 2.51 and 2.47)
Publish lot of events to pubsub topic which has more than enough throughput reservation (20k/s to 100k/s burst for 5 mins)
Other information (workarounds you have tried, documentation consulted, etc):
Tried custom DoFn with python pubsub lite client.. that has similar issues as well. will report them seprately.
Seems like pubsub lite servers throughput is crappy.
Latency of each message publish is way to high then pubsub.