Status Update
No update yet.
Comments
va...@google.com
nr...@google.com
Hello,
To troubleshoot the issue further, I have created a private ticket to provide some information about the issue (for which you should have received a notification). Please provide requested information there. Don't put any personal information, including project identifiers in this public ticket.
nr...@google.com
Hello,
Thank you for reaching out to us with your request.
We have duly noted your feedback and will thoroughly validate it. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the issue, please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.
We appreciate your continued trust and support in improving our Google Cloud Platform products. In case you want to report a new issue, Please do not hesitate to create a new issue on the
Once again, we sincerely appreciate your valuable feedback; Thank you for your understanding and collaboration.
Description
I was able to find a related (now closed) SPARK issue:
Setting the property `spark.sql.optimizer.canChangeCachedPlanOutputPartitioning` to `false` (the default before Spark 3.5.0) did fix the issue.
I was able to confirm locally that bumping to Spark 3.5.1 did fix the problem too.
I could not find any information about this around Dataproc, sorry if I missed it, but I though it was worth raising awareness on this.
I was also wondering when a dataproc image based on Spark 3.5.1 was planned? Could not find the information.