IssueTracker

This will create a public issue which anybody can view and comment on.

Problem you have encountered:

• A deprecated Spark property ("spark:spark.yarn.executor.failuresValidityInterval": "1h") was configured unexpectedly when Dataproc EFM [1] enabled
• The Spark property has been deprecated since Spark 3.5 [2] and the default value is None [3], instead of "1h"

What you expected to happen:

• Spark property spark.yarn.executor.failuresValidityInterval should be set only for images before Dataproc 2.2, which has Apache Spark 3.3.2 installed
• Spark property spark.executor.failuresValidityInterval should be set for the latest Dataproc 2.2+ images, which has Apache Spark 3.5.1 installed

[1] https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/enhanced-flexibility-mode

[2] https://spark.apache.org/docs/3.5.2/core-migration-guide.html#upgrading-from-core-34-to-35 (DISCLAIMER: This is an external link and Google is not responsible for the content provided in external documentation)

[3] https://spark.apache.org/docs/3.5.0/running-on-yarn.html#spark-properties (DISCLAIMER: This is an external link and Google is not responsible for the content provided in external documentation)