IssueTracker

Problem you have encountered: We have an hourly DAG that will create a Dataproc cluster, submit several pyspark jobs then delete the cluster afterward. But some hours the jobs all failed due to java.lang.ClassNotFoundException: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions. This error seems to happen at random time and interval so we are not sure of the cause either.

What you expected to happen: No java.lang.ClassNotFoundException for iceberg extension since the jar is already attached.

Steps to reproduce:

• Create dataproc cluster with this cluster properties:

"software_config": {
                "image_version": "2.1",
                "properties": {
                    "dataproc:pip.packages": "google-cloud-secret-manager==2.15.0",
                    "spark:spark.jars.packages": "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0",
                    "spark:spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
                    "spark:spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog",
                    "spark:spark.sql.catalog.spark_catalog.type": "hive",
                    "spark:spark.sql.catalog.amartha": "org.apache.iceberg.spark.SparkCatalog",
                    "spark:spark.sql.catalog.amartha.type": "hive",
                    "spark:spark.sql.session.timeZone": "Asia/Jakarta"
                }
            }

• submit a pyspark job that requires the iceberg extension
• delete the cluster after job completed
• it may need several iterations from creating cluster until deleting it to encounter this java.lang.ClassNotFoundException

Other information (workarounds you have tried, documentation consulted, etc):

• already tried adding iceberg jar package in job properties but after several iterations, we still encounter java.lang.ClassNotFoundException