IssueTracker

Problem you have encountered:

Toree Scala Jupyter Kernel keeps loading indefinitely when deploying a dataproc cluster with spark property spark:spark.submit.deployMode=cluster. When trying to start spark session in Jupyter, we dont get any response:

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
  .appName("MySparkApp")
  .master("yarn")
  .getOrCreate()

The error in yarn is:

24/02/07 10:36:11 ERROR ApplicationMaster: Uncaught exception: 
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259) ~[scala-library-2.12.18.jar:?]
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263) ~[scala-library-2.12.18.jar:?]
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:293) ~[spark-core_2.12-3.3.2.jar:3.3.2]
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:511) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:936) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:935) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:935) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.2.jar:3.3.2]
24/02/07 10:36:11 INFO Main$$anon$1: Shutting down kernel
24/02/07 10:36:11 INFO Main$$anon$1: Shutting down interpreters
24/02/07 10:36:11 INFO ScalaInterpreter: Shutting down interpreter
24/02/07 10:36:11 INFO Main$$anon$1: Shutting down actor system

What you expected to happen:

I expect the Toree Scala Kernel to work when starting a spark session.

Steps to reproduce:

Create a dataproc cluster with the following specs:

• dataproc components gateway enabled
• --properties=spark:spark.submit.deployMode=cluster
• Issue is seen in image 2.1-debian11 and 2.2-debian12 where toree scala is introduced

Other information (workarounds you have tried, documentation consulted, etc):

I am suspecting the kernel is getting installed without specifying the deploy mode and hence it only works when cluster is created with deploy mode client spark property.