IssueTracker

Problem you have encountered:

• Setting internal artifactory channels in environment.yml does not work in private IP Dataproc cluster.
• The cluster create script still tries to connect to the external internet despite specifying nodefaults in the channels fails with error

<13>May 31 10:33:47 startup-script[753]: <13>May 31 10:33:47 activate-component-miniconda3[2458]: StructuredError{miniconda3, Component miniconda3 failed to activate}

What you expected to happen: The cluster create script should not try to connect to the internet however it should only use the internal artifactory.

Steps to reproduce:

Create a dataproc cluster and pass the dataproc:conda.env.config.uri flag. The environment.yml file used specifies an internal factory as channels that are to be used to install packages.

Workarounds:

1. Use an init script to create a custom environment and override the default channels inline. For example

#!/bin/bash
conda create --name <env_name> -c <channel> --override-channels

CONDA_BASE=$(conda info --base) 
source $CONDA_BASE/etc/profile.d/conda.sh

conda activate <env_name>

conda list # to ensure that there are no existing packages that could cause conflicts

conda install <package> -c <channel> --override-channels

2. Create a new cluster using the above init script.

3. To run the job, explicity set the Spark job properties to point to virtual environment's Python for example --properties="spark.pyspark.python=/opt/conda/miniconda3/envs/<env_name>/bin/python,spark.pyspark.driver.python=/opt/conda/miniconda3/envs/env_name/bin/python". Replace env_name with the name of the environment created in step 1.