How to Run Bash Script As Initialization Action Upon Creating Dataproc Cluster?

Question

I would like the Dataproc cluster to download a custom library I created that's not pip installable, so it would require a user to clone it from cloud source repo and then do sudo python setup.py install. I tried creating a bash script; the cluster was created without any issue but I don't think it ran the bash script because I didn't notice any changes.

Here's my bash script that I want to initialize to the cluster:

#! /bin/bash

# download jars
gsutil -m cp gs://dataproc-featurelib/spark-lib/*.jar .

# download credential files
gsutil -m cp gs://mlflow_feature_pipeline/secrets/*.json .

# install feature_library
gcloud source repos clone feature_library --project=<project_id>
cd feature_library
sudo python3 setup.py install
cd ../

Here is how I setup my cluster:

gcloud beta dataproc clusters create featurelib-cluster \
    --zone=us-east1-b \
    --master-machine-type n1-highmem-16 \
    --worker-machine-type n1-highmem-16 \
    --num-workers 4 \
    --image-version 1.4-debian9 \
    --initialization-actions gs://dataproc-initialization-actions/python/pip-install.sh,gs://dataproc-featurelib/initialization-scripts/dataproc_featurelib_init.sh \
    --metadata 'PIP_PACKAGES=google-cloud-storage hvac cryptography mlflow sqlalchemy snowflake-sqlalchemy snowflake-connector-python snowflake' \
    --optional-components=ANACONDA \
    --enable-component-gateway \
    --project <project_id> \
    --autoscaling-policy=featurelib-policy \
    --tags feature-lib \
    --no-address \
    --subnet composer-us-east1 \
    --bucket dataproc-featurelib

Answer 1

I resolved this issue by authorizing the service account. Example of Bash Script below:

#! /bin/bash

# download jars
gsutil -m cp gs://dataproc-featurelib/spark-lib/*.jar .

# download credential files
gsutil -m cp gs://mlflow_feature_pipeline/secrets/*.json .

# authenticate
gcloud config set account <gserviceaccount_email_id>
gcloud auth activate-service-account <gserviceaccount_email_id> --project=dao-aa-poc-uyim --key-file=<path_to_key_file>

# install package
gcloud source repos clone feature_library --project=<project_id>
cd feature_library
python3 setup.py install
cd ../