How can I change SparkContext.sparkUser() setting (in pyspark)?

Question

I am new with Spark and pyspark.
I use pyspark, after my rdd processing, I tried to save it to hdfs using the saveAsTextfile() function. But I get a 'permission denied' error message because pyspark tries to write hdfs using my local account, 'kjlee', which does not exist on the hdfs system.

I can check the spark user name by SparkContext().sparkUser(), But I can't find how to change the spark user name.

How can I change the spark user name?

Answer 1

In Scala could be done with System.setProperty:

  System.setProperty("HADOOP_USER_NAME","newUserName")

  val spark = SparkSession
    .builder()
    .appName("SparkSessionApp")
    .master("local[*]")
    .getOrCreate()

  println(spark.sparkContext.sparkUser)