How to add a column to the beginning of the schema?

Question

Dataset.withColumn() seems to append the column to the end of the schema. How to prepend the column to the beginning of the schema?

Answer 1

Use select with wildcard:

df.select(new_column, col("*"))

Answer 2

Above answer is right but one catch is duplicate column name is coming at the first and last... so tried with another variation, using selectExpr (Selects a set of SQL expressions is)

    val someData = Seq(
      Row(8, "Ram"),
      Row(64, "erwaman"),
      Row(-27, "user8371915")
    )

    val someSchema = List(
      StructField("number", IntegerType, true),
      StructField("word", StringType, true)
    )

    val someDF = spark.createDataFrame(
      spark.sparkContext.parallelize(someData),
      StructType(someSchema)
    )
    println("initial")
    someDF.show(false)


    println("mycolumn added at the end")
    val dataframecolumnOrderChanged = someDF.withColumn("mycolumn", functions.lit("mycolumnadded"))
    dataframecolumnOrderChanged.show(false)
    println("mycolumn added at the beginning")
    moveColumnToFirstPos(dataframecolumnOrderChanged, "mycolumn")


 /**
    * moveColumnToFirstPos : This function can be used after withColumn applied to a data frame
    * which will add column to the right most position
    * if you want to move column to first position in the dataframe selection and drop duplicate as well
    *
    * @param dataframecolumnOrderChanged
    */
   def moveColumnToFirstPos(dataframecolumnOrderChanged: DataFrame, colname: String) = {
    val xs = Array(colname) ++ dataframecolumnOrderChanged.columns.dropRight(1)
    val fistposColDF = dataframecolumnOrderChanged.selectExpr(xs: _*)
    fistposColDF.show(false)
    fistposColDF
  }

---

Result :

initial
+------+-----------+
|number|word       |
+------+-----------+
|8     |Ram        |
|64    |erwaman    |
|-27   |user8371915|
+------+-----------+

mycolumn added at the end
+------+-----------+-------------+
|number|word       |mycolumn     |
+------+-----------+-------------+
|8     |Ram        |mycolumnadded|
|64    |erwaman    |mycolumnadded|
|-27   |user8371915|mycolumnadded|
+------+-----------+-------------+

mycolumn added at the beginning
+-------------+------+-----------+
|mycolumn     |number|word       |
+-------------+------+-----------+
|mycolumnadded|8     |Ram        |
|mycolumnadded|64    |erwaman    |
|mycolumnadded|-27   |user8371915|
+-------------+------+-----------+