How to update struct field spark/scala

Question

I have a struct as part of my json.

store: struct (c1, c2, c3, c4)

I would like to update c2 in place so no new field is created. After update it should be same struct with new value for c2.

In spark/scala, I have already tried:

df.withColumn("store.c2", newVal)

But this creates a new field store.c2, Columns not part of struct, I am able to update.

df.withColumn("columnTen", newValue)

does not create new field and updated to newValue.

Answer 1

Since Spark 3.1+, you can use withField on a struct column:

An expression that adds/replaces field in StructType by name.

df.withColumn("store", $"store".withField("c2", lit(newVal)))