fill na with random numbers in Pyspark

Question

I'm using Pyspark DataFrame.

I'd like to update NA values in Age column with a random value in the range 14 to 46.

How can I do it?

Answer 1

Mara's answer is correct if you would like to replace the null values with the same random number, but if you'd like a random value for each age, you should do something coalesce and F.rand() as illustrated below:

from pyspark.sql import functions as F
from pyspark.sql.types import IntegerType
from random import randint

df = sqlContext.createDataFrame(
    [(1, "a", 23.0), (3, "B", -23.0)], ("x1", "x2", "x3"))

df = (df
    .withColumn("x4", F.lit(None).cast(IntegerType()))
    .withColumn("x5", F.lit(None).cast(IntegerType()))
    )
    
df.na.fill({'x4':randint(0,100)}).show()
df.withColumn('x5', F.coalesce(F.col('x5'), (F.round(F.rand()*100)))).show()


+---+---+-----+---+----+
| x1| x2|   x3| x4|  x5|
+---+---+-----+---+----+
|  1|  a| 23.0|  9|null|
|  3|  B|-23.0|  9|null|
+---+---+-----+---+----+
+---+---+-----+----+----+
| x1| x2|   x3|  x4|  x5|
+---+---+-----+----+----+
|  1|  a| 23.0|null|44.0|
|  3|  B|-23.0|null| 2.0|
+---+---+-----+----+----+