PySpark Dataframe : comma to dot

Question

I have imported data using comma in float numbers and I am wondering how can I 'convert' comma into dot. I am using pyspark dataframe so I tried this :

commaToDot = udf(lambda x : str(x).replace(',', '.'), FloatType())

myData.withColumn('area',commaToDot(myData.area))

And it definitely does not work. So can we replace directly it in dataframe from spark or should we switch in numpy type or something else ?

Thanks !

Answer 1

Another way to do it (without using UDFs) is:

myData = myData.withColumn('area', regexp_replace('area', ',', '.').cast('float'))

Answer 2

I think you are missing

from pyspark.sql.types import FloatType

As Pushkr suggested udf with replace will give you back string column if you don't convert result to float

from pyspark import SQLContext
from pyspark.sql.functions import udf
from pyspark.sql.types import FloatType
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName("ReadCSV")
sc = SparkContext(conf=conf)
sqlctx = SQLContext(sc)
df = sqlctx.read.option("delimiter", ";").load("test.csv", format="csv")
df.show()
commaToDot = udf(lambda x : float(str(x).replace(',', '.')), FloatType())
df2=df.withColumn('area',commaToDot(df._c0))
df2.printSchema()
df2.show()

I used single column file , tested on spark 2.11/python 3.6