Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to remove specific character from string in spark-sql

I want to convert '123,456' to 123456 (string to int) by spark-sql.

I tried like this.

REPLACE('123,456', ',', '')

But it didn't work, because spark-sql don't permit REPLACE()

like image 739
hyeon Avatar asked Sep 01 '25 10:09

hyeon


2 Answers

replace works as expected for me (Spark 2.3.0):

spark.sql("select int(replace('123,456', ',', '')) result").collect()
# [Row(result=123456)]
like image 81
MichaelChirico Avatar answered Sep 04 '25 03:09

MichaelChirico


assuming all data are numeric while stored in varchar convert function should solve your issue

conv(Column num,
          int fromBase,
          int toBase)
like image 35
RoMEoMusTDiE Avatar answered Sep 04 '25 03:09

RoMEoMusTDiE