Menu
I'm having a dataframe which contains a really big integer value, example:
42306810747081022358
When I've tried to convert it to long it was working in the Java but not under the spark envrironment, I was getting
NumberFormatException: For input string("42306810747081022358")
Then I tried to convert it too Decimal (BigDecimal) value. Again, easily can do it in Java, but in Spark: dframe.withColumn("c_number",col("c_a").cast(new DecimalType()));
This way I don't get any exceptions, however I can see that all result values are null.
I also tried to use UDF for this purpose but get the same results:
UDF1 cTransformer = new UDF1<String, BigDecimal>() {
@Override
public BigDecimal call(String aString) throws Exception {
return new BigDecimal(aString);
}
};
sqlContext.udf().register("cTransformer", cTransformer, new DecimalType());
dframe = dframe.withColumn("c_number", callUDF("cTransformer", dframe.col("c_a")));
And here again all I'm getting is a column with all zeroes.
How should I proceed?
483
Use the CAST() function to convert an integer to a DECIMAL data type. This function takes an expression or a column name as the argument, followed by the keyword AS and the new data type. In our example, we converted an integer (12) to a decimal value (12.00).
Converting a string to a decimal value or decimal equivalent can be done using the Decimal. TryParse() method. It converts the string representation of a number to its decimal equivalent.
To convert a varchar type to a numeric type, change the target type as numeric or BIGNUMERIC as shown in the example below: SELECT CAST('344' AS NUMERIC) AS NUMERIC; SELECT CAST('344' AS BIGNUMERIC) AS big_numeric; The queries above should return the specified value converted to numeric and big numeric.
In order to typecast an integer to decimal in pyspark we will be using cast() function with DecimalType() as argument, To typecast integer to float in pyspark we will be using cast() function with FloatType() as argument.
Try:
dframe.withColumn("c_number", dframe.col("c_a").cast("decimal(38,0)"))
answered Oct 13 '22 14:10
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With