How to do LabelEncoding or categorical value in Apache Spark

Question

I am having dataset contains String columns . How can I encode the string based columns like the one we do in scikit-learn LabelEncoder

Answer 1

StringIndexer is what you need https://spark.apache.org/docs/1.5.1/ml-features.html#stringindexer

from pyspark.ml.feature import StringIndexer

df = sqlContext.createDataFrame(
            [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
            ["id", "category"]) 
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex") 
indexed = indexer.fit(df).transform(df) 
indexed.show()