Log Loss function in pyspark

Question

Is there a built-in log loss function in pyspark?

I have a pyspark dataframe with columns: probability, rawPrediction, label

and I want to use mean log loss to evaluate these predictions.

Answer 1

No such function exists directly, as far as I can tell. But given a PySpark dataframe df with the columns named as in the question, one can explicitly calculate the average log loss:

import pyspark.sql.functions as f

df = (
    df.withColumn(
        'logloss'
        , -f.col('label')*f.log(f.col('probability')) - (1.-f.col('label'))*f.log(1.-f.col('probability'))
    )
)

logloss = df.agg(f.mean('logloss').alias('ll')).collect()[0]['ll']

I'm assuming here that label is numerical (i.e. 0 or 1), and that probability represents the predictions of the model. (Not sure what rawPrediction might mean.)