Is there no "inverse_transform" method for a scaler like MinMaxScaler in spark?

Question

When train a model, say linear regression, we may make a normalization, like MinMaxScaler, on the train an test dataset.

After we got a trained model and use it to make predictions, and scale back the predictions to the original representation.

In python, there is "inverse_transform" method. For example:

from sklearn.preprocessing import MinMaxScaler
scalerModel.inverse_transform

from sklearn.preprocessing import MinMaxScaler

data = [[-1, 2], [-0.5, 6], [0, 10], [1, 18]]

scaler = MinMaxScaler()
MinMaxScaler(copy=True, feature_range=(0, 1))
print(data)

dataScaled = scaler.fit(data).transform(data)
print(dataScaled)

scaler.inverse_transform(dataScaled)

Is there similar method in spark?

I have googled a lot, but found no answer. Can anyone give me some suggestions? Thank you very much!

Answer 1

In our company, in order to solve the same problem on the StandardScaler, we extended spark.ml with this (among other things):

package org.apache.spark.ml

import org.apache.spark.ml.linalg.DenseVector
import org.apache.spark.ml.util.Identifiable

package object feature {

    implicit class RichStandardScalerModel(model: StandardScalerModel) {

        private def invertedStdDev(sigma: Double): Double = 1 / sigma

        private def invertedMean(mu: Double, sigma: Double): Double = -mu / sigma

        def inverse(newOutputCol: String): StandardScalerModel = {
            val sigma: linalg.Vector = model.std
            val mu: linalg.Vector = model.mean
            val newSigma: linalg.Vector = new DenseVector(sigma.toArray.map(invertedStdDev))
            val newMu: linalg.Vector = new DenseVector(mu.toArray.zip(sigma.toArray).map { case (m, s) => invertedMean(m, s) })
            val inverted: StandardScalerModel = new StandardScalerModel(Identifiable.randomUID("stdScal"), newSigma, newMu)
                .setInputCol(model.getOutputCol)
                .setOutputCol(newOutputCol)

            inverted
                .set(inverted.withMean, model.getWithMean)
                .set(inverted.withStd, model.getWithStd)
        }
    }

}

It should be fairly easy to modify it or do something similar for your specific case.

Keep in mind that due to JVM's double implementation, you normally lose precision in these operations, so you will not recover the exact original values you had before the transformation (e.g.: you will probably get something like 1.9999999999999998 instead of 2.0).