Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Do you apply min max scaling separately on training and test data?

While applying min max scaling to normalize your features, do you apply min max scaling on the entire dataset before splitting it into training, validation and test data?

Or do you split first and then apply min max on each set, using the min and max values from that specific set?

Lastly , when making a prediction on a new input, should the features of that input be normalized using the min, max values from the training data before being fed into the network?

like image 300
shekit Avatar asked Apr 09 '17 03:04

shekit


People also ask

When should I use MinMaxScaler?

Use MinMaxScaler if you want to have a light touch. It's non-distorting. You could use RobustScaler if you have outliers and want to reduce their influence. Use Normalizer sparingly — it normalizes sample rows, not feature columns.

Do we scale the test set?

The test set must use identical scaling to the training set. And the point is given that: Do not scale the training and test sets using different scalars: this could lead to random skew in the data.

Should I scale the test data in machine learning?

Scaling data is essential before applying a lot of Machine Learning techniques. For example, distance-based methods such as K-Nearest Neighbors, Principal Component Analysis or Support-Vector Machines will artificially attribute a great importance to a given feature if its range is extremely broad.


1 Answers

Split it, then scale. Imagine it this way: you have no idea what real-world data looks like, so you couldn't scale the training data to it. Your test data is the surrogate for real-world data, so you should treat it the same way.

To reiterate: Split, scale your training data, then use the scaling from your training data on the testing data.

like image 72
Arya McCarthy Avatar answered Oct 13 '22 13:10

Arya McCarthy