Choosing right metrics for regression model

Question

I have always been using r2 score metrics. I know there are several evaluation metrics out there i have read several articles about it. Since i'm still a beginner in machine learning. I'm still very confused of

1. When to use each of it, is depending on our case, if yes please give me example
2. I read this article and it said, r2 score is not straightforward, we need other stuff to measure the performance of our model. Does it mean we need more than 1 evaluation metrics in order to get better insight of our model performance?
3. Is it recommended if we only measure our model performance by just one evaluation metrics?
4. From this article it said knowing the distribution of our data and our business goal helps us to understand choose appropriate metrics. What does it mean by that?
5. How to know for each metrics that the model is 'good' enough?

Answer 1

There are different evaluation metrics for regression problems like below.

1. Mean Squared Error(MSE)
2. Root-Mean-Squared-Error(RMSE)
3. Mean-Absolute-Error(MAE)
4. R² or Coefficient of Determination
5. Mean Square Percentage Error (MSPE)
6. so on so forth..

As you mentioned you need to use them based on your problem type, what you want to measure and the distribution of your data.

1. To do this, you need to understand how these metrics evaluate the model. You can check the definitions and pros/cons of evaluation metrics from this nice blog post.
2. R² shows what variation of your purpose variable is described by independent variables. A good model can give R² score close to 1.0 but it does not mean it should be. Models which have low R² can also give low MSE score. So to ensure your predictive power of your model it is better to use MSE, RMSE or other metrics besides the R².
3. No. You can use multiple evaluation metrics. The important thing is if you compare two models, you need to use same test dataset and the same evaluation metrics.
4. For example, if you want to penalize your bad predictions too much, you can use MSE evaluation metric because it basically measures the average squared error of our predictions or if your data have too much outlier MSE give too much penalty to this examples.
5. The good model definition changes based on your problem complexity. For example if you train a model which predicts that heads or tails and gives %49 accuracy it is not good enough because the baseline of this problem is %50. But for any other problem, %49 accuracy may enough for your problem. So in a summary, it depends on your problem and you need to define or think that human(baseline) threshold.