SVM - Difference between Energy vs Loss vs Regularization vs Cost function

Question

I am reading A Tutorial on Energy Based Learning and I am trying to understand the difference between all those terms stated above in the context of SVMs. This link summarizes the differences between a loss, cost and an objective function. Based on my understanding,

Objective function: Something we want to minimize. For example ||w||^2 for SVM.

Loss function: Penalty between prediction and label which is also equivalent to the regularization term. Example is the hinge loss function in SVM.

Cost function: A general formulation that combines the objective and loss function.

Now, the 1st link states that the hinge function is max(0, m + E(W,Yi,Xi) - E(W,Y,X)) i.e. it is a function of the energy term. Does that mean that the energy function of the SVM is 1 - y(wx + b) ? Are energy functions are a part of a loss function. And a loss + objective function a part of the cost function ?

A concise summary of the 4 terms would immensely help my understanding. Also, do correct me if my understanding is wrong. The terms sound so confusing. Thanks !

Answer 1

Objective function: Something we want to minimize. For example ||w||^2 for SVM.

Objective function is - as name suggests - objective of optimization. It can be either something we want to minimize (like cost function) or maximize (like likelihood). In general - function that measures how good is our current solution (usually by returning a real number)

Loss function: Penalty between prediction and label which is also equivalent to the regularization term. Example is the hinge loss function in SVM.

First of all, loss is not equivalent to regularization, in any sense. Loss function is a a penalty between a model and truth. This can be a prediction of class conditional distribuition vs true label, thus can also be a data distribution vs. empirical sample, and many more.

Regularization

Regularization is a term, penalty, measure which is supposed to be a penalty for too complex model. In ML, or generally in statistics when dealing with estimators, you always try to balance two sources of error - variance (coming from too complex models, overfitting) and bias (coming from too simple models, bad learning methods, underfitting). Regularization is a technique of penalizing high-variance models in the optimization process in order to get less overfitted one. In other words - for techniques which can fit training set perfectly, it is important to have a measure which forbids it in order to preserve ability to generalize.

Cost function: A general formulation that combines the objective and loss function.

Cost function is just an objective function which one minimizes. It can be composed of some agglomeration of loss functions and regularizers.

Now, the 1st link states that the hinge function is max(0, m + E(W,Yi,Xi) - E(W,Y,X)) i.e. it is a function of the energy term. Does that mean that the energy function of the SVM is 1 - y(wx + b) ? Are energy functions are a part of a loss function. And a loss + objective function a part of the cost function ?

The hinge loss is max(0, 1 - y(<w,x> - b)). The one defined here is not really for SVM but for general factor graphs, I would strongly suggest to start learning ML from basics and not from advanced techniques. Without good understanding of basics of ML, this paper will not be possible to understand.

To show example with SVM and naming convention

C SUM_i=1^N max(0, 1 - y_i(<w, x_i> - b)) + ||w||^2

            \__________________________/    \_____/
                         loss            regularization
\_________________________________________________/
            cost / objective function