Why does scipy.optimize.least_squares exist when scipy.optimize.minimize could potentially be used for the same things?

Question

I am trying to understand why the scipy.optimize.least_squares exists in scipy. This function can be used to perform model-fitting. However, one could use scipy.optimize.minimize to do the same thing. The only difference is that scipy.optimize.least_squares does the calculation of the chi-squared internally, while if one wants to use scipy.optimize.minimize, he/she will have to calculate the chi-squared manually inside the function the user want to minimize. Also, scipy.optimize.least_squares can not be considered a wrapper around scipy.optimize.minimize because the three methods it supports (trf, dogbox, lm), are not supported at all by scipy.optimize.minimize.

So my questions are:

• Why scipy.optimize.least_squares exists when the same result can be achieved with scipy.optimize.minimize?
• Why scipy.optimize.minimize does not support the trf, dogbox, and lm methods?

Thank you.

Answer 1

The algorithms in scipy.optimize.least_squares utilize the least-squares structure of the minimization problem for better convergence (or lower order of the used derivatives).

It's similar to the difference between the Gauss-Newton algorithm and Newton's method, see Wikipedia or this question.

In particular, Gauss-Newton only uses the Jacobian (first derivatives), whereas Newton's method also uses the Hessian (second derivatives), which is expensive to calculate.