What is inductive bias in machine learning? [closed]

Question

What is inductive bias in machine learning? Why is it necessary?

Answer 1

Every machine learning algorithm with any ability to generalize beyond the training data that it sees has some type of inductive bias, which are the assumptions made by the model to learn the target function and to generalize beyond training data.

For example, in linear regression, the model assumes that the output or dependent variable is related to independent variable linearly (in the weights). This is an inductive bias of the model.

Answer 2

What is inductive bias?

Pretty much every design choice in machine learning signifies some sort of inductive bias. "Relational inductive biases, deep learning, and graph networks" (Battaglia et. al, 2018) is an amazing 🙌 read, which I will be referring to throughout this answer.

An inductive bias allows a learning algorithm to prioritize one solution (or interpretation) over another, independent of the observed data. [...] Inductive biases can express assumptions about either the data-generating process or the space of solutions.

Examples in deep learning

Concretely speaking, the very composition of layers 🍰 in deep learning provides a type of relational inductive bias: hierarchical processing. The type of layer imposes further relational inductive biases:

More generally, non-relational inductive biases used in deep learning include:

• activation non-linearities,
• weight decay,
• dropout,
• batch and layer normalization,
• data augmentation,
• training curricula,
• optimization algorithms,
• anything that imposes constraints on the learning trajectory.

Examples outside of deep learning

In a Bayesian model, inductive biases are typically expressed through the choice and parameterization of the prior distribution. Adding a Tikhonov regularization penalty to your loss function implies assuming that simpler hypotheses are more likely.

Conclusion

The stronger the inductive bias, the better the sample efficiency--this can be understood in terms of the bias-variance tradeoff. Many modern deep learning methods follow an “end-to-end” design philosophy which emphasizes minimal a priori representational and computational assumptions, which explains why they tend to be so data-intensive. On the other hand, there is a lot of research into baking stronger relational inductive biases into deep learning architectures, e.g. with graph networks.

An aside about the word "inductive"

In philosophy, inductive reasoning refers to generalization from specific observations to a conclusion. This is a counterpoint to deductive reasoning, which refers to specialization from general ideas to a conclusion.