Member-only story
What is Inductive Bias in ML?
In the last post we looked at what is the Empirical Risk Minimzer.
In today’s post we look at a problem that comes with this.
Machine Learning from First Principles: Blog Post 3
The minimum value of loss you could have on the training set is 0. In order for the ERM to achieve that minimum, it would do the following,
You call every papaya 0 (not tasty) by default. If the example that you see belongs to the training sample space, you label it Yi (the true label available, and it is available as this is supervised learning). You can achieve 0 loss with this because essentially what you are doing is matching every sample according to the knowledge that you have. The problem with this is that the algorithm fails to generalize. It would perfectly predict previously seen data but for unseen data, the error would be huge. This is termed as Overfitting in ML. Overfitting because as you can see in the plot above, the curve fits exactly as needed according to the training data. We don’t want perfectly fitting curve though.
Intuitively what does this mean? Imagine that you are shown pictures of different birds flying (if they could) and…
