Tradeoff between bias-variance
When I started with my Machine Learning journey, it was always difficult for me to remember the difference between Bias and Variance. Now that I understand these terms, I hope my post will help others to form a basic understanding of these two terms.
Let’s talk about bias first. In simple terms, bias is the difference between the model prediction and the ground truth i.e. actual value of the response variable. High Bias means that your model does not capture some important features and is underfitting the training data (high error on the training data). This could be because you are using a simple model that is not flexible to capture the true relationship of the data.
The most simplified representation for bias and variance is depicted in the image below. In the image on the left, it tries to fit a linear model which underfits our training split and thus leading to high bias. In the image on the right, the complex model tries to fit well to every data point in our training split and can’t generalize well to data apart from the training, thus leading to high variance.
Variance is the difference in the model prediction on the train and test data. If the model performs well on the training data (low training error) but fails to generalize well to the test data (high test dataset error), it said to have a high variance. This essentially means that your model is overfitting the training data. It could be because the model has captured the random noise in the data or has not seen enough data while training to generalize well to the unseen examples.
High Bias (UNDESIRABLE): This type of model will either predict the same output every time or will take a random guess.
Low Bias/High Variance (UNDESIRABLE): Models which have low bias and high variance overfit the training data. Such models perform poorly on the unseen data.
Low Bias/Low Variance (DESIRABLE): This kind of models almost always gives the best results. They perform well on both, seen and unseen examples.
To avoid underfitting (high bias), try to increase the number of features by finding new features or making new features from the existing ones that can better represent the relationship between the data.
To avoid overfitting (high variance), try the following -
1. Increase the training data (collecting more data/augment the training dataset)
2. L1/L2 regularization to simplify your model.
Bias-variance is a tradeoff which should be optimized by carefully selecting the models, and different hyperparameters. A model with a low bias and variance is preferred.
Hope this post helped you form a basic understanding of bias and variance. Please feel free to leave any feedback and questions below in the comments. Thanks for reading this today.
Links to learn more about bias and variance -
