Bias-Variance Tradeoff
The bias‑variance tradeoff is the central challenge in machine learning. It explains why models underfit (high bias) or overfit (high variance), and guides you to find the sweet spot.
Bias: error from wrong assumptions. Variance: error from sensitivity to training data fluctuations.
High Bias (Underfitting)
Model is too simple. It fails to capture patterns in data. Training error and test error are both high. Example: linear regression on non‑linear data.
High Variance (Overfitting)
Model is too complex. It learns noise in the training data. Training error is low, but test error is high. Example: deep decision tree without pruning.
The Tradeoff
As model complexity increases, bias decreases but variance increases. The optimal model minimizes total error (bias² + variance + irreducible error).
Simple model → high bias, low variance
Complex model → low bias, high variance
Sweet spot → balance that minimizes test errorHow to Diagnose
- High training error → high bias (underfitting) → increase model complexity.
- Low training error, high test error → high variance (overfitting) → reduce complexity or add regularization.
Two Minute Drill
- High bias = underfitting (model too simple).
- High variance = overfitting (model too complex).
- Goal: balance bias and variance to minimize test error.
- Use training/test error to diagnose.
Need more clarification?
Drop us an email at career@quipoinfotech.com