Overfitting and Underfitting
Deep learning models are prone to overfitting (memorizing training data) or underfitting (failing to learn patterns). Recognizing and addressing these issues is critical.
Signs of Overfitting
Training loss is very low, but validation loss is high and may even increase. Model performs well on seen data but poorly on unseen data.
- Fix overfitting: Add dropout, increase weight decay, reduce model size, increase data augmentation, use early stopping, simplify architecture.
Signs of Underfitting
Both training and validation losses remain high and do not decrease. Model fails to capture underlying patterns.
- Fix underfitting: Increase model size (more layers/units), train longer, reduce regularization, choose a better optimizer, improve data preprocessing.
Diagnosing with Learning Curves
Plot training and validation loss over epochs:
- Both low and close → good fit.
- Train low, val high → overfitting.
- Both high and flat → underfitting.
- Train low, val oscillating → may need smaller learning rate or batch size.
Practical Approach
Start with a small model, ensure it can overfit a tiny subset (e.g., 10 samples). Then gradually add complexity, monitor validation loss, and use regularization as needed.
Two Minute Drill
- Overfitting: train loss low, validation high → add regularization.
- Underfitting: both losses high → increase model capacity.
- Plot learning curves to diagnose.
- Start simple, then scale up.
Need more clarification?
Drop us an email at career@quipoinfotech.com