Diagnosing overfitting in deep learning: a field guide.

Every deep learning model walks a line between two failure modes: learning too little from the data, and learning the data itself instead of the pattern behind it. Knowing which side of that line your model is on — and how far over — is the single most important diagnostic skill in applied deep learning. This post is a field guide to it: what over- and underfitting are, and then the full toolbox of signals that reveal fit quality, from the everyday (validation curves) to the less commonly used (weight distributions, gradient norms, the random-label test, calibration).

Overfitting and underfitting, properly defined

A model is trained to minimize error on the training set, but what we actually care about is error on data it has never seen — the generalization error. The two failure modes are defined by the relationship between them:

• Underfitting — the model is too simple, too constrained, or too under-trained to capture the real structure in the data. Symptom: high error on the training set itself. If the model can't even fit the data it has seen, it has no chance on data it hasn't.
• Overfitting — the model has captured not just the signal but also the noise: quirks, coincidences, and even individual examples of the training set that don't hold in the population. Symptom: low training error, but substantially higher error on held-out data. The difference between the two is called the generalization gap.

The classical framing is the bias–variance trade-off: simple models have high bias (systematically wrong, everywhere), complex models have high variance (exquisitely right on the training set, unstable off it), and you tune capacity to sit in the valley between them. Deep learning complicates this tidy picture — heavily overparameterized networks can generalize well despite having the capacity to memorize everything, and the test error curve can even improve again past the point of interpolation (the "double descent" phenomenon). But the practical diagnosis workflow is unchanged: you measure training performance and held-out performance, and read the gap.

One prerequisite makes all of the following meaningful: a clean split into training (fit the weights), validation (tune decisions: hyperparameters, early stopping, architecture), and test (touched once, at the end, to report performance). Every diagnostic below leans on the validation set — and every one of them is silently invalidated by leakage between the splits, which is why leakage gets its own section near the end.

Signal 1: learning curves — train vs. validation loss

The workhorse. Log training loss and validation loss every epoch and plot them together; the shape of the two curves is a remarkably complete diagnosis:

The corresponding training loop, with the logging and early stopping built in:

import copy
import torch

def train(model, train_loader, val_loader, loss_fn, optimizer,
          max_epochs=200, patience=10):
    history = {"train_loss": [], "val_loss": []}
    best_val, best_state, wait = float("inf"), None, 0

    for epoch in range(max_epochs):
        # --- training pass ---
        model.train()
        train_loss = 0.0
        for X, y in train_loader:
            optimizer.zero_grad()
            loss = loss_fn(model(X), y)
            loss.backward()
            optimizer.step()
            train_loss += loss.item() * len(X)
        train_loss /= len(train_loader.dataset)

        # --- validation pass: no gradients, eval mode ---
        model.eval()
        val_loss = 0.0
        with torch.no_grad():
            for X, y in val_loader:
                val_loss += loss_fn(model(X), y).item() * len(X)
        val_loss /= len(val_loader.dataset)

        history["train_loss"].append(train_loss)
        history["val_loss"].append(val_loss)
        print(f"epoch {epoch:3d}  train {train_loss:.4f}  val {val_loss:.4f}")

        # --- early stopping on validation loss ---
        if val_loss < best_val:
            best_val, wait = val_loss, 0
            best_state = copy.deepcopy(model.state_dict())
        else:
            wait += 1
            if wait >= patience:
                print(f"early stop at epoch {epoch}, best val {best_val:.4f}")
                break

    model.load_state_dict(best_state)   # roll back to the best epoch
    return history

Early stopping — halting when validation loss hasn't improved for patience epochs and restoring the best checkpoint — is both a diagnostic and the cheapest regularizer you own: it simply refuses to keep training into the memorization regime. Note the two easy-to-miss details in the code: model.eval() (disables dropout, freezes batch-norm statistics) and torch.no_grad() during validation. Getting either wrong corrupts the very curve you're diagnosing with.

Signal 2: metrics beyond the loss

Track the metric you actually care about (accuracy, F1, AUC, MAE) on both splits, alongside the loss — because the two can disagree, and the disagreement is informative. A classic example: validation loss rises while validation accuracy holds steady. That usually means the model's predictions are still mostly correct but growing overconfident — wrong answers are becoming very confidently wrong, which cross-entropy punishes hard. That's mild overfitting in confidence before it shows up in correctness, and a hint that calibration (signal 6) is drifting.

For imbalanced problems, loss and accuracy can both look fine while the minority-class recall collapses — slice your validation metrics by class and by any segment you care about (time period, user group, data source). Overfitting is often local: the model memorizes the sparse regions of the data first.

Signal 3: weight distributions and norms

The loss curves tell you that something is wrong; the weights can tell you what. A healthy network's weight distribution typically stays roughly zero-centered and bell-shaped as training progresses, drifting smoothly from its initialization. Two patterns are worth watching for:

• Growing weight magnitudes / heavy tails. A steadily climbing weight norm, or histograms sprouting extreme outliers, often accompanies memorization: fitting noise tends to require large, finely tuned weights, which is precisely why weight decay (L2 regularization) — a penalty on that norm — combats overfitting. If the norm inflates in lockstep with a widening train/val gap, you have converging evidence.
• Dead or collapsed layers. Histograms frozen at initialization, or collapsed onto a spike at zero, indicate a layer that isn't learning (vanishing gradients, dead ReLUs, a too-aggressive learning rate earlier in training). That's an underfitting signature, localized to a layer.

Logging this costs three lines with TensorBoard:

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter("runs/experiment-1")

# inside the epoch loop:
for name, param in model.named_parameters():
    writer.add_histogram(f"weights/{name}", param, epoch)
writer.add_scalar("weights/global_l2_norm",
                  torch.sqrt(sum(p.pow(2).sum() for p in model.parameters())),
                  epoch)
writer.add_scalar("loss/train", train_loss, epoch)
writer.add_scalar("loss/val", val_loss, epoch)

Then tensorboard --logdir runs gives you per-layer histograms over time. Read them comparatively — against earlier epochs of the same run and against a run that generalized well — rather than hunting for an absolute "good" shape; architectures differ too much for universal thresholds.

Signal 4: gradient norms

Gradients are the training signal itself, and their magnitude over time is a useful fit thermometer. Log the global gradient norm each step (before optimizer.step()):

grad_norm = torch.sqrt(sum(
    p.grad.pow(2).sum() for p in model.parameters() if p.grad is not None
))
writer.add_scalar("grad_norm/global", grad_norm, step)

• Gradients collapsing toward zero while training loss is still high — the optimizer has stalled (vanishing gradients, saturated activations, learning rate decayed too far): an underfitting mechanism.
• Exploding or spiky norms — unstable optimization; the loss curves that result are too noisy to diagnose fit from. Fix stability first (gradient clipping, lower learning rate), then re-read the curves.
• Near-zero training loss with near-zero gradients — the model has fully fit the training set. Whatever the validation curve says at this point is final; more epochs can only entrench memorization.

Signal 5: the random-label test — measuring memorization capacity

A famous result (Zhang et al., 2017) showed that standard deep networks can reach near-perfect training accuracy on datasets whose labels have been randomly shuffled — pure memorization, since there is no signal left to learn. You can turn this into a practical diagnostic for your own setup:

import torch

shuffled = torch.randperm(len(y_train))
y_random = y_train[shuffled]          # destroys the input–label relationship

# train an identical fresh model on (X_train, y_random) and watch training loss

Two readings come out of it. If your model drives training loss to zero on random labels quickly, its capacity vastly exceeds what the dataset constrains — not fatal, but a warning that regularization and early stopping are load-bearing. And comparing the real-label and random-label training curves shows how much of your model's training performance could be achieved by memorization alone: the earlier and lower the real-label curve separates from the random-label curve, the more actual structure it is learning. As a bonus, a model that reaches high accuracy on validation data under shuffled training labels has just proven your pipeline leaks (see signal 7).

Signal 6: prediction confidence and calibration

An overfit classifier is typically an overconfident one: softmax probabilities pile up near 1.0, including on the examples it gets wrong. Calibration measures whether confidence means what it says — among predictions made with 80% confidence, are about 80% correct? The standard tools are the reliability diagram (plot accuracy against binned confidence; the diagonal is perfect calibration) and its scalar summary, expected calibration error (ECE):

import torch

@torch.no_grad()
def expected_calibration_error(model, loader, n_bins=10):
    model.eval()
    confs, correct = [], []
    for X, y in loader:
        probs = torch.softmax(model(X), dim=1)
        conf, pred = probs.max(dim=1)
        confs.append(conf)
        correct.append(pred.eq(y))
    conf = torch.cat(confs); corr = torch.cat(correct).float()

    ece = torch.zeros(1)
    edges = torch.linspace(0, 1, n_bins + 1)
    for lo, hi in zip(edges[:-1], edges[1:]):
        in_bin = (conf > lo) & (conf <= hi)
        if in_bin.any():
            ece += in_bin.float().mean() * (corr[in_bin].mean() - conf[in_bin].mean()).abs()
    return ece.item()

Rising ECE on the validation set across training epochs is overfitting-in-confidence, and it often precedes the accuracy drop. It also matters in its own right: downstream systems that threshold on model confidence (triage, human-in-the-loop routing) silently degrade when calibration drifts, even at constant accuracy.

Signal 7: too good to be true — data leakage

The inverted diagnosis: validation performance that is suspiciously good, or a validation curve glued to the training curve on a hard problem. Before celebrating, rule out leakage — information from the validation set reaching training. The usual suspects:

• Duplicates (or near-duplicates) present in both splits — endemic in scraped and augmented datasets. Deduplicate before splitting.
• Group leakage — multiple samples from the same patient, user, session, or recording split across train and validation. Split by group, not by row.
• Temporal leakage — random splits on time series let the model train on the future. Split by time.
• Preprocessing leakage — normalization statistics, feature selection, or augmentation policies computed on the full dataset before splitting. Fit all preprocessing on the training split only.

Leakage is the most expensive fit-quality failure in practice because it inverts your compass: it makes the overfit model look like the best one, and everything downstream (model selection, early stopping, reported metrics) inherits the error. When a result looks too good, treat that as a bug report until proven otherwise.

Signal 8: robustness checks — does the fit survive contact with variation?

Three quick probes that reveal fragile fits which look fine on the standard curves:

• Seed variance. Retrain with 3–5 random seeds. Large spread in validation performance means your model's fate hangs on initialization noise — a high-variance (overfitting-side) regime, and a sign that single-run comparisons between architectures or hyperparameters are meaningless.
• Input perturbation. Evaluate on lightly corrupted validation data (noise, small shifts, paraphrases — whatever is natural for the modality). A model whose accuracy craters under perturbations that don't confuse a human has fit surface statistics rather than the underlying concept.
• Out-of-distribution slices. Hold out an entire site, year, or cohort during training and evaluate on it. Standard validation measures generalization to the same distribution; deployment rarely has that courtesy. (In my satellite-imagery work, region-held-out evaluation is the difference between a model that works in the training country and one that works across Western Africa.)

Relatedly, for small datasets, prefer k-fold cross-validation over a single split: it both stabilizes the estimate (every sample gets a turn in validation) and exposes variance across folds, which is itself a fit-quality signal.

A worked diagnosis routine

Putting it together as the checklist I actually run when a model's quality is in doubt:

1. Sanity-check the measurement — clean splits (grouped/temporal where needed), preprocessing fit on train only, validation set large enough for stable numbers.
2. Read the learning curves — classify into underfit / overfit / good fit from the table above; note where validation loss bottoms out.
3. Overlay the task metric — check for loss/metric divergence and per-slice collapse.
4. Check the weights and gradients — norm inflation, dead layers, optimization stalls; distinguish "can't learn" from "learned too much."
5. Probe memorization and confidence — random-label run if capacity is in question; ECE trend if downstream uses confidence.
6. Stress the fit — seeds, perturbations, held-out slices.
7. Then, and only then, intervene — more data or augmentation, weight decay, dropout, smaller model, earlier stopping for overfitting; more capacity, longer training, higher learning rate, less regularization for underfitting. Change one thing at a time and re-read the curves.

A closing note on the modern regime: with pretrained backbones and foundation models, classical overfitting on the pretraining corpus is rarely your problem — but overfitting during fine-tuning on your small task dataset absolutely is, and it moves fast (sometimes within a fraction of an epoch). Everything above applies, just on a compressed timescale: validate frequently, not once per epoch, and keep the early-stopping patience short.

References

• Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR 2017). arXiv:1611.03530.

All posts