GPU memory usage going up every epoch

[AlejandroRigau]

I am trying to figure out why the GPU memory usage keeps going up after each epoch. What causes this to happen? I was in the understanding that this should remain constant.

These are my hyperparameters:

python scripts/train.py "datasets/data (2)_balanced_train_val_test" \
--workers 6 \
--model efficientnet_b0 \
--batch-size 64 \
--num-classes 2 \
--lr 0.05 \
--opt sgd \
--momentum 0.9 \
--epochs 100 \
--warmup-epochs 3 \
--seed 42 \
--drop 0.0 \
--sched step \
--weight-decay 0.03 \
--clip-grad 1 \
--clip-mode norm \
--train-interpolation bicubic \
--hflip 0 \
--vflip 0 \
--color-jitter 0.0 \
--cutmix 0.0 \
--scale 1 1 \
--ratio 1 1 \
--log-wandb \
--save-images

[rwightman]

@AlejandroRigau with caching allocators (like the pytorch one) that's not a reliable indicator of a leak, memory churn can increase the overall allocated without actually causing any issues (just more cached allocations that may get reused, only recovered if needed).

I've used the scripts enough for training on the order of months sometimes so I'm fairly certain no issues. If you're only using 70-somthing% at the start you should try pushing your batch size up such that you use 90-95%, going all the way to the limit can result in a OOM when you transition between train-eval-train