[QUESTION] Hello, how to accurately calculate memory usage based on operating parameters?

[13416157913]

Hello, how to accurately calculate memory usage based on operating parameters ?
For example,
TP_SIZE=1
PP_SIZE=1
WORLD_SIZE=8
MICRO_BATCH_SIZE=2
GLOBAL_BATCH_SIZE=128
--finetune
--sequence-parallel
--num-layers 32
--hidden-size 4096
--num-attention-heads 32
--seq-length 4096
--max-position-embeddings 4096
--no-position-embedding
--use-rotary-position-embeddings
--swiglu
--ffn-hidden-size 11008
--disable-bias-linear
--RMSNorm
--layernorm-epsilon 1e-6
--causal-lm
--distributed-optimizer
--use-flash-attn

[hwdef]

#482
Please check this

[13416157913]

#482 Please check this

Thank you very much.

[deepakn94]

We also now have a report_theoretical_memory.py script now that should take the same set of arguments as pretrain_gpt.py.

You can use like this:

CUDA_DEVICE_MAX_CONNECTIONS=1 WORLD_SIZE=<WORLD_SIZE> python -u report_theoretical_memory.py ${options}

[github-actions]

Marking as stale. No activity in 60 days.

[deepakn94]

Going to close this. Feel free to re-open if you are still running into issues.