Megatron Memory Estimator

Configuration

For detailed explanations of each parameter, please see the Megatron-LM arguments documentation.

Select a Local Config:

Total GPUs:

micro batch size:

SeqLen:

Use Distributed Optimizer

Recomputation:

TP:

PP:

EP:

CP:

VPP:

ETP:

First Stage Layers:

Last Stage Layers:

Overhead per GPU:

Pipeline Layout (comma-separated layers per stage):

Model	Weight Gradient Optimizer (GB)	Activation (GB)	Total (GB/GPU)	Actions