OpenRecipes
‹ All models
GLM

Training GLM-5.2-744B-A40B on B200

Ranked by tokens/s per GPU · 8,192-token SFT rows · global batch 128 · BF16
#Tokens/s per GPU$ / 1M tokensGPUsLayoutMeasuredAuthor
#1146.8$26.9564TP4 PP2 CP4 EP32 DP2
max 16,384 tokens/GPU, CPU optimizer offload (fraction 0.9)		2026-10-13ATTUNERecipe ›
#2146.7$26.9764TP4 PP2 CP4 EP32 DP2
max 12,288 tokens/GPU, CPU optimizer offload (fraction 0.9)		2026-10-11ATTUNERecipe ›
#3136.5$28.9964TP4 PP2 CP8 EP32 DP1
max 8,192 tokens/GPU, CPU optimizer offload (fraction 0.9)		2026-10-13ATTUNERecipe ›

Notes for GLM-5.2-744B-A40B

Patch slime's mask_utils.py with data/patch_glm5_mask.py before building the rows.

The steps every recipe shares: Running a recipe.

Recipes

#1attune-tp4-pp2-cp4-mt16k146.8 tokens/s/GPUTP4 PP2 CP4 EP32 DP2max 16,384 tokens/GPU, CPU optimizer offload (fraction 0.9)
Author
ATTUNE (cite)
GPUs
64 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout
TP 4, PP 2 (first stage 38 layers, last 40), CP 4, EP 32, ETP 1, DP 2, sequence parallel; max 16,384 tokens per GPU; full recompute (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware optimizer; alltoall dispatcher; --allgather-cp
Tokens/s per GPU
146.80
$ / 1M tokens
$26.95
MFU
1.6% (N = 40,297,758,720 active parameters per token)
Step-time CV
1.8%
Convergence check
correlation 0.999958, final loss difference 7.4e-06, 0 pointwise failures over 20 steps
Component test
optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference: B200, 2026-10-11. Shape: one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert bias, EP 8, ETP 1. worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against 1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
Measured
2026-10-13
GPU-hours
48.85
Rows
glm52-tulu3-fixed8192-v1 (rows_sha256 bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455)

Stack

framework
slime 0.3.0 with local patches; commit not recorded
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
deepep
1.2.1
python
3.12.3
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • INDEXER_ROPE_NEOX_STYLE=0
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  • PYTORCH_ALLOC_CONF=expandable_segments:True
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export INDEXER_ROPE_NEOX_STYLE=0
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

python3 train_async.py \
  --actor-num-nodes 8 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 4 \
  --sequence-parallel \
  --pipeline-model-parallel-size 2 \
  --decoder-first-pipeline-num-layers 38 \
  --decoder-last-pipeline-num-layers 40 \
  --context-parallel-size 4 \
  --expert-model-parallel-size 32 \
  --expert-tensor-parallel-size 1 \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 16384 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --moe-token-dispatcher-type alltoall \
  --allgather-cp
recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp4-mt16k.yamlDownloadGitHub
# recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp4-mt16k.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: GLM-5.2-744B-A40B
gpu:
  type: B200
  memory_gb: 180
  count: 64
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime 0.3.0 with local patches; commit not recorded
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
  deepep: 1.2.1
  python: 3.12.3
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod version dump of the measured run''s training image (same digest), 2026-10-11: Megatron-LM git commit
    and status, installed megatron-core, TE, PyTorch, NCCL, DeepEP and slime, Python'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455
    build:
      name: glm52-tulu3-fixed8192-v1
      tokenizer: GLM-5.2 (tokenizer.json sha256 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d)
      loss_mask: glm5
      rows: 12800
      row_tokens: 8192
      padding_tokens: 0
      seed: 20260828
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.661
      parquet_sha256: 5d951089abd622511a40e0b3a1e5e36ad56859459e4d8c3aff14538cadbd0743
      parquet_bytes: 169929706
      builder: data/generate_fixed_sft.py with patch_glm5_mask.py (data/README.md)
layout:
  label: TP4 PP2 CP4 EP32 DP2
  summary: TP 4, PP 2 (first stage 38 layers, last 40), CP 4, EP 32, ETP 1, DP 2, sequence parallel; max 16,384 tokens
    per GPU; full recompute (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware
    optimizer; alltoall dispatcher; --allgather-cp
  highlights: max 16,384 tokens/GPU, CPU optimizer offload (fraction 0.9)
  tp: 4
  pp: 2
  vpp: 1
  cp: 4
  ep: 32
  etp: 1
  dp: 2
  sequence_parallel: true
  layer_split:
    first: 38
    last: 40
  max_tokens_per_gpu: 16384
  rows_per_microbatch: 8
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload:
    optimizer_fraction: 0.9
    overlap_d2h_h2d: true
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 146.8
  usd_per_million_tokens: 26.95
  step_time_cv_percent: 1.8
  mfu: 0.0158
  mfu_note: null
  active_params:
    count: 40297758720
    source: https://huggingface.co/zai-org/GLM-5.2
    revision: cf457fa734ab149ffef225f80893eb38c6ff5cdc
  measured: 2026-10-13
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 48.85
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; 1 component test passed at this model's shape
  reference: Not published. No public recipe existed for this pair as of 2026-10-11.
  configuration: pass
  component_tests:
  - test: optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference
    shape: 'one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert
      bias, EP 8, ETP 1'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against
      1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.999958
    final_loss_difference: 7.4e-06
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  INDEXER_ROPE_NEOX_STYLE: '0'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  PYTORCH_ALLOC_CONF: expandable_segments:True
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export INDEXER_ROPE_NEOX_STYLE=0
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

  python3 train_async.py \
    --actor-num-nodes 8 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 4 \
    --sequence-parallel \
    --pipeline-model-parallel-size 2 \
    --decoder-first-pipeline-num-layers 38 \
    --decoder-last-pipeline-num-layers 40 \
    --context-parallel-size 4 \
    --expert-model-parallel-size 32 \
    --expert-tensor-parallel-size 1 \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 16384 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --moe-token-dispatcher-type alltoall \
    --allgather-cp
#2attune-tp4-pp2-cp4-mt12k146.7 tokens/s/GPUTP4 PP2 CP4 EP32 DP2max 12,288 tokens/GPU, CPU optimizer offload (fraction 0.9)
Author
ATTUNE (cite)
GPUs
64 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout
TP 4, PP 2 (first stage 38 layers, last 40), CP 4, EP 32, ETP 1, DP 2; max 12,288 tokens per GPU; full recompute (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware optimizer; alltoall dispatcher; --allgather-cp
Tokens/s per GPU
146.68
$ / 1M tokens
$26.97
MFU
1.6% (N = 40,297,758,720 active parameters per token)
Step-time CV
1.58%
Convergence check
correlation 0.9998, final loss difference 0.00051, 0 pointwise failures over 20 steps
Component test
optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference: B200, 2026-10-11. Shape: one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert bias, EP 8, ETP 1. worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against 1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
Measured
2026-10-11, 14-step run scored over steps 4–13
GPU-hours
37.6
Rows
glm52-tulu3-fixed8192-v1 (rows_sha256 bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455)

Stack

framework
slime 0.3.0 with local patches; commit not recorded
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
deepep
1.2.1
python
3.12.3
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • INDEXER_ROPE_NEOX_STYLE=0
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  • PYTORCH_ALLOC_CONF=expandable_segments:True
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export INDEXER_ROPE_NEOX_STYLE=0
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

python3 train_async.py \
  --actor-num-nodes 8 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 14 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 4 \
  --sequence-parallel \
  --pipeline-model-parallel-size 2 \
  --decoder-first-pipeline-num-layers 38 \
  --decoder-last-pipeline-num-layers 40 \
  --context-parallel-size 4 \
  --expert-model-parallel-size 32 \
  --expert-tensor-parallel-size 1 \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 12288 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --moe-token-dispatcher-type alltoall \
  --allgather-cp
recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp4-mt12k.yamlDownloadGitHub
# recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp4-mt12k.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: GLM-5.2-744B-A40B
gpu:
  type: B200
  memory_gb: 180
  count: 64
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime 0.3.0 with local patches; commit not recorded
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
  deepep: 1.2.1
  python: 3.12.3
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod version dump of the measured run''s training image (same digest), 2026-10-11: Megatron-LM git commit
    and status, installed megatron-core, TE, PyTorch, NCCL, DeepEP and slime, Python'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455
    build:
      name: glm52-tulu3-fixed8192-v1
      tokenizer: GLM-5.2 (tokenizer.json sha256 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d)
      loss_mask: glm5
      rows: 12800
      row_tokens: 8192
      padding_tokens: 0
      seed: 20260828
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.661
      parquet_sha256: 5d951089abd622511a40e0b3a1e5e36ad56859459e4d8c3aff14538cadbd0743
      parquet_bytes: 169929706
      builder: data/generate_fixed_sft.py with patch_glm5_mask.py (data/README.md)
layout:
  label: TP4 PP2 CP4 EP32 DP2
  summary: TP 4, PP 2 (first stage 38 layers, last 40), CP 4, EP 32, ETP 1, DP 2; max 12,288 tokens per GPU; full recompute
    (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware optimizer;
    alltoall dispatcher; --allgather-cp
  highlights: max 12,288 tokens/GPU, CPU optimizer offload (fraction 0.9)
  tp: 4
  pp: 2
  vpp: 1
  cp: 4
  ep: 32
  etp: 1
  dp: 2
  sequence_parallel: true
  layer_split:
    first: 38
    last: 40
  max_tokens_per_gpu: 12288
  rows_per_microbatch: 6
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload:
    optimizer_fraction: 0.9
    overlap_d2h_h2d: true
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 146.68
  usd_per_million_tokens: 26.97
  step_time_cv_percent: 1.58
  mfu: 0.0158
  mfu_note: null
  active_params:
    count: 40297758720
    source: https://huggingface.co/zai-org/GLM-5.2
    revision: cf457fa734ab149ffef225f80893eb38c6ff5cdc
  measured: 2026-10-11
  steps_per_run: 14
  window:
  - 4
  - 13
  window_tokens: 10485760
  runs: 1
  profiler: false
  gpu_hours: 37.6
  gpu_hours_note: GPUs x (job start to end), including startup
correctness:
  result: pass
  summary: passed; the optimizer offload fraction passed its component test at this model's shape
  reference: Not published. No public recipe existed for this pair as of 2026-10-11.
  configuration: pass
  component_tests:
  - test: optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference
    shape: 'one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert
      bias, EP 8, ETP 1'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against
      1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.9998
    final_loss_difference: 0.00051
    pointwise_failures: 0
    note: on the measured run's training image
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  INDEXER_ROPE_NEOX_STYLE: '0'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  PYTORCH_ALLOC_CONF: expandable_segments:True
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export INDEXER_ROPE_NEOX_STYLE=0
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

  python3 train_async.py \
    --actor-num-nodes 8 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 14 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 4 \
    --sequence-parallel \
    --pipeline-model-parallel-size 2 \
    --decoder-first-pipeline-num-layers 38 \
    --decoder-last-pipeline-num-layers 40 \
    --context-parallel-size 4 \
    --expert-model-parallel-size 32 \
    --expert-tensor-parallel-size 1 \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 12288 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --moe-token-dispatcher-type alltoall \
    --allgather-cp
#3attune-tp4-pp2-cp8136.5 tokens/s/GPUTP4 PP2 CP8 EP32 DP1max 8,192 tokens/GPU, CPU optimizer offload (fraction 0.9)
Author
ATTUNE (cite)
GPUs
64 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout
TP 4, PP 2 (first stage 38 layers, last 40), CP 8, EP 32, ETP 1, DP 1, sequence parallel; max 8,192 tokens per GPU; full recompute (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware optimizer; alltoall dispatcher; --allgather-cp
Tokens/s per GPU
136.48
$ / 1M tokens
$28.99
MFU
1.5% (N = 40,297,758,720 active parameters per token)
Step-time CV
2.47%
Convergence check
correlation 0.999935, final loss difference 0.00085, 0 pointwise failures over 20 steps
Component test
optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference: B200, 2026-10-11. Shape: one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert bias, EP 8, ETP 1. worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against 1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
Measured
2026-10-13
GPU-hours
51.68
Rows
glm52-tulu3-fixed8192-v1 (rows_sha256 bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455)

Stack

framework
slime 0.3.0 with local patches; commit not recorded
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
deepep
1.2.1
python
3.12.3
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • INDEXER_ROPE_NEOX_STYLE=0
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  • PYTORCH_ALLOC_CONF=expandable_segments:True
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export INDEXER_ROPE_NEOX_STYLE=0
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

python3 train_async.py \
  --actor-num-nodes 8 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 4 \
  --sequence-parallel \
  --pipeline-model-parallel-size 2 \
  --decoder-first-pipeline-num-layers 38 \
  --decoder-last-pipeline-num-layers 40 \
  --context-parallel-size 8 \
  --expert-model-parallel-size 32 \
  --expert-tensor-parallel-size 1 \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --moe-token-dispatcher-type alltoall \
  --allgather-cp
recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp8.yamlDownloadGitHub
# recipes/glm-5.2-744b-a40b/b200/attune-tp4-pp2-cp8.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: GLM-5.2-744B-A40B
gpu:
  type: B200
  memory_gb: 180
  count: 64
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime 0.3.0 with local patches; commit not recorded
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0) with local patches
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (PyTorch's NCCL wheel; printed by the measured run)
  deepep: 1.2.1
  python: 3.12.3
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod version dump of the measured run''s training image (same digest), 2026-10-11: Megatron-LM git commit
    and status, installed megatron-core, TE, PyTorch, NCCL, DeepEP and slime, Python'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: bb302c134974b96f3905e9c87bfc4c0bb2be674a50e6dad09121aed2a0eb3455
    build:
      name: glm52-tulu3-fixed8192-v1
      tokenizer: GLM-5.2 (tokenizer.json sha256 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d)
      loss_mask: glm5
      rows: 12800
      row_tokens: 8192
      padding_tokens: 0
      seed: 20260828
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.661
      parquet_sha256: 5d951089abd622511a40e0b3a1e5e36ad56859459e4d8c3aff14538cadbd0743
      parquet_bytes: 169929706
      builder: data/generate_fixed_sft.py with patch_glm5_mask.py (data/README.md)
layout:
  label: TP4 PP2 CP8 EP32 DP1
  summary: TP 4, PP 2 (first stage 38 layers, last 40), CP 8, EP 32, ETP 1, DP 1, sequence parallel; max 8,192 tokens
    per GPU; full recompute (uniform, 1 layer); CPU optimizer offload (fraction 0.9) with overlapped copies and the precision-aware
    optimizer; alltoall dispatcher; --allgather-cp
  highlights: max 8,192 tokens/GPU, CPU optimizer offload (fraction 0.9)
  tp: 4
  pp: 2
  vpp: 1
  cp: 8
  ep: 32
  etp: 1
  dp: 1
  sequence_parallel: true
  layer_split:
    first: 38
    last: 40
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 8
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload:
    optimizer_fraction: 0.9
    overlap_d2h_h2d: true
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 136.48
  usd_per_million_tokens: 28.99
  step_time_cv_percent: 2.47
  mfu: 0.0147
  mfu_note: null
  active_params:
    count: 40297758720
    source: https://huggingface.co/zai-org/GLM-5.2
    revision: cf457fa734ab149ffef225f80893eb38c6ff5cdc
  measured: 2026-10-13
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 51.68
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; 1 component test passed at this model's shape
  reference: Not published. No public recipe existed for this pair as of 2026-10-11.
  configuration: pass
  component_tests:
  - test: optimizer offload fraction 0.9 against on-GPU Adam and an FP64 reference
    shape: 'one GLM-5.2-744B-A40B MoE layer: hidden 6144, 256 experts, top-8, expert FFN 2048, sigmoid router with expert
      bias, EP 8, ETP 1'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 8.0e-7 against a bound of 1.0e-3 after one step and 7.7e-5 against
      1.10e-3 after three steps at the entry's hyperparameters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.999935
    final_loss_difference: 0.00085
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  INDEXER_ROPE_NEOX_STYLE: '0'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  PYTORCH_ALLOC_CONF: expandable_segments:True
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export INDEXER_ROPE_NEOX_STYLE=0
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
  export PYTORCH_ALLOC_CONF=expandable_segments:True  # set in the pod environment

  python3 train_async.py \
    --actor-num-nodes 8 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 4 \
    --sequence-parallel \
    --pipeline-model-parallel-size 2 \
    --decoder-first-pipeline-num-layers 38 \
    --decoder-last-pipeline-num-layers 40 \
    --context-parallel-size 8 \
    --expert-model-parallel-size 32 \
    --expert-tensor-parallel-size 1 \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --optimizer-cpu-offload --optimizer-offload-fraction 0.9 --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --moe-token-dispatcher-type alltoall \
    --allgather-cp

Citation

If you use a recipe by ATTUNE, please cite the paper.

BibTeX
@article{fan2026attune,
  title   = {{ATTUNE: Agentic Throughput Tuning for LLM Training}},
  author  = {Leo Fan and Hongyang Wang and Wenbo Chen and Jingguo Liu and Xian Li and Xuan Wang and
             Ruochen Wen and Yiming Yan and Hao Wu and Zheshen Wang and Benoit Dumoulin},
  journal = {arXiv preprint arXiv:2610.17528},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.17528}
}

GLM-5.2-744B-A40B on other GPUs

H100 · 256 GPUs
68.3
tokens/s/GPU · $28.00 / 1M tokens
H200 · 128 GPUs
161.8
tokens/s/GPU · $13.58 / 1M tokens
B200 · 64 GPUs · this page
146.8
tokens/s/GPU · $26.95 / 1M tokens