OpenRecipes
‹ All models
Qwen

Training Qwen3.5-122B-A10B on H200

Ranked by tokens/s per GPU · 8,192-token SFT rows · global batch 128 · BF16
#Tokens/s per GPU$ / 1M tokensGPUsLayoutMeasuredAuthor
#12,451.4$0.9032TP1 PP4 EP8 DP8
max 8,192 tokens/GPU, recompute 2 layers, load-balancing loss off		2026-10-13ATTUNERecipe ›
#22,366.7$0.9332TP1 PP4 EP8 DP8
max 8,192 tokens/GPU, load-balancing loss off		2026-10-11ATTUNERecipe ›
#32,289.7$0.9632TP1 PP4 EP8 DP8
max 8,192 tokens/GPU, flex dispatcher, load-balancing loss off		2026-10-13ATTUNERecipe ›

Recipes

#1attune-tp1-pp4-ep8-rc22,451.4 tokens/s/GPUTP1 PP4 EP8 DP8max 8,192 tokens/GPU, recompute 2 layers, load-balancing loss off
Author
ATTUNE (cite)
GPUs
32 x H200 141 GB (AWS p5en.48xlarge, 8 per node)
Layout
TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 2 layers); precision-aware distributed optimizer; alltoall dispatcher; load-balancing loss off; TE cross-entropy fusion; no offload
Tokens/s per GPU
2,451.44
$ / 1M tokens
$0.90
MFU
13.4% (N = 9,008,136,192 active parameters per token)
Step-time CV
0.57%
Convergence check
correlation 0.999997, final loss difference 0.00042, 0 pointwise failures over 20 steps
Measured
2026-10-13
GPU-hours
8.96
Rows
qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework
slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output gate at TP above the query-group count
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep
1.2.1
python
3.12
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  • TORCHINDUCTOR_COMPILE_THREADS=1
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

# MODEL_ARGS: the model script's, with the load-balancing loss off
python3 -m <slime entry module> --async \
  --actor-num-nodes 4 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --loss-mask-type qwen3_5 \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 4 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 2 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
  --distributed-timeout-minutes 30
recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8-rc2.yamlDownloadGitHub
# recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8-rc2.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-122B-A10B
gpu:
  type: H200
  memory_gb: 141
  count: 32
  per_node: 8
  instance: AWS p5en.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: 'slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step'
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output
    gate at TP above the query-group count
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP1 PP4 EP8 DP8
  summary: TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 2 layers);
    precision-aware distributed optimizer; alltoall dispatcher; load-balancing loss off; TE cross-entropy fusion; no offload
  highlights: max 8,192 tokens/GPU, recompute 2 layers, load-balancing loss off
  tp: 1
  pp: 4
  vpp: 1
  cp: 1
  ep: 8
  etp: 1
  dp: 8
  sequence_parallel: false
  layer_split: null
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 1
  recompute:
    granularity: full
    method: uniform
    num_layers: 2
  offload: null
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 2451.44
  usd_per_million_tokens: 0.9
  step_time_cv_percent: 0.57
  mfu: 0.134
  mfu_note: null
  active_params:
    count: 9008136192
    source: https://huggingface.co/Qwen/Qwen3.5-122B-A10B
    revision: dc4d348443bc740c68e2d77492492c11606384d5
  measured: 2026-10-13
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 8.96
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed
  reference: Not published.
  configuration: pass
  component_tests: []
  convergence:
    steps: 20
    result: pass
    correlation: 0.999997
    final_loss_difference: 0.00042
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  # MODEL_ARGS: the model script's, with the load-balancing loss off
  python3 -m <slime entry module> --async \
    --actor-num-nodes 4 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --loss-mask-type qwen3_5 \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 1 \
    --pipeline-model-parallel-size 4 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 2 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
    --distributed-timeout-minutes 30
#2attune-tp1-pp4-ep82,366.7 tokens/s/GPUTP1 PP4 EP8 DP8max 8,192 tokens/GPU, load-balancing loss off
Author
ATTUNE (cite)
GPUs
32 x H200 141 GB (AWS p5en.48xlarge, 8 per node)
Layout
TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 1 layer); alltoall dispatcher with shared-expert overlap; load-balancing loss off; TE cross-entropy fusion; precision-aware distributed optimizer; no offload
Tokens/s per GPU
2,366.73
$ / 1M tokens
$0.93
MFU
12.9% (N = 9,008,136,192 active parameters per token)
Step-time CV
0.61%
Convergence check
correlation 0.999993, final loss difference 0.000819, 0 pointwise failures over 20 steps
Measured
2026-10-11, 14-step run scored over steps 4–13
GPU-hours
8.2
Rows
qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework
slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output gate at TP above the query-group count
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep
1.2.1
python
3.12
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  • TORCHINDUCTOR_COMPILE_THREADS=1
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

# MODEL_ARGS: the model script's, with the load-balancing loss off
python3 -m <slime entry module> --async \
  --actor-num-nodes 4 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 14 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --loss-mask-type qwen3_5 \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 4 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --moe-shared-expert-overlap \
  --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
  --distributed-timeout-minutes 30
recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8.yamlDownloadGitHub
# recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-122B-A10B
gpu:
  type: H200
  memory_gb: 141
  count: 32
  per_node: 8
  instance: AWS p5en.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: 'slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step'
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output
    gate at TP above the query-group count
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP1 PP4 EP8 DP8
  summary: TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 1 layer);
    alltoall dispatcher with shared-expert overlap; load-balancing loss off; TE cross-entropy fusion; precision-aware
    distributed optimizer; no offload
  highlights: max 8,192 tokens/GPU, load-balancing loss off
  tp: 1
  pp: 4
  vpp: 1
  cp: 1
  ep: 8
  etp: 1
  dp: 8
  sequence_parallel: false
  layer_split: null
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 1
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload: null
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 2366.73
  usd_per_million_tokens: 0.93
  step_time_cv_percent: 0.61
  mfu: 0.1293
  mfu_note: null
  active_params:
    count: 9008136192
    source: https://huggingface.co/Qwen/Qwen3.5-122B-A10B
    revision: dc4d348443bc740c68e2d77492492c11606384d5
  measured: 2026-10-11
  steps_per_run: 14
  window:
  - 4
  - 13
  window_tokens: 10485760
  runs: 1
  profiler: false
  gpu_hours: 8.2
  gpu_hours_note: GPUs x (job start to end), including startup
correctness:
  result: pass
  summary: passed, with the load-balancing loss off in both arms
  reference: Not published.
  configuration: pass
  component_tests: []
  convergence:
    steps: 20
    result: pass
    correlation: 0.999993
    final_loss_difference: 0.000819
    pointwise_failures: 0
    note: both arms with the load-balancing loss off, same Tulu 3 rows and seed, on the measured run's training image;
      0 of 16 scored points outside tolerance
    worst_step_tolerance_ratio: 0.026
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  # MODEL_ARGS: the model script's, with the load-balancing loss off
  python3 -m <slime entry module> --async \
    --actor-num-nodes 4 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 14 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --loss-mask-type qwen3_5 \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 1 \
    --pipeline-model-parallel-size 4 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --moe-shared-expert-overlap \
    --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
    --distributed-timeout-minutes 30
#3attune-tp1-pp4-ep8-flex2,289.7 tokens/s/GPUTP1 PP4 EP8 DP8max 8,192 tokens/GPU, flex dispatcher, load-balancing loss off
Author
ATTUNE (cite)
GPUs
32 x H200 141 GB (AWS p5en.48xlarge, 8 per node)
Layout
TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 1 layer); precision-aware distributed optimizer; flex dispatcher; load-balancing loss off; TE cross-entropy fusion; no offload
Tokens/s per GPU
2,289.65
$ / 1M tokens
$0.96
MFU
12.5% (N = 9,008,136,192 active parameters per token)
Step-time CV
0.66%
Convergence check
correlation 0.999998, final loss difference 0.00013, 0 pointwise failures over 20 steps
Component test
flex dispatcher against an FP32 reference: H200, 2026-10-13. Shape: one Qwen3.5-122B-A10B MoE layer: hidden 3072, 256 experts, top-8, expert FFN 1024, EP 8, ETP 1, TP 1; 4,096, 8,192 and 16,384 tokens per rank. routing identical; worst tensor (output) relative L2 error 4.11e-03 against a bound of 8.76e-03 (2 x the default dispatcher's error + 1e-3) at 8,192 tokens per rank, within bound at 4,096 and 16,384; the rerun is bitwise identical and the negative controls fail
Measured
2026-10-13
GPU-hours
9.0
Rows
qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework
slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step
megatron lm
commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output gate at TP above the query-group count
transformer engine
2.10.0
pytorch
2.11.0 (CUDA 12.9 build)
cuda
12.9
nccl
2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep
1.2.1
python
3.12
network
8 GPUs per node, EFA, 3.2 Tb/s RDMA per node

Environment

  • CUDA_DEVICE_MAX_CONNECTIONS=1
  • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  • TORCHINDUCTOR_COMPILE_THREADS=1
  • NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
CommandHow to run
export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

# MODEL_ARGS: the model script's, with the load-balancing loss off
python3 -m <slime entry module> --async \
  --actor-num-nodes 4 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --loss-mask-type qwen3_5 \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 1 \
  --pipeline-model-parallel-size 4 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
  --distributed-timeout-minutes 30 \
  --moe-token-dispatcher-type flex
recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8-flex.yamlDownloadGitHub
# recipes/qwen3.5-122b-a10b/h200/attune-tp1-pp4-ep8-flex.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-122B-A10B
gpu:
  type: H200
  memory_gb: 141
  count: 32
  per_node: 8
  instance: AWS p5en.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: 'slime 0.3.0: THUDM/slime a897e1f plus 8 commits that do not change the training step'
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0), with a backported upstream fix for the Qwen3.5 attention output
    gate at TP above the query-group count
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP1 PP4 EP8 DP8
  summary: TP 1, PP 4, CP 1, EP 8, ETP 1, DP 8; one 8,192-token row per microbatch; full recompute (uniform, 1 layer);
    precision-aware distributed optimizer; flex dispatcher; load-balancing loss off; TE cross-entropy fusion; no offload
  highlights: max 8,192 tokens/GPU, flex dispatcher, load-balancing loss off
  tp: 1
  pp: 4
  vpp: 1
  cp: 1
  ep: 8
  etp: 1
  dp: 8
  sequence_parallel: false
  layer_split: null
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 1
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload: null
  dispatcher: flex
result:
  tokens_per_s_per_gpu: 2289.65
  usd_per_million_tokens: 0.96
  step_time_cv_percent: 0.66
  mfu: 0.1251
  mfu_note: null
  active_params:
    count: 9008136192
    source: https://huggingface.co/Qwen/Qwen3.5-122B-A10B
    revision: dc4d348443bc740c68e2d77492492c11606384d5
  measured: 2026-10-13
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 9.0
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; 1 component test passed at this model's shape
  reference: Not published.
  configuration: pass
  component_tests:
  - test: flex dispatcher against an FP32 reference
    shape: 'one Qwen3.5-122B-A10B MoE layer: hidden 3072, 256 experts, top-8, expert FFN 1024, EP 8, ETP 1, TP 1; 4,096,
      8,192 and 16,384 tokens per rank'
    gpu: H200
    date: 2026-10-13
    result: pass
    detail: routing identical; worst tensor (output) relative L2 error 4.11e-03 against a bound of 8.76e-03 (2 x the default
      dispatcher's error + 1e-3) at 8,192 tokens per rank, within bound at 4,096 and 16,384; the rerun is bitwise identical
      and the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.999998
    final_loss_difference: 0.00013
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches: []
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  # MODEL_ARGS: the model script's, with the load-balancing loss off
  python3 -m <slime entry module> --async \
    --actor-num-nodes 4 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --loss-mask-type qwen3_5 \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 1 \
    --pipeline-model-parallel-size 4 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --cross-entropy-loss-fusion --cross-entropy-fusion-impl te \
    --distributed-timeout-minutes 30 \
    --moe-token-dispatcher-type flex

Citation

If you use a recipe by ATTUNE, please cite the paper.

BibTeX
@article{fan2026attune,
  title   = {{ATTUNE: Agentic Throughput Tuning for LLM Training}},
  author  = {Leo Fan and Hongyang Wang and Wenbo Chen and Jingguo Liu and Xian Li and Xuan Wang and
             Ruochen Wen and Yiming Yan and Hao Wu and Zheshen Wang and Benoit Dumoulin},
  journal = {arXiv preprint arXiv:2610.17528},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.17528}
}

Qwen3.5-122B-A10B on other GPUs

H100 · 48 GPUs
999.7
tokens/s/GPU · $1.91 / 1M tokens
H200 · 32 GPUs · this page
2,451.4
tokens/s/GPU · $0.90 / 1M tokens
B200 · 16 GPUs
2,475.8
tokens/s/GPU · $1.60 / 1M tokens