Qwen

Training Qwen3.5-397B-A17B on B200

Ranked by tokens/s per GPU · 8,192-token SFT rows · global batch 128 · BF16

#	Tokens/s per GPU	$ / 1M tokens	GPUs	Layout	Measured	Author
#1	1,232.5	$3.21	64	TP2 PP8 VPP2 EP8 DP4 max 16,384 tokens/GPU, VPP 2, no offload	2026-10-18	ATTUNE	Recipe ›
#2	1,130.7	$3.50	64	TP2 PP8 VPP2 EP8 DP4 max 8,192 tokens/GPU, VPP 2, no offload	2026-10-18	ATTUNE	Recipe ›
#3	1,075.5	$3.68	48	TP2 PP6 EP8 DP4 max 8,192 tokens/GPU, no offload	2026-10-18	ATTUNE	Recipe ›

Recipes

#1attune-tp2-pp8-ep8-vpp2-mt16k1,232.5 tokens/s/GPUTP2 PP8 VPP2 EP8 DP4max 16,384 tokens/GPU, VPP 2, no offload

B200 runs of this model need patches/fla_blackwell_guard.py: with the image's Triton 3.6, FLA 0.4.1's autotuner picks gated-delta-net kernel configs that race on Blackwell and give a different, wrong result on every call, and the patch keeps those kernels on the configs FLA main uses there. Every training process on every node imports it before FLA (patches/README.md) and prints a [fla-guard] line.

Author: ATTUNE (cite)
GPUs: 64 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout: TP 2, PP 8 (first and last stage 6 layers), VPP 2, CP 1, EP 8, ETP 1, DP 4; two 8,192-token rows per microbatch (max 16,384 tokens per GPU); full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher; no offload
Tokens/s per GPU: 1,232.53
$ / 1M tokens: $3.21
MFU: 5.4% (N = 16,331,407,360 active parameters per token)
Step-time CV: 2.72%
Convergence check: correlation 0.9999986, final loss difference 8.6e-05, 0 pointwise failures over 20 steps
Component test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an FP64 reference: B200, 2026-10-11. Shape: one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens per rank. worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit FP32 masters; the negative controls fail
Measured: 2026-10-18
GPU-hours: 22.49
Rows: qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework: slime with local patches, not pinned
megatron lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
transformer engine: 2.10.0
pytorch: 2.11.0 (CUDA 12.9 build)
cuda: 12.9
nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep: 1.2.1
python: 3.12
fla: 0.4.1 (flash-linear-attention)
triton: 3.6.0
network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
runtime fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA

Environment

CUDA_DEVICE_MAX_CONNECTIONS=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
TORCHINDUCTOR_COMPILE_THREADS=1
NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

Patches

patches/fla_blackwell_guard.py

CommandHow to run

export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

python3 -m <slime entry module> --async \
  --actor-num-nodes 8 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 2 \
  --pipeline-model-parallel-size 8 \
  --decoder-first-pipeline-num-layers 6 \
  --decoder-last-pipeline-num-layers 6 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 16384 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --distributed-timeout-minutes 30 \
  --num-virtual-stages-per-pipeline-rank 2

recipes/qwen3.5-397b-a17b/b200/attune-tp2-pp8-ep8-vpp2-mt16k.yamlDownload GitHub

# recipes/qwen3.5-397b-a17b/b200/attune-tp2-pp8-ep8-vpp2-mt16k.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-397B-A17B
gpu:
  type: B200
  memory_gb: 180
  count: 64
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime with local patches, not pinned
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  fla: 0.4.1 (flash-linear-attention)
  triton: 3.6.0
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  runtime_fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log; FLA and
    Triton from an in-pod check of the same digest on 2026-10-18'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP2 PP8 VPP2 EP8 DP4
  summary: TP 2, PP 8 (first and last stage 6 layers), VPP 2, CP 1, EP 8, ETP 1, DP 4; two 8,192-token rows per microbatch
    (max 16,384 tokens per GPU); full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher;
    no offload
  highlights: max 16,384 tokens/GPU, VPP 2, no offload
  tp: 2
  pp: 8
  vpp: 2
  cp: 1
  ep: 8
  etp: 1
  dp: 4
  sequence_parallel: true
  layer_split:
    first: 6
    last: 6
  max_tokens_per_gpu: 16384
  rows_per_microbatch: 2
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload: null
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 1232.53
  usd_per_million_tokens: 3.21
  step_time_cv_percent: 2.72
  mfu: 0.0537
  mfu_note: null
  active_params:
    count: 16331407360
    source: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
    revision: 8472618112abcbd45acbcdc58436aff4233c23f7
  measured: 2026-10-18
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 22.49
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; the on-GPU optimizer passed its component test at this model's shape
  reference: Not published.
  configuration: pass
  component_tests:
  - test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an
      FP64 reference
    shape: 'one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens
      per rank'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound
      after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit
      FP32 masters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.9999986
    final_loss_difference: 8.6e-05
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both; both with patches/fla_blackwell_guard.py'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches:
- patches/fla_blackwell_guard.py
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  python3 -m <slime entry module> --async \
    --actor-num-nodes 8 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 2 \
    --pipeline-model-parallel-size 8 \
    --decoder-first-pipeline-num-layers 6 \
    --decoder-last-pipeline-num-layers 6 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 16384 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --distributed-timeout-minutes 30 \
    --num-virtual-stages-per-pipeline-rank 2
notes: |
  B200 runs of this model need `patches/fla_blackwell_guard.py`: with the image's Triton 3.6, FLA 0.4.1's autotuner
  picks gated-delta-net kernel configs that race on Blackwell and give a different, wrong result on every call, and
  the patch keeps those kernels on the configs FLA main uses there. Every training process on every node imports it
  before FLA (patches/README.md) and prints a `[fla-guard]` line.

#2attune-tp2-pp8-ep8-vpp2-mt8k1,130.7 tokens/s/GPUTP2 PP8 VPP2 EP8 DP4max 8,192 tokens/GPU, VPP 2, no offload

Author: ATTUNE (cite)
GPUs: 64 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout: TP 2, PP 8 (first and last stage 6 layers), VPP 2, CP 1, EP 8, ETP 1, DP 4, sequence parallel; one 8,192-token row per microbatch; full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher; no offload
Tokens/s per GPU: 1,130.65
$ / 1M tokens: $3.50
MFU: 4.9% (N = 16,331,407,360 active parameters per token)
Step-time CV: 0.58%
Convergence check: correlation 0.9999967, final loss difference 0.0002, 0 pointwise failures over 20 steps
Component test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an FP64 reference: B200, 2026-10-11. Shape: one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens per rank. worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit FP32 masters; the negative controls fail
Measured: 2026-10-18
GPU-hours: 28.94
Rows: qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework: slime with local patches, not pinned
megatron lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
transformer engine: 2.10.0
pytorch: 2.11.0 (CUDA 12.9 build)
cuda: 12.9
nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep: 1.2.1
python: 3.12
fla: 0.4.1 (flash-linear-attention)
triton: 3.6.0
network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
runtime fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA

Environment

CUDA_DEVICE_MAX_CONNECTIONS=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
TORCHINDUCTOR_COMPILE_THREADS=1
NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

Patches

patches/fla_blackwell_guard.py

CommandHow to run

export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

python3 -m <slime entry module> --async \
  --actor-num-nodes 8 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 2 \
  --pipeline-model-parallel-size 8 \
  --decoder-first-pipeline-num-layers 6 \
  --decoder-last-pipeline-num-layers 6 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --distributed-timeout-minutes 30 \
  --num-virtual-stages-per-pipeline-rank 2

recipes/qwen3.5-397b-a17b/b200/attune-tp2-pp8-ep8-vpp2-mt8k.yamlDownload GitHub

# recipes/qwen3.5-397b-a17b/b200/attune-tp2-pp8-ep8-vpp2-mt8k.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-397B-A17B
gpu:
  type: B200
  memory_gb: 180
  count: 64
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime with local patches, not pinned
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  fla: 0.4.1 (flash-linear-attention)
  triton: 3.6.0
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  runtime_fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log; FLA and
    Triton from an in-pod check of the same digest on 2026-10-18'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP2 PP8 VPP2 EP8 DP4
  summary: TP 2, PP 8 (first and last stage 6 layers), VPP 2, CP 1, EP 8, ETP 1, DP 4, sequence parallel; one 8,192-token
    row per microbatch; full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher;
    no offload
  highlights: max 8,192 tokens/GPU, VPP 2, no offload
  tp: 2
  pp: 8
  vpp: 2
  cp: 1
  ep: 8
  etp: 1
  dp: 4
  sequence_parallel: true
  layer_split:
    first: 6
    last: 6
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 1
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload: null
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 1130.65
  usd_per_million_tokens: 3.50
  step_time_cv_percent: 0.58
  mfu: 0.0492
  mfu_note: null
  active_params:
    count: 16331407360
    source: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
    revision: 8472618112abcbd45acbcdc58436aff4233c23f7
  measured: 2026-10-18
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 28.94
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; 1 component test passed at this model's shape
  reference: Not published.
  configuration: pass
  component_tests:
  - test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an
      FP64 reference
    shape: 'one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens
      per rank'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound
      after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit
      FP32 masters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.9999967
    final_loss_difference: 0.0002
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both; both with patches/fla_blackwell_guard.py'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches:
- patches/fla_blackwell_guard.py
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  python3 -m <slime entry module> --async \
    --actor-num-nodes 8 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 2 \
    --pipeline-model-parallel-size 8 \
    --decoder-first-pipeline-num-layers 6 \
    --decoder-last-pipeline-num-layers 6 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --distributed-timeout-minutes 30 \
    --num-virtual-stages-per-pipeline-rank 2
notes: |
  B200 runs of this model need `patches/fla_blackwell_guard.py`: with the image's Triton 3.6, FLA 0.4.1's autotuner
  picks gated-delta-net kernel configs that race on Blackwell and give a different, wrong result on every call, and
  the patch keeps those kernels on the configs FLA main uses there. Every training process on every node imports it
  before FLA (patches/README.md) and prints a `[fla-guard]` line.

#3attune-48gpu-tp2-pp6-ep81,075.5 tokens/s/GPUTP2 PP6 EP8 DP4max 8,192 tokens/GPU, no offload

Author: ATTUNE (cite)
GPUs: 48 x B200 180 GB (AWS p6-b200.48xlarge, 8 per node)
Layout: TP 2, PP 6, CP 1, EP 8, ETP 1, DP 4, sequence parallel; one 8,192-token row per microbatch; full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher; no offload
Tokens/s per GPU: 1,075.50
$ / 1M tokens: $3.68
MFU: 4.7% (N = 16,331,407,360 active parameters per token)
Step-time CV: 7.39%
Convergence check: correlation 0.9999971, final loss difference 2.6e-05, 0 pointwise failures over 20 steps
Component test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an FP64 reference: B200, 2026-10-11. Shape: one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens per rank. worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit FP32 masters; the negative controls fail
Measured: 2026-10-18
GPU-hours: 19.73
Rows: qwen35-tulu3-fixed8192-v1 (rows_sha256 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1)

Stack

framework: slime with local patches, not pinned
megatron lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
transformer engine: 2.10.0
pytorch: 2.11.0 (CUDA 12.9 build)
cuda: 12.9
nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
deepep: 1.2.1
python: 3.12
fla: 0.4.1 (flash-linear-attention)
triton: 3.6.0
network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
runtime fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA

Environment

CUDA_DEVICE_MAX_CONNECTIONS=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
TORCHINDUCTOR_COMPILE_THREADS=1
NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

Patches

patches/fla_blackwell_guard.py

CommandHow to run

export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_COMPILE_THREADS=1
export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

python3 -m <slime entry module> --async \
  --actor-num-nodes 6 \
  --actor-num-gpus-per-node 8 \
  ${MODEL_ARGS[@]} \
  --hf-checkpoint "$HF_CKPT" \
  --ref-load "$TORCH_DIST" \
  --rollout-function-path pretokenized_sft_rollout.generate_rollout \
  --prompt-data "$SFT_DATA" \
  --input-key messages \
  --metadata-key metadata \
  --num-rollout 20 \
  --rollout-batch-size 128 \
  --global-batch-size 128 \
  --loss-type sft_loss \
  --calculate-per-token-loss \
  --disable-compute-advantages-and-returns \
  --debug-train-only \
  --tensor-model-parallel-size 2 \
  --pipeline-model-parallel-size 6 \
  --context-parallel-size 1 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1 \
  --sequence-parallel \
  --recompute-granularity full \
  --recompute-method uniform \
  --recompute-num-layers 1 \
  --use-dynamic-batch-size \
  --max-tokens-per-gpu 8192 \
  --data-pad-size-multiplier 256 \
  --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
  --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
  --use-distributed-optimizer \
  --use-precision-aware-optimizer \
  --attention-dropout 0.0 --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
  --attention-backend flash \
  --distributed-timeout-minutes 30

recipes/qwen3.5-397b-a17b/b200/attune-48gpu-tp2-pp6-ep8.yamlDownload GitHub

# recipes/qwen3.5-397b-a17b/b200/attune-48gpu-tp2-pp6-ep8.yaml
# An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml.
schema_version: 1
model: Qwen3.5-397B-A17B
gpu:
  type: B200
  memory_gb: 180
  count: 48
  per_node: 8
  instance: AWS p6-b200.48xlarge
author:
  name: ATTUNE
  url: https://arxiv.org/abs/2610.17528
status: verified
stack:
  framework: slime with local patches, not pinned
  megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0)
  transformer_engine: 2.10.0
  pytorch: 2.11.0 (CUDA 12.9 build)
  cuda: '12.9'
  nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run)
  deepep: 1.2.1
  python: '3.12'
  fla: 0.4.1 (flash-linear-attention)
  triton: 3.6.0
  network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node
  runtime_fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA
  source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on
    2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log; FLA and
    Triton from an in-pod check of the same digest on 2026-10-18'
workload:
  row_tokens: 8192
  rows: SFT conversations
  global_batch: 128
  precision: bf16
  dataset:
    name: Tulu 3 SFT mixture
    source: allenai/tulu-3-sft-mixture
    revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be
    license: ODC-By 1.0
    rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1
    build:
      name: qwen35-tulu3-fixed8192-v1
      tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42)
      loss_mask: qwen3_5
      rows: 6400
      row_tokens: 8192
      padding_tokens: 0
      seed: 42
      packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut
        into consecutive 8,192-token rows; rows without supervised targets are skipped
      supervised_token_ratio: 0.676
      parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01
      parquet_bytes: 76127393
      builder: data/generate_fixed_sft.py (data/README.md)
layout:
  label: TP2 PP6 EP8 DP4
  summary: TP 2, PP 6, CP 1, EP 8, ETP 1, DP 4, sequence parallel; one 8,192-token row per microbatch; full recompute
    (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher; no offload
  highlights: max 8,192 tokens/GPU, no offload
  tp: 2
  pp: 6
  vpp: 1
  cp: 1
  ep: 8
  etp: 1
  dp: 4
  sequence_parallel: true
  layer_split: null
  max_tokens_per_gpu: 8192
  rows_per_microbatch: 1
  recompute:
    granularity: full
    method: uniform
    num_layers: 1
  offload: null
  dispatcher: alltoall
result:
  tokens_per_s_per_gpu: 1075.50
  usd_per_million_tokens: 3.68
  step_time_cv_percent: 7.39
  mfu: 0.0468
  mfu_note: null
  active_params:
    count: 16331407360
    source: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
    revision: 8472618112abcbd45acbcdc58436aff4233c23f7
  measured: 2026-10-18
  steps_per_run: 20
  window:
  - 5
  - 19
  window_tokens: 15728640
  runs: 1
  profiler: false
  gpu_hours: 19.73
  gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the
    convergence check
correctness:
  result: pass
  summary: passed; 1 component test passed at this model's shape
  reference: Not published.
  configuration: pass
  component_tests:
  - test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an
      FP64 reference
    shape: 'one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens
      per rank'
    gpu: B200
    date: 2026-10-11
    result: pass
    detail: worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound
      after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit
      FP32 masters; the negative controls fail
  convergence:
    steps: 20
    result: pass
    correlation: 0.9999971
    final_loss_difference: 2.6e-05
    pointwise_failures: 0
    note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and
      rows; the parity probe off in both; both with patches/fla_blackwell_guard.py'
  rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge
    validate_convergence with its pipeline thresholds
env:
  CUDA_DEVICE_MAX_CONNECTIONS: '1'
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
  TORCHINDUCTOR_COMPILE_THREADS: '1'
  NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)
patches:
- patches/fla_blackwell_guard.py
run: |
  export CUDA_DEVICE_MAX_CONNECTIONS=1
  export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  export TORCHINDUCTOR_COMPILE_THREADS=1
  export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0)

  python3 -m <slime entry module> --async \
    --actor-num-nodes 6 \
    --actor-num-gpus-per-node 8 \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint "$HF_CKPT" \
    --ref-load "$TORCH_DIST" \
    --rollout-function-path pretokenized_sft_rollout.generate_rollout \
    --prompt-data "$SFT_DATA" \
    --input-key messages \
    --metadata-key metadata \
    --num-rollout 20 \
    --rollout-batch-size 128 \
    --global-batch-size 128 \
    --loss-type sft_loss \
    --calculate-per-token-loss \
    --disable-compute-advantages-and-returns \
    --debug-train-only \
    --tensor-model-parallel-size 2 \
    --pipeline-model-parallel-size 6 \
    --context-parallel-size 1 \
    --expert-model-parallel-size 8 \
    --expert-tensor-parallel-size 1 \
    --sequence-parallel \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1 \
    --use-dynamic-batch-size \
    --max-tokens-per-gpu 8192 \
    --data-pad-size-multiplier 256 \
    --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \
    --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \
    --use-distributed-optimizer \
    --use-precision-aware-optimizer \
    --attention-dropout 0.0 --hidden-dropout 0.0 \
    --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \
    --attention-backend flash \
    --distributed-timeout-minutes 30
notes: |
  B200 runs of this model need `patches/fla_blackwell_guard.py`: with the image's Triton 3.6, FLA 0.4.1's autotuner
  picks gated-delta-net kernel configs that race on Blackwell and give a different, wrong result on every call, and
  the patch keeps those kernels on the configs FLA main uses there. Every training process on every node imports it
  before FLA (patches/README.md) and prints a `[fla-guard]` line.

Citation

If you use a recipe by ATTUNE, please cite the paper.

BibTeX

@article{fan2026attune,
  title   = {{ATTUNE: Agentic Throughput Tuning for LLM Training}},
  author  = {Leo Fan and Hongyang Wang and Wenbo Chen and Jingguo Liu and Xian Li and Xuan Wang and
             Ruochen Wen and Yiming Yan and Hao Wu and Zheshen Wang and Benoit Dumoulin},
  journal = {arXiv preprint arXiv:2610.17528},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.17528}
}

Qwen3.5-397B-A17B on other GPUs

H100 · 128 GPUs

313.3

tokens/s/GPU · $6.10 / 1M tokens

H200 · 128 GPUs

545.3

tokens/s/GPU · $4.03 / 1M tokens

B200 · 64 GPUs · this page

1,232.5

tokens/s/GPU · $3.21 / 1M tokens