# recipes/qwen3.5-397b-a17b/b200/attune-48gpu-tp2-pp6-ep8.yaml # An OpenRecipes recipe; the fields are documented in schema/recipe.schema.yaml. schema_version: 1 model: Qwen3.5-397B-A17B gpu: type: B200 memory_gb: 180 count: 48 per_node: 8 instance: AWS p6-b200.48xlarge author: name: ATTUNE url: https://arxiv.org/abs/2610.17528 status: verified stack: framework: slime with local patches, not pinned megatron_lm: commit 1dcf0daf (megatron-core 0.16.0rc0) transformer_engine: 2.10.0 pytorch: 2.11.0 (CUDA 12.9 build) cuda: '12.9' nccl: 2.28.9 (CUDA 12.9 build; printed by the measured run) deepep: 1.2.1 python: '3.12' fla: 0.4.1 (flash-linear-attention) triton: 3.6.0 network: 8 GPUs per node, EFA, 3.2 Tb/s RDMA per node runtime_fixes: patches/fla_blackwell_guard.py, imported in every training process before FLA source: 'in-pod environment dump of the measured run''s training image (same digest), taken by the component tests on 2026-10-11: installed TE, PyTorch and DeepEP, Megatron-LM git commit; NCCL from the measured run''s log; FLA and Triton from an in-pod check of the same digest on 2026-10-18' workload: row_tokens: 8192 rows: SFT conversations global_batch: 128 precision: bf16 dataset: name: Tulu 3 SFT mixture source: allenai/tulu-3-sft-mixture revision: b14afda60f1bbebe55d5d2fa1e4df5042f97f8be license: ODC-By 1.0 rows_sha256: 7a837641dcde9ebcac9d3ed5ed84ddd941fc9bfc173606cc701e9a3f51c307f1 build: name: qwen35-tulu3-fixed8192-v1 tokenizer: Qwen3.5 (the Qwen3.5-122B-A10B tokenizer; tokenizer.json sha256 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42) loss_mask: qwen3_5 rows: 6400 row_tokens: 8192 padding_tokens: 0 seed: 42 packing: source conversations in a seeded global permutation, concatenated with an EOS token between them and cut into consecutive 8,192-token rows; rows without supervised targets are skipped supervised_token_ratio: 0.676 parquet_sha256: f215bdc6aba60d6d0dff4471eaef805765cb393e40b95762190535376491ab01 parquet_bytes: 76127393 builder: data/generate_fixed_sft.py (data/README.md) layout: label: TP2 PP6 EP8 DP4 summary: TP 2, PP 6, CP 1, EP 8, ETP 1, DP 4, sequence parallel; one 8,192-token row per microbatch; full recompute (uniform, 1 layer); precision-aware distributed optimizer; alltoall dispatcher; no offload highlights: max 8,192 tokens/GPU, no offload tp: 2 pp: 6 vpp: 1 cp: 1 ep: 8 etp: 1 dp: 4 sequence_parallel: true layer_split: null max_tokens_per_gpu: 8192 rows_per_microbatch: 1 recompute: granularity: full method: uniform num_layers: 1 offload: null dispatcher: alltoall result: tokens_per_s_per_gpu: 1075.50 usd_per_million_tokens: 3.68 step_time_cv_percent: 7.39 mfu: 0.0468 mfu_note: null active_params: count: 16331407360 source: https://huggingface.co/Qwen/Qwen3.5-397B-A17B revision: 8472618112abcbd45acbcdc58436aff4233c23f7 measured: 2026-10-18 steps_per_run: 20 window: - 5 - 19 window_tokens: 15728640 runs: 1 profiler: false gpu_hours: 19.73 gpu_hours_note: GPUs x (scheduler admission to finish), including startup; the same run is the candidate arm of the convergence check correctness: result: pass summary: passed; 1 component test passed at this model's shape reference: Not published. configuration: pass component_tests: - test: precision-aware Adam on GPU (TE FusedAdam, FP32 masters stored as BF16 parameter plus remainder) against an FP64 reference shape: 'one Qwen3.5-397B-A17B MoE layer: hidden 4096, 512 experts, top-10, expert FFN 1024, EP 8, ETP 1, 4,096 tokens per rank' gpu: B200 date: 2026-10-11 result: pass detail: worst tensor (update) relative L2 error 5.12e-7 against a bound of 1.0e-3 after one step, and within bound after three steps at the entry's hyperparameters; reconstructed FP32 masters bitwise identical to an arm with explicit FP32 masters; the negative controls fail convergence: steps: 20 result: pass correlation: 0.9999971 final_loss_difference: 2.6e-05 pointwise_failures: 0 note: 'candidate arm: the measured run; reference arm: a 20-step run of the reference recipe on the same image and rows; the parity probe off in both; both with patches/fla_blackwell_guard.py' rules: argument classes of ATTUNE's skill/scripts/correctness/rules/numerics_list.yaml; 20-step check by Megatron-Bridge validate_convergence with its pipeline thresholds env: CUDA_DEVICE_MAX_CONNECTIONS: '1' PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True TORCHINDUCTOR_COMPILE_THREADS: '1' NCCL_NVLS_ENABLE: $([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0) patches: - patches/fla_blackwell_guard.py run: | export CUDA_DEVICE_MAX_CONNECTIONS=1 export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True export TORCHINDUCTOR_COMPILE_THREADS=1 export NCCL_NVLS_ENABLE=$([ "$NVLINK_COUNT" -gt 0 ] && echo 1 || echo 0) python3 -m --async \ --actor-num-nodes 6 \ --actor-num-gpus-per-node 8 \ ${MODEL_ARGS[@]} \ --hf-checkpoint "$HF_CKPT" \ --ref-load "$TORCH_DIST" \ --rollout-function-path pretokenized_sft_rollout.generate_rollout \ --prompt-data "$SFT_DATA" \ --input-key messages \ --metadata-key metadata \ --num-rollout 20 \ --rollout-batch-size 128 \ --global-batch-size 128 \ --loss-type sft_loss \ --calculate-per-token-loss \ --disable-compute-advantages-and-returns \ --debug-train-only \ --tensor-model-parallel-size 2 \ --pipeline-model-parallel-size 6 \ --context-parallel-size 1 \ --expert-model-parallel-size 8 \ --expert-tensor-parallel-size 1 \ --sequence-parallel \ --recompute-granularity full \ --recompute-method uniform \ --recompute-num-layers 1 \ --use-dynamic-batch-size \ --max-tokens-per-gpu 8192 \ --data-pad-size-multiplier 256 \ --optimizer adam --lr 1e-5 --lr-decay-style cosine --min-lr 1e-6 \ --lr-warmup-fraction 0.1 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 \ --use-distributed-optimizer \ --use-precision-aware-optimizer \ --attention-dropout 0.0 --hidden-dropout 0.0 \ --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 \ --attention-backend flash \ --distributed-timeout-minutes 30 notes: | B200 runs of this model need `patches/fla_blackwell_guard.py`: with the image's Triton 3.6, FLA 0.4.1's autotuner picks gated-delta-net kernel configs that race on Blackwell and give a different, wrong result on every call, and the patch keeps those kernels on the configs FLA main uses there. Every training process on every node imports it before FLA (patches/README.md) and prints a `[fla-guard]` line.