Protocol

Every recipe is measured one way, so recipes for the same model and GPU type, a pair, compare directly. A pair can hold recipes from several authors; they rank by tokens/s per GPU.

Workload

The rows are SFT conversations, tokenized with the model's own tokenizer and packed to exactly 8,192 tokens each with no padding: a seeded permutation of the source conversations, joined with an EOS token between conversations and cut into consecutive rows. A step trains on a global batch of 128 rows in BF16, and every run of a model reads the same rows in the same order.

The recipes use the Tulu 3 SFT mixture (allenai/tulu-3-sft-mixture, revision b14afda60f1bbebe55d5d2fa1e4df5042f97f8be, ODC-By 1.0), built once per tokenizer. data/ rebuilds the rows, and data/verify_rows.py checks a build against the rows_sha256 a recipe records. Some subsets of the mixture carry their own licenses and terms, including non-commercial ones (see the dataset card). This repository does not redistribute the data.

A recipe may run on any number of GPUs of its type and records the count.

Measurement

A run lasts 20 optimizer steps (0–19) with the profiler off. Steps 0–4 absorb startup, compilation and allocator warm-up and are discarded; steps 5–19 form the measured window W. The score is the aggregate throughput over the window, summed before dividing:

tokens/s/GPU = \frac{\sum_{i \in W} T_{i}}{n_{GPU} \times \sum_{i \in W} t_{i}}

For each step i in the measured window W, T_i is the number of tokens and t_i is the step time in seconds; n_GPU is the number of GPUs.

A recipe is one run. It reports the score and the coefficient of variation of the step time over W. The same 20-step run is the recipe's arm of the convergence check below, so one run covers both. A run that also needs a profiler trace continues past step 19: step 20 sets the profiler up, step 21 warms it up and step 22 (up to 24) is recorded. Those steps are never scored, and a run that long is not a convergence arm, because the run length can change the learning-rate schedule. 7 recipes, measured on 2026-10-11, ran 14 steps under an earlier window and are scored over steps 4–13; they record steps_per_run and their window, and their convergence checks ran separate 20-step arms.

GPU-hours count GPUs times the time from scheduler admission to finish, including startup.

Correctness

A faster recipe is listed once it passes two checks against its reference recipe, the recipe it was tuned from. The configuration check diffs the recipe's resolved arguments against the reference recipe's and classes every changed argument: structural changes (placement, packing) and same-math swaps (allowlisted) are eligible, and every same-math swap needs a passed component test at the model's shape, which records what it compares, the layer shape, the bound and the worst error.

The convergence check compares the per-step loss of a 20-step arm of the recipe with a 20-step arm of the reference recipe on the same image and rows, with Megatron-Bridge's validate_convergence and its pipeline thresholds. A recipe records the verdict, the correlation of the two loss curves, the final loss difference and the number of pointwise failures. The check compares 20 steps, so it catches changes that move the loss curve early; a change whose effect shows only later in training can pass it.

The measured run serves as the recipe's arm when nothing in it adds work to the step. A reference arm serves every recipe of its pair that runs on the same image and rows. Publishing the reference recipe is optional; a recipe states it when it is public.

Objective conventions. Qwen3.5 and Qwen3.6 recipes train with the load-balancing loss off (--moe-aux-loss-coeff 0), in the reference recipe and the recipe alike. Megatron computes that loss per microbatch, so with the loss on, the number of rows per microbatch would be part of the objective.

Cost

$ per 1M tokens = \frac{per-GPU hourly rate}{tokens/s/GPU \times 3,600} \times 10^{6}

The rates are AWS EC2 on-demand, us-east-1, instance price / 8 GPUs, fetched 2026-09-18, from gpus.yaml. Prices change; recompute for your region and commitment. Within a pair every recipe uses the same rate, so the fastest recipe is also the cheapest; each model's pages compare its GPU types on both.

Type	GPU	Instance	$ / GPU-hour	Dense BF16 peak, TFLOP/s
H100	NVIDIA H100 80GB	AWS p5.48xlarge	$6.88	989	source
H200	NVIDIA H200 141GB	AWS p5en.48xlarge	$7.912	989	source
B200	NVIDIA B200 180GB	AWS p6-b200.48xlarge	$14.2416	2,250	source

MFU

MFU = \frac{6 \times N \times tokens/s/GPU}{peak FLOPs/s}

N is the model's active parameters per token: every weight the text model applies to each token in a matrix multiplication (attention, linear attention, sparse-attention indexer, dense and shared-expert MLPs, routers and the output projection), with routed experts counted at top-k of the expert count. N leaves out the input embedding (a lookup), norms and biases, position embeddings and integer index tables, multi-token prediction layers and vision towers; FP4 weights stored two per byte count at their unpacked size. tools/active_params.py counts N from the checkpoint's safetensors headers, and a recipe records N with the checkpoint and its revision. The peak is the GPU's dense BF16 peak in the table above, converted from TFLOP/s to FLOP/s. Attention-score FLOPs and recomputation are not counted, so MFU compares models with softmax, linear and sparse attention on one basis and sits below the hardware's actual FLOP rate.

MFU on this workload is low for large mixture-of-experts models, and it reflects the workload more than the recipe. A step of 128 rows of 8,192 tokens is about 1.05 million tokens. The recipes run it on 8 to 256 GPUs, which leaves each GPU between 4K and 128K tokens per step; the large mixture-of-experts recipes sit at the low end, so pipeline bubbles, the expert all-to-all and, where memory forces it, the CPU optimizer step take a large share of every step. Compare MFU between recipes of one pair, not with large-batch pretraining.

Stack

A recipe records the versions it ran on: the framework (slime or miles, with its commit), Megatron-LM, Transformer Engine, PyTorch, CUDA and NCCL, plus any communication library it uses (for example DeepEP), the GPUs per node and the inter-node network. A version the run's evidence does not show reads not pinned.

Running a recipe

A recipe's command is the training command as run, with paths replaced by variables: $HF_CKPT is the model's Hugging Face checkpoint, $TORCH_DIST the Megatron torch_dist weights the run loads and $SFT_DATA the rows. <slime entry module> stands for the training image's slime entry point, and MODEL_ARGS come from the framework's model script for the model (scripts/models/ in slime) unless the model's notes say otherwise. A model's notes, on each of its pages, give what all of its recipes need.

Build the rows a recipe names with data/ and check them with python3 data/verify_rows.py <rows.parquet> --recipe <recipe file>. Put data/ on PYTHONPATH so --rollout-function-path pretokenized_sft_rollout.generate_rollout resolves, and patches/ too when the recipe lists patches.

For another GPU count, change DP, the number of replicas; a replica takes TP x PP x CP GPUs. The global batch of 128 rows must divide by DP, and the GPUs per pipeline stage (GPU count / PP) must stay a multiple of EP x ETP. For another row length, set AUTOTUNE_SEQ_LEN to it and size --max-tokens-per-gpu x CP to hold at least one row. Re-measure after any change, and re-run the convergence check after any change beyond the parallel layout and packing.

Status and authors

A verified recipe was measured as above and passed both checks; OpenRecipes lists verified recipes only. Each recipe names its author: the GitHub handle of the person or team that submitted it, or a name and a link for a tool.