# Eight files, six axes, one session — and the two axes we lose on every file

- **Canonical URL:** https://www.probackend.com/developer-platforms-tooling/eight-files-six-axes-one-session-and-the-two-axes-we-lose
- **Section:** News & Analysis
- **Published:** 2026-11-14
- **Summary:** The claim under test was that the engine is at or above llama.cpp on every axis, for every model it can run. This is the session that tested it across…

---

The claim was simple enough to write down: **at or above llama.cpp on every
axis, for every model we can run.** This is the run that tested it across a
whole model directory, and the run that refuted it.

The refutation is not a collapse. Prefill at 512 tokens is faster on 8 of 8
files, by +7.71% to +58.33%. Eight-slot serve is faster on 7 of 8, by +4.71% to
+18.64%, with the eighth gated out. Perplexity on a fixed corpus agrees with
`llama-perplexity` to within 0.23% on every comparable file, and repeated runs
of the same file return the same number to four decimals. Peak VRAM is higher
than llama.cpp on 8 of 8 files, by +10.09% to +13.52%. Decode is not a win:
against llama.cpp's faster backend the engine is below on 6 of 8 files at depth
0 and on 8 of 8 at 8,192.

The VRAM column is the verdict. A claim that says "at or above on every axis"
fails on the axis where the engine does not win a single cell, and no amount of
prefill throughput answers it. The rest of this article is the table that says
so, the losses named with their margins, and the one result that most articles
would leave out: the gate that decides which rows are allowed to be compared is
shallower than it looks.

**Correction, 14 November.** The first version of this table compared the engine
against llama.cpp's ROCm build only, and reported decode as a win on 8 of 8. On
this card llama.cpp also has a Vulkan build, and on decode it is 2.07% to 5.05%
faster than the ROCm build at both depths, outside every cell's spread. A reader
choosing an engine can choose a backend, so the decode and serve columns below
are now measured against the faster of the two, and each cell says which one it
used. Prefill is unchanged, because there the ROCm build is the faster one by
13.69% to 33.48%; quality and VRAM are unchanged.


## Sixteen files, eight of which run

`hipml load` is not a runnability test. Four of the sixteen `.gguf` files in the
model directory upload every tensor, report `unsupported: none`, and then fail
to produce a token. Any attempt table built on `load` alone overstates the
runnable set by half.

| # | file | GiB | `hipml load` | `hipml generate` | verdict |
| ---: | --- | ---: | --- | --- | --- |
| 1 | `Arete-27B-Q4_K_M.gguf` | 16.02 | ok | ok | **loads+generates** |
| 2 | `Flint-GGUF-Serve-mixed-4-8bit.gguf` | 23.29 | fail | fail | load fails |
| 3 | `MiniCPM-V-4.7-35B-A3B-GGUF-Serve-mixed-4-8bit.gguf` | 20.62 | ok | fail | loads, generate fails |
| 4 | `MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf` | 19.71 | ok | fail | loads, generate fails |
| 5 | `OrcaCyber-Nano-1.0-AI-Defense-DFlash2-Q8_0.gguf` | 1.92 | fail | fail | load fails |
| 6 | `OrcaCyber-Nano-1.0-AI-Defense-Mix-15.7G.gguf` | 14.62 | ok | ok | **loads+generates** |
| 7 | `Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf` | 21.28 | ok | fail | loads, generate fails |
| 8 | `Qwen3.8-27B-Q4_K_M.gguf` | 15.66 | ok | ok | **loads+generates** |
| 9 | `Qwen3.8-27B-UD-Q4_K_M.gguf` | 15.43 | ok | ok | **loads+generates** |
| 10 | `Qwen3.8-27B-UD-Q4_K_XL.gguf` | 16.35 | ok | ok | **loads+generates** |
| 11 | `Qwen3.8-27B-UD-Q6_K.gguf` | 20.47 | ok | ok | **loads+generates** |
| 12 | `Qwen3.8-27B-UD-Q6_K_XL.gguf` | 23.56 | ok | ok | **loads+generates** |
| 13 | `Sage-A1.5-Q4_K_M.gguf` | 20.22 | ok | fail | loads, generate fails |
| 14 | `Swift-1.5-Qwen3.8-27B-Q4_K_M.gguf` | 16.24 | ok | ok | **loads+generates** |
| 15 | `dspark-MiniCPM-V-4_7-35B-A3B-Q8_0.gguf` | 0.40 | fail | fail | load fails |
| 16 | `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.28 | fail | — | sidecar |

Eight run, four load and cannot generate, three fail to load, one is a sidecar.
The exact errors are worth reading, because they name four different gaps, and
only one of them is a file that was never meant to run alone:

| file | verdict | exact error |
| --- | --- | --- |
| `Flint-GGUF-Serve-mixed-4-8bit.gguf` | load fails | `load failed: config: GGUF metadata key flint.rope.dimension_count is missing` |
| `MiniCPM-V-4.7-35B-A3B-GGUF-Serve-mixed-4-8bit.gguf` | generate fails | `generate: graph: token_embd.weight: expected Q4_K or Q6_K, found Q8_0` |
| `MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf` | generate fails | `generate: graph: model has no tensor blk.0.ffn_gate.weight` |
| `OrcaCyber-Nano-1.0-AI-Defense-DFlash2-Q8_0.gguf` | load fails | `load failed: config: GGUF metadata key dflash.rope.dimension_count is missing` |
| `Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf` | generate fails | `generate: graph: token_embd.weight: expected Q4_K or Q6_K, found Q8_0` |
| `Sage-A1.5-Q4_K_M.gguf` | generate fails | `generate: graph: model has no tensor blk.0.ffn_gate.weight` |
| `dspark-MiniCPM-V-4_7-35B-A3B-Q8_0.gguf` | load fails | `load failed: config: GGUF metadata key dflash.rope.dimension_count is missing` |
| `mtp-Qwen3.8-27B-Q4_0.gguf` | sidecar | `load failed: output.weight: Q3_K has no GEMV entry` |

Two files carry a Q8_0 embedding table and the graph accepts only Q4_K or Q6_K
there. Two declare an FFN without `blk.0.ffn_gate.weight`, a fused layout the
graph does not build. Three are missing a RoPE metadata key under a vendor
prefix the config reader does not know. The sidecar is a drafter, not a model:
it has no output tensor in a type the GEMV path takes, and it is never expected
to run alone. None of these is a speed result; they are the boundary of what
"every model we can run" means.

An earlier recon list named eight runnable files. The count was right and the
membership was not: one file on that list fails to load, one file that runs was
not on it, and the four files that load and cannot generate were on neither
list. A count is not a set.

## The method, and why the numbers are comparable

**One hold, one binary, one card.** The GPU lock was acquired at 00:15:46Z and
released at 02:23:00Z, 127 minutes later, with no lost-lock event. Every number
below came from a binary of 3,044,864 bytes, sha256 `ef5b66bd…`, built from tree
`433bfbe`, on one Radeon AI PRO R9700 reporting 31.9 GiB. The reference is
llama.cpp's ROCm build, 11820, commit `d7725699`, byte-identical to the build
the previous session used. There is a second llama.cpp build on the same card —
the Vulkan one, 11836, commit `f5027fb21` — and the method below explains why
the baseline is the faster of the two rather than the one the session happened
to run.

**The same file, prompt and corpus on both engines.** Both engines were given
the same `.gguf` path literally on the command line, not through an environment
variable. The timed CLI cells generate 33 tokens from the same 169-id fixture.
The quality cells score the same 8,184-token corpus: 28,035 bytes, 15 windows of
512 tokens, 3,825 scored positions. `llama-tokenize` read the same text and
agreed with the id stream the engine scored, 8,184 tokens on both.

**Interleaved, not blocked.** Within each file the two CLI processes ran in
opposite order — hipml-then-llama in one, llama-then-hipml in the other — and no
file was interleaved with another inside a stage. The llama.cpp cells are not a
block measured after the engine was finished for the day.

**Three reps, a median and a spread.** Every CLI cell in the session is three
reps in each of two processes, so six values; every serve cell is six waves. The
backend check that supplies the decode and serve baselines is one process per
cell, one discarded warm-up request and three timed reps, and three waves for
serve. The number quoted is the median of the cell, and the spread is max−min as
a percentage of that median. Twenty-four cells exceed 3%: all sixteen
prefill-at-512 cells, seven of the eight llama.cpp serve cells, and one hipml
serve cell, the gated file. The matrix marks those columns `†`. The worst are
llama.cpp's prefill at 512 tokens, which ranges from 6.16% to 26.37% across
files because its first request of each process is cold.

**llama.cpp is two builds on this card, and the baseline is the faster one.**
Both builds take the flags the matrix uses — `-ctk q8_0 -ctv q8_0 -ngl 99 -c …
--parallel 8` — and both see the same 31.9 GiB card. They are not the same
engine. Vulkan is **2.07% to 5.05% faster on decode** at both depths and
**13.69% to 33.48% slower on prefill**, so the faster backend is not the same
one on every axis, and "llama.cpp" without a backend is an ambiguous baseline.

The decode and serve columns below therefore use the faster backend per cell,
and the cell carries the mark: `r` for the ROCm build, `v` for the Vulkan build.
Prefill stays on ROCm, where the ROCm build is the faster one, and the session's
prefill numbers are unchanged. The check was one hold, one process per cell, the
same fixture ids, q8 KV on both engines, `-ngl 99`, one warm-up request
discarded, then three timed reps. The warm-up is not decoration: the previous
slice measured a 257 ms cold first request on this box, which is what made the
session's prefill-at-512 spreads as wide as 26%. The Vulkan build was pinned to
the R9700 with `-dev Vulkan1`, because it enumerates the integrated GPU first
and the default would have measured the wrong device.

hipml was re-measured in the same hold on three of the eight files as a control.
Its decode agreed with the table to 0.04%–0.86% and its serve to 0.17%–0.99%, so
the engine column stands and only the baseline moved.

**VRAM measured from outside both engines.** Peak memory is WDDM dedicated
usage for the whole process, sampled by the same external PowerShell script for
both engines. That choice is the difference between a comparison and two
self-reports: each engine accounts for its own allocations in its own way, so a
number printed by the engine measures that engine's bookkeeping, not the memory
the card gave it. An external sample is the same quantity for both. The two
samples per CLI cell agree to within 0.01 MiB, so the memory axis has no
measurement noise at all.

The session also has a reproducibility check built in. The same file, harness
and llama.cpp binary were measured 4.7 hours earlier on a different tree. Every
one of the six cells moved by less than 1.4%: prefill at 512 by −1.37%, prefill
at 8,192 by −0.06%, decode at depth 0 by −0.16%, decode at 8,192 by +0.08%,
serve by +0.02%, peak VRAM by 0.00%. The engine changed; the measurement did
not.

## The matrix

`Δ` is (hipml − llama.cpp) on the medians, so a negative number means the engine
is below. Each cell is hipml / llama.cpp · Δ, and the llama.cpp number is the
faster of its two backends on that cell, marked `r` (ROCm) or `v` (Vulkan). File
names are shortened; every row is the `.gguf` named in the attempt table. `†`
marks a column that contains a cell whose spread exceeds 3% of its median.

| file | prefill @512 † | prefill @8,192 | decode @0 | decode @8,192 | serve b † | peak VRAM |
| --- | --- | --- | --- | --- | --- | --- |
| `Arete-27B-Q4_K_M` | 1,417.50 / 1,066.24 r · +32.94% | 1,367.85 / 1,185.93 r · +15.34% | 31.26 / 31.49 v · −0.75% | 30.20 / 30.91 v · −2.27% | 133.98 / 126.05 v · +6.29% | 25,003 / 22,490 · +11.18% |
| `OrcaCyber-…-Mix-15.7G` | 1,326.45 / 1,231.50 r · +7.71% | 1,325.76 / 1,376.98 r · −3.72% | 33.80 / 34.30 v · −1.45% | 32.60 / 33.45 v · −2.56% | 127.93 / 120.07 r · +6.55% | 23,504 / 20,705 · +13.52% |
| `Qwen3.8-27B-Q4_K_M` | 1,362.67 / 1,038.76 r · +31.18% | 1,351.40 / 1,160.14 r · +16.49% | 32.41 / 32.75 v · −1.02% | 31.29 / 32.07 v · −2.43% | 129.00 / 123.19 v · +4.71% | 24,571 / 21,736 · +13.05% |
| `Qwen3.8-27B-UD-Q4_K_M` | 1,335.29 / 1,203.89 r · +10.91% | 1,327.13 / 1,351.43 r · −1.80% | 33.16 / 32.92 v · +0.74% | 32.02 / 32.25 v · −0.70% | 131.99 / 114.56 r · +15.21% | 24,288 / 21,429 · +13.34% |
| `Qwen3.8-27B-UD-Q4_K_XL` | 1,334.29 / 1,133.48 r · +17.72% | 1,319.83 / 1,272.25 r · +3.74% | 31.54 / 31.62 v · −0.25% | 30.51 / 30.81 v · −0.95% | 131.56 / 110.89 v · +18.64% | 25,263 / 22,367 · +12.94% |
| `Qwen3.8-27B-UD-Q6_K` | 1,304.31 / 823.80 r · +58.33% | 1,292.58 / 904.63 r · +42.88% | 26.07 / 26.08 v · −0.03% | 25.33 / 25.61 v · −1.11% | 119.95 / 101.65 v · +18.01% | 29,500 / 26,279 · +12.26% |
| `Qwen3.8-27B-UD-Q6_K_XL` | 1,318.80 / 884.78 r · +49.05% | 1,320.77 / 990.30 r · +33.37% | 23.01 / 22.92 v · +0.39% | 22.45 / 22.56 v · −0.49% | 68.90 / 96.86 v · −28.87% | 32,436 / 29,462 · +10.09% |
| `Swift-1.5-Qwen3.8-27B-Q4_K_M` | 1,399.81 / 1,054.35 r · +32.77% | 1,361.03 / 1,183.00 r · +15.05% | 31.16 / 31.60 v · −1.39% | 30.13 / 31.04 v · −2.95% | 132.43 / 125.97 v · +5.13% | 25,226 / 22,477 · +12.23% |

Read by column, the six axes say six different things.

**Prefill at 512 is the widest win and the least clean measurement.** The engine
wins on 8 of 8, from +7.71% to +58.33%. Every one of those cells is marked `†`
on both sides: the engine's own spreads are 3.16% to 4.12%, and llama.cpp's are
6.16% to 26.37% because its first request in each process pays a cold-start
cost. The medians separate anyway, by 95 to 481 tokens per second, but the
honest reading is that the engine's prefill is stable and llama.cpp's is not.

**Prefill at 8,192 is where the range collapses.** The engine wins 6 of 8, from
+3.74% to +42.88%, and loses 2. Both losses are below, with their margins. The
larger point is what the column does *not* show: across the seven 27B files the
engine spans 1,292.58 to 1,367.85 tok/s at this depth — 5.7% of its own median —
while llama.cpp spans 904.63 to 1,351.43, 37.8% of its median. The engine's
prefill does not care which quant the file is. llama.cpp's does, and the biggest
wins in the column (+58.33% and +42.88%) are not the engine getting faster; they
are llama.cpp falling to 823.80 and 904.63 on the two Q6 files.

**Decode at depth 0 is not a win once the baseline is the faster backend.**
Against the ROCm build it was 8 of 8, +0.63% to +3.02%. Against the Vulkan
build it is 2 of 8: `UD-Q4_K_M` at +0.74% (33.16 against 32.92 tok/s) and
`UD-Q6_K_XL` at +0.39% (23.01 against 22.92), with `UD-Q6_K` level at −0.03%.
The other five are below, from −0.25% to −1.45%, which is −0.08 to −0.50 tokens
per second. The whole column spans 0.74 tok/s, so this is a near-tie either
way. What it is not is a win, and the previous version of this table said it was
because it compared against the slower backend.

**Decode at 8,192 is below on 8 of 8.** −0.49% to −2.95%, or −0.11 to −0.92
tokens per second, on cell spreads of 0.06% to 0.65%. The Vulkan build is 2.11%
to 5.05% faster than the ROCm build at this depth on every file, and that gap is
larger than every cell's spread, so the sign is not in doubt. The engine loses
2.4% to 3.6% of its own decode between the two depths; the Vulkan build loses
1.6% to 2.6%, so the gap widens with context rather than closing.

**Eight-slot serve is still a win, and a much narrower one.** 7 of 8, +4.71% to
+18.64% on aggregate goodput, which is 5.81 to 20.67 tokens per second across
eight streams. The eighth is the gated file at −28.87%, and it is below. The
Vulkan build is the faster serve backend on six of the eight files — by 2.88% on
`UD-Q6_K_XL`, 7.33% on `UD-Q6_K`, 11.71% on `UD-Q4_K_XL`, 25.61% on
`Qwen3.8-27B-Q4_K_M`, 27.64% on `Arete` and 39.91% on `Swift` — and the ROCm
build is faster on two, `OrcaCyber` by 28.74% and `UD-Q4_K_M` by 7.92%. Those
two rows keep the ROCm number. The previous version of this column quoted
+6.55% to +38.84%; the top of that range was `Arete` against the ROCm build, and
against Vulkan the same cell is +6.29%.

The same cell records median time to first token, for the same server the
throughput baseline uses: lower on 6 of 8, by 0.82% to 38.22%, and higher on two
— +21.55% on `Qwen3.8-27B-Q4_K_M` and +18.61% on the gated file. Against the
ROCm build it was lower on 7 of 8. The Vulkan build's prefill is the slow part
of that server, and the latency no longer tracks the throughput on every row.

**Peak VRAM is lost on 8 of 8.** +10.09% to +13.52%, which is 2,514 to 3,221
MiB on the files where both engines ran. There is no cell in the session where
the engine peaks lower, and the axis has no measurement noise to explain it
away.

## Quality: a tie, and a quant ladder that is nearly flat

Perplexity is one number on one 28 KB corpus, so it is a narrow quality check
rather than a benchmark suite. It is also a deterministic sum, which makes it a
clean comparison: repeated runs of the same file and variant returned the same
value to four decimals, and the largest spread over any file and variant was
0.0000.

| file | GiB | hipml ppl | llama ppl (`-fa off`, f32 KV) | Δ | llama ppl (default) | Δ |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `Arete-27B-Q4_K_M` | 16.02 | 2.0384 | 2.0410 | −0.13% | 2.0399 | −0.07% |
| `OrcaCyber-…-Mix-15.7G` | 14.62 | 2.0497 | 2.0505 | −0.04% | 2.0490 | +0.03% |
| `Qwen3.8-27B-Q4_K_M` | 15.66 | 2.0353 | 2.0368 | −0.07% | 2.0377 | −0.12% |
| `Qwen3.8-27B-UD-Q4_K_M` | 15.43 | 2.0291 | 2.0266 | +0.12% | 2.0295 | −0.02% |
| `Qwen3.8-27B-UD-Q4_K_XL` | 16.35 | 2.0353 | 2.0363 | −0.05% | 2.0399 | −0.23% |
| `Qwen3.8-27B-UD-Q6_K` | 20.47 | 2.0238 | 2.0277 | −0.19% | 2.0247 | −0.04% |
| `Qwen3.8-27B-UD-Q6_K_XL` | 23.56 | 2.0235 | 2.0242 | −0.03% | 2.0268 | −0.16% |
| `Swift-1.5-Qwen3.8-27B-Q4_K_M` | 16.24 | 2.0495 | 2.0507 | −0.06% | 2.0508 | −0.06% |

The largest disagreement on either llama.cpp configuration is 0.23%, on
`UD-Q4_K_XL` against llama.cpp's default f16 KV, and the largest on the f32-KV
configuration is 0.19%. On the seven files that pass the gate, the two engines
are not distinguishable on perplexity. The eighth is gated out of the speed
comparison and its perplexity agrees to 0.035%; that row is discussed below.

The useful finding falls out of the same table. The 27B family runs from 15.43
to 23.56 GiB and buys very little quality for the size:

| step | weights | hipml ppl | decode @0 | peak VRAM @8,192 |
| --- | ---: | ---: | ---: | ---: |
| `Qwen3.8-27B-Q4_K_M` | 15.66 GiB | 2.0353 | 32.41 tok/s | 18,677 MiB |
| `Qwen3.8-27B-UD-Q4_K_M` | 15.43 GiB | 2.0291 | 33.16 tok/s | 18,395 MiB |
| `Qwen3.8-27B-UD-Q4_K_XL` | 16.35 GiB | 2.0353 | 31.54 tok/s | 19,369 MiB |
| `Qwen3.8-27B-UD-Q6_K` | 20.47 GiB | 2.0238 | 26.07 tok/s | 23,606 MiB |
| `Qwen3.8-27B-UD-Q6_K_XL` | 23.56 GiB | 2.0235 | 23.01 tok/s | 26,779 MiB |

From the stock `Q4_K_M` to `UD-Q6_K_XL` is **0.58% of perplexity** — 2.0353 to 2.0235 — for
**−29.00% decode throughput** and **+7.90 GiB of weights**, which is +7.91 GiB
of peak VRAM at 8,192 tokens. The ladder is nearly flat in quality and steep in
cost. If the choice is "which quant fits", the answer is at the low end of this
ladder, not the top.

Two steps inside it are worth naming. `UD-Q4_K_M` is **smaller than the stock
Q4_K_M** (15.43 against 15.66 GiB), **better on perplexity** (2.0291 against
2.0353, −0.30%) and **2.32% faster in decode**. And `UD-Q4_K_XL` is a dead
step: its perplexity is 2.0353, the same as the stock file to four decimals,
while it costs 0.69 GiB more than the stock file and gives up 2.69% of decode
against it. It is also 0.92 GiB larger than `UD-Q4_K_M` and 0.31% worse on
perplexity. Paying 0.92 GiB for a worse score is the kind of trade a table makes
visible and a model card does not.

## The losses

**Prefill at 8,192: two cells, both real.** `OrcaCyber-…-Mix-15.7G` is −3.72%,
1,325.76 against 1,376.98 tok/s, on spreads of 0.73% and 0.44% — separated, and
the cell passes the 12-id gate, so it is a comparison rather than a gated row.
`Qwen3.8-27B-UD-Q4_K_M` is −1.80%, 1,327.13 against 1,351.43, on spreads of
0.73% and 0.70% — also separated, also gated in. Neither is noise, and neither
is explained away by saying the engine had a bad cell: 1,325.76 and 1,327.13 are
inside the engine's own 5.7% band at this depth. These are the two files with
llama.cpp's two highest prefill-at-8,192 numbers in the session. The engine did
not get slower; llama.cpp got faster.

**Decode: eight of eight below at 8,192, and six of eight at depth 0.** The
largest is `Swift-1.5` at 8,192, −2.95%, 30.13 against 31.04 tok/s, on spreads
of 0.41% and 0.15% — separated. The smallest is `UD-Q6_K_XL` at 8,192, −0.49%,
22.45 against 22.56, on spreads of 0.57% and 0.24% — also separated, and small
in absolute terms. At depth 0 the two wins are `UD-Q4_K_M` at +0.74% (33.16
against 32.92) and `UD-Q6_K_XL` at +0.39% (23.01 against 22.92), and `UD-Q6_K`
is level at −0.03%. Neither win is larger than 0.24 tokens per second. The
column is a tie in magnitude and a loss in sign, and the reason the first
version read it the other way is the backend, not the engine.

**Serve b: one cell below by a wide margin, and gated out.** On
`UD-Q6_K_XL` the engine does 68.90 tok/s against llama.cpp's Vulkan build at
96.86, −28.87%, with the engine's own six waves spanning 62.12 to 69.00 (9.99%)
and the Vulkan server's three spanning 90.5 to 96.9 (6.56%). The cell sits at
32,436 MiB, the largest peak anywhere in the session. The cause is
**unexplained**: all 65 server and gate logs were scanned for seven markers —
`failed to allocate`, `out of memory`, `to cpu`, `cpu buffer`, `hipMalloc`,
`hip_out_of_memory` and `page fault` — and none
matched. It
is not called paging, because the logs do not say paging; it is called
unexplained, and the row is gated out of the comparison because the file
diverges at id 4. That does not make the number go away, and it is in the table
for that reason.

**Peak VRAM: eight of eight, with no exception.**

| file | cli @0 h / l | cli @8,192 h / l | serve b h / l | serve Δ |
| --- | ---: | ---: | ---: | ---: |
| `Arete-27B-Q4_K_M` | 17,682 / 16,606 | 19,046 / 16,870 | 25,003 / 22,490 | +11.18% |
| `OrcaCyber-…-Mix-15.7G` | 16,183 / 14,822 | 17,610 / 15,085 | 23,504 / 20,705 | +13.52% |
| `Qwen3.8-27B-Q4_K_M` | 17,314 / 15,865 | 18,677 / 16,128 | 24,571 / 21,736 | +13.05% |
| `Qwen3.8-27B-UD-Q4_K_M` | 17,031 / 15,546 | 18,395 / 15,809 | 24,288 / 21,429 | +13.34% |
| `Qwen3.8-27B-UD-Q4_K_XL` | 17,941 / 16,497 | 19,369 / 16,760 | 25,263 / 22,367 | +12.94% |
| `Qwen3.8-27B-UD-Q6_K` | 22,179 / 20,408 | 23,606 / 20,671 | 29,500 / 26,279 | +12.26% |
| `Qwen3.8-27B-UD-Q6_K_XL` | 25,352 / 23,579 | 26,779 / 23,843 | 32,436 / 29,462 | +10.09% |
| `Swift-1.5-Qwen3.8-27B-Q4_K_M` | 17,905 / 16,606 | 19,332 / 16,870 | 25,226 / 22,477 | +12.23% |

The gap is present at depth 0 before any long context, grows with context, and
grows again with eight slots: on `UD-Q4_K_M` it is 1,485 MiB at depth 0, 2,586
MiB at 8,192 and 2,859 MiB in serve. Both serve configurations are 16,384
tokens per slot — llama.cpp's `-c 131072` is the pool divided across its eight
slots — so the serve cell is like-for-like on context. The engine peaks higher
anyway. This session measured the size of the gap, not its cause; the
decomposition in [`../notes/serve-vram.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/notes/serve-vram.md) is the place
to look for that, and it was measured on a different tree.

## The gate, and what it does not prove

Before any file is timed, both engines generate 12 greedy ids from the same
169-id prompt and the ids are diffed. A file that does not reproduce is not
compared. This is the check that makes the rest of the table a comparison
rather than two independent benchmark runs.

Seven of the eight files that run reproduce **12 of 12 ids**. The eighth,
`Qwen3.8-27B-UD-Q6_K_XL.gguf`, diverges at **id 4**: the engine continues
`310 2649 264 47057 …` where llama.cpp continues `4087 1156 25 12870 …`. Its
perplexity on the fixed corpus agrees with `llama-perplexity` to **0.035%**
(2.0235 against 2.0242). The divergence is a near-tie in the argmax, not a
misread file. It is still gated out, because a model whose ids we do not
reproduce is not a model we quote a speed ratio for.

That is the easy half of the gate's story. The hard half is the control. A
128-token greedy probe on the same prompt was run on the gated file and on
`Qwen3.8-27B-UD-Q6_K.gguf`, a file that passes the 12-id gate:

| file | ids identical | first difference |
| --- | ---: | ---: |
| `Qwen3.8-27B-UD-Q6_K.gguf` (passes 12) | 21 of 128 | id 20 |
| `Qwen3.8-27B-UD-Q6_K_XL.gguf` (gated out) | 17 of 128 | id 4 |

The control diverges at id 20. A file that passes the 12-id gate fails a
128-token one, which means **12 ids is a shallow discriminator**: it catches a
divergence that starts early, and it cannot see one that begins after the
twelfth id.

What the gate establishes is narrow and worth stating precisely: at 12 tokens,
the two engines agree on the argmax for seven files; a disagreement is
detectable; and a file that disagrees is removed from the comparison rather than
averaged into it. What it does not establish is that the engines compute the
same thing beyond 12 tokens, that agreement implies numerical equivalence, or
that either engine is exact against its own reference. The control is the
evidence for the last point: a passing file diverges by id 20, so "passes the
gate" means "passes 12 ids", not "reproduces". Any reader who takes the gate as
a correctness certificate is taking it further than this run supports, and the
128-token probe is here so that is visible rather than implied.

## What this does not claim

It does not claim the engine is faster. It wins prefill at 512 on every file,
wins serve on seven of eight, is level or behind on decode, ties on quality, and
loses memory on all eight. Any summary that leads with the prefill column
without the VRAM column is not this table.

It is one engine, one card and one session, plus the backend check that
re-measured the llama.cpp baseline. The card is a 31.9 GiB R9700 and
every file here fits in 32 GB; nothing in this run says what happens on a
smaller card, a different vendor, or a file that needs offload. The context
lengths are two: 576 and 8,256 tokens for the CLI cells, 16,384 per slot for
serve. There is no claim about 32k, 128k or 262k, which is the context the
files declare.

It is eight files from one directory, not a model census. The other eight
either cannot load or cannot generate, and the attempt table is the boundary of
the claim. It is also not a quality ranking: one 28 KB corpus and one
perplexity number per file cannot rank models, only show that the two engines
score the same weights the same way.

The serve cell is aggregate goodput at eight slots on one prompt distribution.
It is not a claim about sixteen slots, other arrival patterns, or tail latency
under load. And the deltas are medians from one session — two, for the decode
and serve columns, whose baseline was re-measured — and the reproducibility
check against the run 4.7 hours earlier is the reason to trust the engine
column, not a second session that would make it stronger.

## The next file is a mixture of experts

The interesting failure is the mixture-of-experts file.
`Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf` is 753 tensors, 40 decoder blocks plus an
inline MTP block, and 256 experts with 8 used per token. It loads — every tensor
uploads — and generation stops at its embedding table, which is Q8_0 where the
graph takes Q4_K or Q6_K. The MoE forward pass itself is the work in progress,
and that is why the file is not in the matrix.

What it would cost is arithmetic, not measurement. **There is no hipml MoE
throughput number in this article, because none exists yet.** The GPU path is
being built now, and everything in this section is a prediction from the file's
tensor inventory and the bandwidth the dense model already achieves, not a
result.

llama.cpp already runs the file, and it is the bar the engine's MoE path will be
measured against. Warm, q8 KV, 33 greedy tokens: **96.35 tok/s decode on its
ROCm build and 131.23 on Vulkan** at depth 0, 93.40 and 126.39 at 8,192; prefill
3,526 and 3,331 at 512 tokens, 4,259 and 3,825 at 8,192. That is 3.0× to 4.0×
the dense model's decode on the same card, in the same direction the arithmetic
below predicts. It also re-prices the prediction: the engine's 176 tok/s would
be **1.34×** llama.cpp's faster backend, not the 5.30× it is against the dense
model.

Per token at 4,096 context, the file would stream **2.92 GB** (2,921,190,016
bytes), against **15.48 GB** for the 27B dense model — a ratio of **5.30**. At
the bandwidth the dense model already achieves in this session, its 33.16 tok/s
becomes about **176 tok/s**.

The reason is the gather. The expert FFN is **88.09% of the file** but only
**21.01% of a token**, because a token reads 8 of 256 experts. Everything else —
the gated-delta-net projections, the full-attention layers, the router, the
shared expert, the LM head — is where the other four fifths go. The knife edge
is that reading every expert instead of gathering 8 would stream **21.94 GB per
token**, which at the same bandwidth is **23.4 tok/s: slower than the 27B dense
model**. The gather is not an optimisation on top of the feature; it is the
feature. A MoE path that reads all experts would be a regression, and the
arithmetic says so before a line of the kernel is written.

## Where the numbers come from

- The single-model session this one extends, and the decode bandwidth it
  establishes: [`../benchmarks.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/benchmarks.md).
- The model file, its tensor histogram and the load checks:
  [`../native-qwen.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/native-qwen.md).
- The MoE tensor inventory and the 256/8/512 configuration:
  [`../notes/moe.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/notes/moe.md).
- The serve VRAM decomposition, measured on an earlier tree:
  [`../notes/serve-vram.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/notes/serve-vram.md).
- The serve prefill chunk that both engines now match:
  [`../notes/serve-prefill-chunk.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/notes/serve-prefill-chunk.md).
- The stock-model release run and the dense bytes-per-token figure:
  [`../release-run.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/release-run.md).
- The decode roofline this article's bandwidth number sits on:
  [`../notes/decode-roofline.md`](https://github.com/pgmharikrishnan/hipml/blob/main/docs/notes/decode-roofline.md).
