A measurement of harness-hub/Qwen3.8-27B-pi-adapted — a rank-32 LoRA for Qwen3.8-27B — served through llama.cpp on a single 32 GB consumer/workstation GPU, at Q4_K_M, with speculative decoding enabled.
The short version:
- The adapter loads cleanly on top of a Q4_K_M GGUF and costs 3.4% decode.
- It coexists with the model's built-in MTP drafter — draft acceptance stays at 78%.
- Served in its trained mode (thinking off) it scores 353/400 on MMLU-Pro, 49/50 GSM8K, and 6/6 on our coding suite — against 347 / 47 / 3 for the base model on the same instrument.
- Served with thinking on, it is no better than base and burns 19% more output tokens.
All numbers below were produced by the same harness, on the same machine, in one session.
Why this measurement exists
The adapter's own card reports large gains under the pi coding agent, evaluated on a Q8_0 base. Most people running a 27B locally do not have room for Q8_0 — on a 32 GB card it leaves roughly 3 GB for KV cache, which is about 45K of context.
So the practical question is not "does the adapter work". It is: does it survive being served at Q4_K_M through llama.cpp, and what does it cost?
Setup
| GPU | AMD Radeon AI PRO R9700, 32 GB, gfx1201 |
| Runtime | llama.cpp 0.7.0-dev, build 11836, Vulkan backend |
| Base weights | Qwen3.8-27B UD-Q4_K_M, 16.56 GB |
| Adapter | Qwen3.8-27B-pi-lora-F16.gguf, 318,802,304 B, F16, rank 32 |
| Speculative decoding | MTP drafter, 0.85 GB, depth 3 |
| Context | 32,768, --fit off, bf16 KV, --parallel 1 |
| Sampling | temperature 0, streaming, 128 tokens per rep, 1 warm-up + 3 reps |
The Vulkan backend and the MTP drafter each help independently and multiply: Vulkan is +21.8% on decode, MTP depth 3 is +2.08×, and the product (2.532×) predicted the measured result to within 0.5%.
The adapter loads, and it is cheap
llama_adapter_lora_init: loading lora adapter ...
- kv 0: general.architecture str = qwen35
- kv 2: adapter.type str = lora
- kv 4: adapter.lora.alpha f32 = 64.000000
Vulkan1 LoRA buffer size = 304.00 MiB
loaded 512 tensors from lora file
512 tensors, 256 target matrices, zero failed, missing or skipped tensors.
| base + MTP d3 | + LoRA | cost | |
|---|---|---|---|
| prefill tok/s | 736.13 | 678.00 | −7.90% |
| decode tok/s | 62.10 | 59.97 | −3.43% |
| TTFT | 41.24 s | 44.79 s | +8.61% |
| peak VRAM | 19,171 MiB | 19,515 MiB | +344 MiB |
Speculative decoding still works underneath the adapter: the server reported draft_n: 51, draft_n_accepted: 40 — 78.4% acceptance. A 3.4% decode cost for a 0.32 GB adapter is close to free.
Thinking off is not optional
The adapter was trained with thinking disabled. llama.cpp's --jinja template enables thinking by default, so a naive deployment runs the adapter outside its trained mode. The symptom is not a crash — it is token burn:
| adapter, thinking off | adapter, thinking on | |
|---|---|---|
| MMLU-Pro output tokens (400 questions) | 545,230 | 654,451 |
| answers truncated at the 8,192 cap | 10 | 36 |
| MMLU-Pro | 353/400 | 343/400 |
The server-level switch is the reliable one:
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf \
--lora Qwen3.8-27B-pi-lora-F16.gguf \
--chat-template-kwargs '{"enable_thinking": false}' \
-md Qwen3.8-27B-Q4_0-mtp.gguf --spec-draft-n-max 3 \
-c 32768 --fit off --jinja
Standard suite results
Frozen 400-question MMLU-Pro, 50-question GSM8K, 6 repository-repair coding tasks with hidden tests. Same prompts, same caps, same seed.
| configuration | MMLU-Pro 400 | GSM8K | coding |
|---|---|---|---|
| base, thinking on | 347/400 (86.8%) | 47/50 | 3/6 |
| adapter, thinking on | 343/400 (85.8%) | 46/50 | 5/6 |
| adapter, thinking off | 353/400 (88.2%) | 49/50 | 6/6 |
| earlier base run, same instrument | 350/400 (87.5%) | 46/50 | 4/6 |
Read this carefully. The base cell reproduces the earlier run (347 vs 350, p=0.65), so the instrument has not drifted. But at 400 questions one question is 0.25 pp, and none of the differences above is statistically significant on its own (353 vs 350, p=0.71). The Wilson intervals overlap.
What is worth noting is the shape: the adapter in its trained mode is ahead on all three suites simultaneously, and the largest single gap is coding, 6/6 against 3/6. That is consistent with the adapter's published profile — it was trained on repository-repair environments — but with n=6 it is a signal, not a result.
The truncation counts also matter: the thinking-on cells hit the 8,192-token output cap 30 and 36 times out of 400, while thinking-off hit it 10 times. Some of the thinking-off advantage is that it finishes its answers.
Tool calls, not tokens
On a separate 10-task agentic suite (tool-using repository repair with hidden pytest), served thinking-off:
| base | adapter | |
|---|---|---|
| tool calls per task | 44.9 | 27.4 |
| turns per task | 30.1 | 16.7 |
| prompt tokens, whole suite | 3.57 M | 1.07 M |
| wall clock, whole suite | 907 s | 884 s |
Tool calls fall 39% (paired, 95% CI −24% to −54% per task; 9 of 10 tasks used fewer). This reproduces the −39% that the same recipe reports for LFM2.5-24B-A2B, which is the strongest evidence that it is a real policy effect rather than a quirk of one training run.
But wall clock barely moved — 2.6%. The adapter writes roughly 240 tokens per turn against 160 for the base: it moves the reasoning inline, between tool calls, instead of into a preamble. The savings land in prompt tokens (3.3× fewer, because there are fewer turns to re-read), and prompt tokens are cheap. Wall time is decode-bound, and decode did not change.
So the honest product is context economy, not speed. It makes a session run longer before compaction. It does not make the box feel faster.
Caveats
- Q4_K_M, not Q8_0. The adapter was trained on BF16 weights, and its card warns that lower-bit quants drift further. These results may understate the adapter, not overstate it.
- 10 tasks cannot settle accuracy. One task is 10 pp. On the agentic suite the adapter solved 4/10 against base 6/10 — inside the noise, and not evidence either way.
- One machine, one backend. Vulkan on
gfx1201; ROCm wins prefill (+49.8%) while Vulkan wins decode (+21.8%). There is no single winner. - The coding suite is 6 tasks. 6/6 versus 3/6 is suggestive, not significant.
Credit
Adapter by harness-hub. These measurements are independent of the adapter's authors and were taken on different hardware, a different runtime, and a lower quantisation than their evaluation.