ProBackend
AI Infrastructure
2 days ago6 min read

Harness adapters on llama.cpp: what a 0.32 GB LoRA actually buys you at Q4

Harness adapters on llama.cpp: what a 0.32 GB LoRA actually buys you at Q4

A measurement of harness-hub/Qwen3.8-27B-pi-adapted — a rank-32 LoRA for Qwen3.8-27B — served through **llama.cpp** on a single 32 GB consumer/workstation GPU, at **Q4_K_M**, with speculative decoding enabled.

A measurement of harness-hub/Qwen3.8-27B-pi-adapted — a rank-32 LoRA for Qwen3.8-27B — served through llama.cpp on a single 32 GB consumer/workstation GPU, at Q4_K_M, with speculative decoding enabled.

The short version:

  • The adapter loads cleanly on top of a Q4_K_M GGUF and costs 3.4% decode.
  • It coexists with the model's built-in MTP drafter — draft acceptance stays at 78%.
  • Served in its trained mode (thinking off) it scores 353/400 on MMLU-Pro, 49/50 GSM8K, and 6/6 on our coding suite — against 347 / 47 / 3 for the base model on the same instrument.
  • Served with thinking on, it is no better than base and burns 19% more output tokens.

All numbers below were produced by the same harness, on the same machine, in one session.

Why this measurement exists

The adapter's own card reports large gains under the pi coding agent, evaluated on a Q8_0 base. Most people running a 27B locally do not have room for Q8_0 — on a 32 GB card it leaves roughly 3 GB for KV cache, which is about 45K of context.

So the practical question is not "does the adapter work". It is: does it survive being served at Q4_K_M through llama.cpp, and what does it cost?

Setup

GPUAMD Radeon AI PRO R9700, 32 GB, gfx1201
Runtimellama.cpp 0.7.0-dev, build 11836, Vulkan backend
Base weightsQwen3.8-27B UD-Q4_K_M, 16.56 GB
AdapterQwen3.8-27B-pi-lora-F16.gguf, 318,802,304 B, F16, rank 32
Speculative decodingMTP drafter, 0.85 GB, depth 3
Context32,768, --fit off, bf16 KV, --parallel 1
Samplingtemperature 0, streaming, 128 tokens per rep, 1 warm-up + 3 reps

The Vulkan backend and the MTP drafter each help independently and multiply: Vulkan is +21.8% on decode, MTP depth 3 is +2.08×, and the product (2.532×) predicted the measured result to within 0.5%.

The adapter loads, and it is cheap

code
llama_adapter_lora_init: loading lora adapter ...
- kv 0: general.architecture str = qwen35
- kv 2: adapter.type str = lora
- kv 4: adapter.lora.alpha f32 = 64.000000
Vulkan1 LoRA buffer size = 304.00 MiB
loaded 512 tensors from lora file

512 tensors, 256 target matrices, zero failed, missing or skipped tensors.

base + MTP d3+ LoRAcost
prefill tok/s736.13678.00−7.90%
decode tok/s62.1059.97−3.43%
TTFT41.24 s44.79 s+8.61%
peak VRAM19,171 MiB19,515 MiB+344 MiB

Speculative decoding still works underneath the adapter: the server reported draft_n: 51, draft_n_accepted: 40 — 78.4% acceptance. A 3.4% decode cost for a 0.32 GB adapter is close to free.

Thinking off is not optional

The adapter was trained with thinking disabled. llama.cpp's --jinja template enables thinking by default, so a naive deployment runs the adapter outside its trained mode. The symptom is not a crash — it is token burn:

adapter, thinking offadapter, thinking on
MMLU-Pro output tokens (400 questions)545,230654,451
answers truncated at the 8,192 cap1036
MMLU-Pro353/400343/400

The server-level switch is the reliable one:

sh
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf \
  --lora Qwen3.8-27B-pi-lora-F16.gguf \
  --chat-template-kwargs '{"enable_thinking": false}' \
  -md Qwen3.8-27B-Q4_0-mtp.gguf --spec-draft-n-max 3 \
  -c 32768 --fit off --jinja

Standard suite results

Frozen 400-question MMLU-Pro, 50-question GSM8K, 6 repository-repair coding tasks with hidden tests. Same prompts, same caps, same seed.

configurationMMLU-Pro 400GSM8Kcoding
base, thinking on347/400 (86.8%)47/503/6
adapter, thinking on343/400 (85.8%)46/505/6
adapter, thinking off353/400 (88.2%)49/506/6
earlier base run, same instrument350/400 (87.5%)46/504/6

Read this carefully. The base cell reproduces the earlier run (347 vs 350, p=0.65), so the instrument has not drifted. But at 400 questions one question is 0.25 pp, and none of the differences above is statistically significant on its own (353 vs 350, p=0.71). The Wilson intervals overlap.

What is worth noting is the shape: the adapter in its trained mode is ahead on all three suites simultaneously, and the largest single gap is coding, 6/6 against 3/6. That is consistent with the adapter's published profile — it was trained on repository-repair environments — but with n=6 it is a signal, not a result.

The truncation counts also matter: the thinking-on cells hit the 8,192-token output cap 30 and 36 times out of 400, while thinking-off hit it 10 times. Some of the thinking-off advantage is that it finishes its answers.

Tool calls, not tokens

On a separate 10-task agentic suite (tool-using repository repair with hidden pytest), served thinking-off:

baseadapter
tool calls per task44.927.4
turns per task30.116.7
prompt tokens, whole suite3.57 M1.07 M
wall clock, whole suite907 s884 s

Tool calls fall 39% (paired, 95% CI −24% to −54% per task; 9 of 10 tasks used fewer). This reproduces the −39% that the same recipe reports for LFM2.5-24B-A2B, which is the strongest evidence that it is a real policy effect rather than a quirk of one training run.

But wall clock barely moved — 2.6%. The adapter writes roughly 240 tokens per turn against 160 for the base: it moves the reasoning inline, between tool calls, instead of into a preamble. The savings land in prompt tokens (3.3× fewer, because there are fewer turns to re-read), and prompt tokens are cheap. Wall time is decode-bound, and decode did not change.

So the honest product is context economy, not speed. It makes a session run longer before compaction. It does not make the box feel faster.

Caveats

  • Q4_K_M, not Q8_0. The adapter was trained on BF16 weights, and its card warns that lower-bit quants drift further. These results may understate the adapter, not overstate it.
  • 10 tasks cannot settle accuracy. One task is 10 pp. On the agentic suite the adapter solved 4/10 against base 6/10 — inside the noise, and not evidence either way.
  • One machine, one backend. Vulkan on gfx1201; ROCm wins prefill (+49.8%) while Vulkan wins decode (+21.8%). There is no single winner.
  • The coding suite is 6 tasks. 6/6 versus 3/6 is suggestive, not significant.

Credit

Adapter by harness-hub. These measurements are independent of the adapter's authors and were taken on different hardware, a different runtime, and a lower quantisation than their evaluation.