# Harness adapters on llama.cpp: what a 0.32 GB LoRA actually buys you at Q4

- **Canonical URL:** https://www.probackend.com/ai-local-model-deployment/harness-adapters-llama-cpp-q4-lora
- **Section:** AI Infrastructure
- **Published:** 2026-11-19
- **Summary:** A measurement of harness-hub/Qwen3.8-27B-pi-adapted, a rank-32 LoRA for Qwen3.8-27B, served through llama.cpp at Q4_K_M on a 32 GB GPU.

---

A measurement of [harness-hub/Qwen3.8-27B-pi-adapted](https://huggingface.co/harness-hub/Qwen3.8-27B-pi-adapted) — a rank-32 LoRA for Qwen3.8-27B — served through **llama.cpp** on a single 32 GB consumer/workstation GPU, at **Q4_K_M**, with speculative decoding enabled.

The short version:

- The adapter **loads cleanly** on top of a Q4_K_M GGUF and **costs 3.4% decode**.
- It **coexists with the model's built-in MTP drafter** — draft acceptance stays at 78%.
- Served in its trained mode (**thinking off**) it scores **353/400 on MMLU-Pro**, **49/50 GSM8K**, and **6/6 on our coding suite** — against **347 / 47 / 3** for the base model on the same instrument.
- Served with thinking **on**, it is no better than base and burns 19% more output tokens.

All numbers below were produced by the same harness, on the same machine, in one session.

## Why this measurement exists

The adapter's own card reports large gains under the `pi` coding agent, evaluated on a **Q8_0** base. Most people running a 27B locally do not have room for Q8_0 — on a 32 GB card it leaves roughly 3 GB for KV cache, which is about 45K of context.

So the practical question is not "does the adapter work". It is: **does it survive being served at Q4_K_M through llama.cpp, and what does it cost?**

## Setup

| | |
|---|---|
| GPU | AMD Radeon AI PRO R9700, 32 GB, `gfx1201` |
| Runtime | llama.cpp `0.7.0-dev`, build **11836**, **Vulkan** backend |
| Base weights | Qwen3.8-27B UD-Q4_K_M, 16.56 GB |
| Adapter | `Qwen3.8-27B-pi-lora-F16.gguf`, 318,802,304 B, F16, rank 32 |
| Speculative decoding | MTP drafter, 0.85 GB, depth 3 |
| Context | 32,768, `--fit off`, bf16 KV, `--parallel 1` |
| Sampling | temperature 0, streaming, 128 tokens per rep, 1 warm-up + 3 reps |

The Vulkan backend and the MTP drafter each help independently and **multiply**: Vulkan is +21.8% on decode, MTP depth 3 is +2.08×, and the product (2.532×) predicted the measured result to within 0.5%.

## The adapter loads, and it is cheap

```
llama_adapter_lora_init: loading lora adapter ...
- kv 0: general.architecture str = qwen35
- kv 2: adapter.type str = lora
- kv 4: adapter.lora.alpha f32 = 64.000000
Vulkan1 LoRA buffer size = 304.00 MiB
loaded 512 tensors from lora file
```

512 tensors, 256 target matrices, **zero failed, missing or skipped tensors**.

| | base + MTP d3 | + LoRA | cost |
|---|---:|---:|---:|
| prefill tok/s | 736.13 | 678.00 | **−7.90%** |
| decode tok/s | 62.10 | 59.97 | **−3.43%** |
| TTFT | 41.24 s | 44.79 s | +8.61% |
| peak VRAM | 19,171 MiB | 19,515 MiB | **+344 MiB** |

Speculative decoding still works underneath the adapter: the server reported `draft_n: 51, draft_n_accepted: 40` — **78.4% acceptance**. A 3.4% decode cost for a 0.32 GB adapter is close to free.

## Thinking off is not optional

The adapter was trained with thinking disabled. llama.cpp's `--jinja` template enables thinking by default, so a naive deployment runs the adapter **outside its trained mode**. The symptom is not a crash — it is token burn:

| | adapter, thinking off | adapter, thinking on |
|---|---:|---:|
| MMLU-Pro output tokens (400 questions) | **545,230** | 654,451 |
| answers truncated at the 8,192 cap | **10** | 36 |
| MMLU-Pro | **353/400** | 343/400 |

The server-level switch is the reliable one:

```sh
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf \
  --lora Qwen3.8-27B-pi-lora-F16.gguf \
  --chat-template-kwargs '{"enable_thinking": false}' \
  -md Qwen3.8-27B-Q4_0-mtp.gguf --spec-draft-n-max 3 \
  -c 32768 --fit off --jinja
```

## Standard suite results

Frozen 400-question MMLU-Pro, 50-question GSM8K, 6 repository-repair coding tasks with hidden tests. Same prompts, same caps, same seed.

| configuration | MMLU-Pro 400 | GSM8K | coding |
|---|---:|---:|---:|
| base, thinking on | 347/400 (86.8%) | 47/50 | 3/6 |
| adapter, thinking on | 343/400 (85.8%) | 46/50 | 5/6 |
| **adapter, thinking off** | **353/400 (88.2%)** | **49/50** | **6/6** |
| earlier base run, same instrument | 350/400 (87.5%) | 46/50 | 4/6 |

**Read this carefully.** The base cell reproduces the earlier run (347 vs 350, p=0.65), so the instrument has not drifted. But at 400 questions one question is 0.25 pp, and none of the differences above is statistically significant on its own (353 vs 350, p=0.71). The Wilson intervals overlap.

What is worth noting is the *shape*: the adapter in its trained mode is ahead on all three suites simultaneously, and the largest single gap is **coding, 6/6 against 3/6**. That is consistent with the adapter's published profile — it was trained on repository-repair environments — but with n=6 it is a signal, not a result.

The truncation counts also matter: the thinking-on cells hit the 8,192-token output cap 30 and 36 times out of 400, while thinking-off hit it 10 times. Some of the thinking-off advantage is that it finishes its answers.

## Tool calls, not tokens

On a separate 10-task agentic suite (tool-using repository repair with hidden pytest), served thinking-off:

| | base | adapter |
|---|---:|---:|
| tool calls per task | 44.9 | **27.4** |
| turns per task | 30.1 | **16.7** |
| prompt tokens, whole suite | 3.57 M | **1.07 M** |
| wall clock, whole suite | 907 s | 884 s |

Tool calls fall **39%** (paired, 95% CI −24% to −54% per task; 9 of 10 tasks used fewer). This reproduces the −39% that the same recipe reports for LFM2.5-24B-A2B, which is the strongest evidence that it is a real policy effect rather than a quirk of one training run.

**But wall clock barely moved — 2.6%.** The adapter writes roughly 240 tokens per turn against 160 for the base: it moves the reasoning *inline*, between tool calls, instead of into a preamble. The savings land in prompt tokens (3.3× fewer, because there are fewer turns to re-read), and prompt tokens are cheap. Wall time is decode-bound, and decode did not change.

So the honest product is **context economy, not speed**. It makes a session run longer before compaction. It does not make the box feel faster.

## Caveats

- **Q4_K_M, not Q8_0.** The adapter was trained on BF16 weights, and its card warns that lower-bit quants drift further. These results may *understate* the adapter, not overstate it.
- **10 tasks cannot settle accuracy.** One task is 10 pp. On the agentic suite the adapter solved 4/10 against base 6/10 — inside the noise, and not evidence either way.
- **One machine, one backend.** Vulkan on `gfx1201`; ROCm wins prefill (+49.8%) while Vulkan wins decode (+21.8%). There is no single winner.
- **The coding suite is 6 tasks.** 6/6 versus 3/6 is suggestive, not significant.

## Credit

Adapter by [harness-hub](https://huggingface.co/harness-hub). These measurements are independent of the adapter's authors and were taken on different hardware, a different runtime, and a lower quantisation than their evaluation.
