Why Shallow Speculative Decoding Won in Our AI Edge Infrastructure Reload Tests

Introduction: Why Shallow Speculative Decoding Won in Our Reload Tests

In our reload sweeps, a shallow draft depth of two or three beat deeper drafts on every model checked, including MiniCPM-V with OpenBMB's DSpark draft, which reached 210/178 tok/s at depth 3 against 170 without drafting. The model cards' deeper suggested depths were consistently the slowest settings on this backend.

Speculative decoding asks a smaller draft model or an auxiliary prediction head to propose tokens that the target can verify. When the target accepts enough proposals, it can reduce serial target-model work. But a deeper draft also adds proposal and verification work. The best setting depends on both acceptance and the cost of each speculative step.

Understanding AI Edge Infrastructure and Speculative Performance

As organizations build out robust ai edge infrastructure, balancing latency and computational overhead is paramount. What is edge infrastructure? It refers to the decentralized hardware and software stack—ranging from local workstations and edge servers to unified memory controllers and runtime engines like llama.cpp—that execute workloads directly on-site rather than in a distant cloud center. As highlighted by What Is Edge AI? | IBM, running machine learning models at the edge brings intelligence closer to data sources, minimizing network hops. However, local inference on hardware like the AMD Radeon AI PRO R9700 requires careful tuning of speculative decoding parameters to ensure that auxiliary draft models accelerate generation rather than creating verification bottlenecks.

Setup and Methodology

Setup. AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000; Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention on; 131K context unless noted; ubatch 2048; four parallel slots sharing one unified KV cache. Tested October 2026.

The public Qwen3.8-27B card describes a dense 27B model with multi-token prediction training, and MiniCPM-V’s card provides the separate DSpark draft model. The model card notes training and architecture for the public weights; our results are specific to the GGUFs, draft pairing, and server used here.

Depth-Two Results in Practical Reload Sweeps

A reload sweep using code and prose prompts with 768 generated tokens found the same directional result for each listed target: depth two outperformed depth four.

The first pair shows code/prose speed respectively; the stock Q4 figures are the benchmark decode comparison. These numbers come from different measurements, so we retain the formats rather than pretending all cells came from one identical prompt set.

The broader model report also found that Swift 1.5 borrowed the stock Qwen3.8-27B MTP drafter, with 82% acceptance at depth two. Built-in MTP showed additional gains: Sage-A1.5 measured 207/174 tokens per second at depth two versus 168 without drafting, and Qwen3.6-35B-A3B reached about 198 tokens per second with its built-in MTP. We did not record a paired no-draft number for Qwen3.6, so that result is a throughput observation rather than a calculated uplift. This pattern is consistent with a useful tuning rule: start shallow, compare the same prompts at each depth, and record both accepted tokens and final throughput. Acceptance rate alone is not the objective. A configuration can accept proposals yet lose time to draft-model work or verification overhead.

DSpark: Depth Decides the Result

We paired MiniCPM-V with OpenBMB's DSpark draft, a five-layer, 389M-parameter model at Q8_0, first with our mixed 4/8-bit target and then with OpenBMB's own Q4_K_M target. Code and prose prompts, thinking off, tokens per second:

Our first test only covered depths 4, 8 and 15, all slower than no drafting, and looked like a target mismatch. The full sweep showed otherwise: acceptance was similar on both targets, and the uneven dips (for example depth 3 on the mixed target) point to verification cost at particular batch sizes on this backend rather than to the draft. With thinking on, the deep settings were slower still (58/57 at depth 15). OpenBMB reports acceptance lengths of 3.5–7.4 in its own setting; that is not directly comparable to our percentages.

On the full quality suite, the Q4_K_M target at depth 3 kept its scores (89.25% MMLU-Pro, 98% GSM8K, 6/6 coding), decoded at 157 tok/s and finished in 35 minutes, against 48 minutes for the undrafted mixed model. The mixed target at depth 2 matched its scores at 153 tok/s but took 62 minutes under four parallel requests, so single-stream gains do not automatically survive concurrency.

OrcaCyber-Nano: Built-in MTP Versus Separate DFlash2

OrcaCyber-Nano-1.0, a Qwen3.8-27B-based model, ships both a built-in MTP head and a separate 2 GiB DFlash2 drafter. On the same code/prose prompts, depth 3 was best for both: MTP gave 79/60 tok/s (71/59 at depth 2) and DFlash2 81/62 (78/49 at depth 4, and 35/19 at the card's suggested depth of 7). The two approaches landed close together; the deep setting was the clear loser. Full quality-suite results for this model are not part of this article.

A Practical Tuning Sequence

Choose one target quantization and one draft that are known to work together. Establish a no-draft baseline on the same prompts and system state. Then test depth two and a deeper option, keeping context, batch, slot count, output length, and temperature constant. Save prompt-level results rather than comparing unrelated headline speeds. Include acceptance and throughput, and repeat enough to see whether the direction is stable.

The standard shoot-out reports model-level decode and suite performance; it should not be used to infer speculative-decoding gains unless the draft setting is matched. The quantization article separately compares Qwen3.8 at IQ3_S, Q4, and Q6. Those distinctions matter because target quantization can alter token probabilities and thus draft acceptance.

What Is Still Missing

We did not measure energy, so faster tokens per second cannot be translated to lower power or cost per token. Sweeps used short code and prose prompts; long-context and multi-user behavior can shift the best depth, as the mixed-target DSpark suite run showed. The evidence supports starting at depth two or three on the models we tested, then checking the workload itself. It does not support a universal optimal depth.

Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, target-draft compatibility, and load. They are not a general ranking or a recommendation.