When engineers look at ai model local deployment in 2026, the headline is never a simple winner-take-all story. On this R9700 run, a sparse mixture-of-experts (MoE) model led decode speed, while the mixed-quant MiniCPM-V run led the tested MMLU-Pro and GSM8K combination. Meanwhile, dense Qwen variants took significantly longer to finish the evaluation suite. No single score captures the best model for every workload, and hardware capability interacts heavily with quantization, draft depth, and prompt length.
We compared eight local model configurations against the same compact benchmark suite to understand how modern hardware handles heavy inference workloads. The central tradeoff is clear: active parameter count and quantization can make a large MoE exceptionally fast, but they do not guarantee the strongest benchmark quality. Prompt truncation also plays a major role. Several MMLU-Pro answers reached the response cap, which limits how confidently we can interpret their raw percentage scores. For related measured comparisons, see Navigating AI Model Local Deployment: What Two MiniCPM Runs Can—and Can't—Tell Us and Flint on the R9700: AI Model Local Deployment Performance and Reasoning Realities.
As software development companies and enterprises explore agentic workflows—where autonomous systems reason, plan, and execute multi-step coding tasks—understanding these local performance profiles becomes critical. What is Agentic AI? In software engineering, Agentic AI refers to systems endowed with reasoning loops, multi-token prediction, and iterative tool use that move beyond simple chat responses to accomplish complex software development tasks autonomously. Deploying these models locally requires robust inference runtimes, precise hardware optimization, and careful balancing of speed and accuracy.
Understanding How to Deploy AI Models Locally
For practitioners learning how to deploy ai model workloads on workstation hardware, understanding runtime environments is essential. Running large language models and vision-language models locally involves configuring backends like llama.cpp with appropriate acceleration layers, managing unified memory, and tuning parallel inference slots.
Setup. AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000; Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention on; 131K context unless noted; ubatch 2048; four parallel slots sharing one unified KV cache. Tested October 2026. Both suites used standard-v2: 400 MMLU-Pro items, 50 GSM8K questions, and six coding tasks. Temperature was zero and the answer cap was 2,048 tokens. Decode speed is single-stream decode, median of five short runs. “At cap” counts answers truncated by that cap. These scores are not measurements of every skill or a statistical estimate of general capability. The MiniCPM-V card identifies an Apache-2.0 multimodal 35B-A3B model. The Flint card describes a 39.8B MoE with approximately 2.5B active parameters. Qwen3.8-27B is a dense 27B vision-language model whose model card documents multi-token prediction and thinking controls. Their public context lengths and training notes describe the model family, not the limits of this particular GGUF benchmark.
Report Card on Local Configurations
| Configuration | MMLU-Pro | GSM8K | Coding | Decode tok/s | Suite wall |
|---|---|---|---|---|---|
| MiniCPM-V-4.7 35B-A3B, mixed 4/8-bit | 90.0% (3 at cap) | 100% (0) | 5/6 | 139 | 48 min |
| Sage-A1.5 35B-A3B Q4_K_M, MTP d2, thinking off | 86.75% (29) | 98% (1) | 6/6 | 195 | 59 min |
| Swift 1.5 Qwen3.8-27B FT Q4_K_M + stock MTP d2 | 89.5% (13) | 94% (0) | 6/6 | 60 | 97 min |
| Qwen3.8-27B UD-Q4_K_XL + MTP d2 | 86.75% (31) | 94% (1) | 5/6 | 62 | 149 min |
| Qwen3.8-27B UD-Q4_K_XL + MTP d4 | 87.5% (30) | 92% (3) | 4/6 | 56 | 127 min |
| Qwen3.8-27B UD-Q6_K_XL + MTP d4 | 86.25% (32) | 96% (1) | 4/6 | 51 | 146 min |
| Qwen3.8-27B GSQ-RCO IQ3_S + built-in MTP d2 | 85.25% (39) | 92% (3) | 5/6 | 57 | 149 min |
| Flint 40B-A2.5B, mixed 4/8-bit | 71.25% (91) | 92% (3) | 4/6 | 120 | 153 min |
Sage’s model card identifies the Qwen3.5-MoE lineage and recommends sampling settings for its thinking behavior; our benchmark configuration turned thinking off at temperature zero. That distinction matters, because the separate “thinking on, greedy” run behaved badly and is analyzed in “When benchmarks lie: Sage-A1.5 thinking loop under greedy decoding.” It should not be combined with the standard Sage row.
Dense Versus MoE Architecture Tradeoffs
The fastest row in our benchmark was Sage-A1.5 at 195 tokens per second, with MTP depth two enabled and thinking off. MiniCPM-V produced 139 tokens per second and a shorter 48-minute suite wall, while scoring 90% MMLU-Pro and 100% GSM8K. Flint’s sparse MoE architecture ran at 120 tokens per second, but its benchmark profile was materially lower at 71.25% MMLU-Pro and 4/6 coding tasks.
The active parameter count helps explain why a large MoE can decode quickly, but it does not explain away quality differences. The Qwen variants clustered around 51–62 tokens per second and required 127–149 minutes for the suite. More draft depth did not automatically improve all outcomes. Q4 with depth two completed in 149 minutes; depth four took 127 minutes but scored lower on GSM8K and coding than the Swift fine-tune row. Q6 also took 146 minutes. These are configuration comparisons, not a controlled test of only quantization: model fine-tune, quantization recipe, and draft model differ across some rows.
Several MMLU-Pro results have many answers at the cap. For example, the Flint row has 91 truncated answers. That count warns us that the final score can be affected by the response budget and evaluation setup. We report it beside the score instead of presenting the percentage as an unqualified ranking.
The Apple Comparison Is Narrow
To contextualize these workstation numbers, we looked at the same suite on an Apple M4 Max. MiniCPM-V mixed 4/8-bit scored 90.0% MMLU-Pro, 100% GSM8K and 6/6 coding at 66.8 decode tokens per second. The R9700 run used the same mixed weights recipe and recorded 139 tokens per second, about 2.1 times the decode rate.
The coding scores differ by one task, so this is not an identical quality result. It is the strongest like-for-like speed comparison in our data. The Apple list also includes separately tuned models and quantizations; those rows should not be read as direct R9700 matches. We did not measure a matching Apple long-context prefill point for these models.
Choosing a Configuration for Local Workloads
For a multimodal-capable model with a strong measured result and shorter suite time, MiniCPM-V is a compelling tested choice. For maximum measured decode among the presented standard rows, Sage-A1.5 with thinking off was fastest. For a Qwen-family workflow, Swift’s reported balance of 89.5% MMLU-Pro, 6/6 coding and 60 tokens per second looks promising, though the suite still took 97 minutes. Flint demonstrates why throughput and benchmark quality belong together in a report card.
These measurements say what happened with the stated suite and setup. They do not establish best-in-class performance, long-term reliability, image/video quality, power efficiency, or total ownership cost. The next articles in the series, such as those exploring speculative decoding and quantization, dive deeper into specific performance vectors, while the upcoming verdict piece summarizes who may find this hardware platform useful.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
