AI model local deployment: the measured result
MiniCPM-V 4.7 35B-A3B paired a strong multimodal-capable model with the fastest decode in our R9700 comparison: 139 tokens per second, versus 66.8 on an M4 Max using the same mixed 4/8-bit recipe. DSpark can add more, but depth matters. In short-prompt checks, shallow draft settings lifted decode; deeper settings were slower than running without a draft. That’s the useful story here—not a blanket promise that speculative decoding makes every run faster.
What is an AI model?
An AI model is a learned computational system that maps input to output; in this case, the input can include text and visual content, and the output is generated language. MiniCPM-V 4.7 is a mixture-of-experts model. The name 35B-A3B refers to roughly 35 billion total parameters and about 3 billion active per token. That sparse computation does not mean the full expert set disappears: the complete model weights still need to be represented.
OpenBMB’s MiniCPM-V 4.7 model card describes image and video understanding, with a Qwen3.5/Qwen3.6 language backbone and SigLIP2 vision module. It also presents early visual-token compression, Lookahead Expert Scheduling, fine-grained layer-wise quantization and DSpark speculative sampling as parts of its on-device approach. OpenBMB’s statements about smooth phone inference and its device results are vendor evaluations, not measurements in our benchmark.
The card’s examples use 16x visual downsampling by default, with 4x available when retaining more detail matters. They allow up to 36 image slices and 128 video frames. For instruct use, the card recommends temperature 0.7, top-p 0.95, top-k 100, min-p 0, presence penalty 0 and repetition penalty 1.03. It documents a 256K context configuration; our local suite used 131K. Apache 2.0 is the license identified on the card.
How to deploy an AI model: our test recipe
For this test, we used a public mixed 4/8-bit GGUF quantization weighing 20.6 GiB. The M4 Max comparison used the same mixed-precision recipe. Both were run against the same standard-v2 suite: 400 MMLU-Pro questions, 50 GSM8K questions and six coding tasks. We used temperature 0, a 2,048-token answer limit and four-way parallelism. “At cap” counts show MMLU-Pro or GSM8K answers that reached that output limit.
Setup. AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000, Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention on; 131K context; ubatch 2048; four parallel slots sharing one unified KV cache. Tested October 2026.
| Run | MMLU-Pro | GSM8K | Coding | Decode | Suite wall time |
|---|---|---|---|---|---|
| R9700, mixed 4/8-bit GGUF | 90.0% (3 at cap) | 100% (0 at cap) | 5/6 | 139 tok/s | 48 min |
| M4 Max, same mixed 4/8-bit recipe | 90.0% | 100% | 6/6 | 66.8 tok/s | — |
| M4 Max, tuned 4-bit | 91.0% | 98% | 6/6 | 50.6 tok/s | — |
The first two rows make the cleanest speed comparison. They share the mixed-weight recipe and report equal MMLU-Pro and GSM8K results, while the R9700 decoded at about 2.1 times the M4 Max rate. The R9700 completed one fewer coding task in this run. The tuned 4-bit MLX variant scored one point higher on MMLU-Pro, but it used a different recipe and ran slower; it isn’t a hardware-only comparison.
This was a compact language suite, not a broad multimodal evaluation. It did not score image or video understanding. We also measured MiniCPM at a 24K prompt: prefill was about 4,100 tokens per second and decode was 121 tokens per second. A separate server-setting check showed that reducing ubatch from 2048 to 512 or 1024 sharply reduced prefill and decode. At 512, observed prefill fell from about 3,850 to 830 tok/s, and decode from 103 to 61 tok/s. The full-suite result uses ubatch 2048.
Recommended settings and DSpark depth
For a direct reproduction of our R9700 suite, use the setup above with the mixed 4/8-bit target. Keep ubatch at 2048 and start with the measured 131K context. For image or video work, OpenBMB’s visual guidance is to start at 16x downsampling for efficiency and use 4x when finer visual detail matters. Those are model-card settings, not extra R9700 benchmark results. Likewise, the card’s instruct sampling recommendations are temperature 0.7, top-p 0.95, top-k 100, min-p 0, presence penalty 0 and repetition penalty 1.03. Our standardized score run deliberately used temperature 0; it does not establish that greedy decoding is best for conversation.
OpenBMB publishes a DSpark draft model, a 5-layer, 389-million-parameter draft. We used it at Q8_0. Our first check was discouraging: with the mixed target, code/prose prompts and thinking off, no drafting produced 141 tok/s. Draft depths 4, 8 and 15 returned 71/56, 102/74 and 87/60 tok/s.
A full depth sweep changed the picture. Depth 2 on the mixed target gave 180/158 tok/s. With OpenBMB’s own Q4_K_M target (19.7 GiB, 170 tok/s without drafting), depth 3 gave 210/178, depth 2 gave 198/181, and depth 4 gave 190/156. Depths 8 and 15 fell to 52/36 and 37/23. Acceptance was similar on both targets, so the deep-setting slowdown reflects verification cost at those batch sizes on this backend, not a mismatched draft.
On the full suite, both drafted setups retained useful quality. The OpenBMB Q4_K_M target at depth 3 scored 89.25% MMLU-Pro, 98% GSM8K and 6/6 coding, with 157 tok/s single-stream decode; it finished in 35 minutes, compared with 48 minutes for the undrafted mixed run. The mixed target at depth 2 scored 89.75%, 100% and 6/6 at 153 tok/s, but took 62 minutes. Under four parallel requests, its decode gain did not carry through to shorter suite time. Measure depth on your own workload. The model card’s suggested depth of 15 was the slowest setting we tried.
Verdict: a strong local multimodal option
MiniCPM-V 4.7 is the strongest speed-and-quality balance in this R9700 set for someone who wants a local multimodal model. It returned strong language-suite results at 139 tok/s, fit the tested setup and more than doubled the measured decode rate of the comparable mixed-recipe M4 Max run. But the benchmark did not exercise images or video; test those workloads with your own prompts rather than treating the language scores as a proxy.
If you add DSpark, start at depth 2–3 and measure under your real concurrency. Shallow drafting was compelling in the short-prompt checks. It wasn’t a universal win across the full suite.
R9700 series: “Sage-A1.5 on the R9700: why thinking mode changed the score”; “Swift 1.5 versus stock Qwen3.8-27B on the R9700”; “Qwen3.8-27B on the R9700: MTP depth and quantization trade-offs”; “Qwen3.8-27B GSQ-RCO IQ3_S: 11.3 GiB and a useful quality floor”; and “Flint 40B-A2.5B on the R9700: fast decode, longer work”.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
