Performance Realities in Modern AI Edge Infrastructure
In our standard-v2 run, MiniCPM-V-4.7 mixed 4/8-bit reached 139 tok/s on the Radeon R9700 and 66.8 tok/s on an M4 Max. Both recorded the same MMLU-Pro and GSM8K scores; the AMD run scored one fewer coding task. This is a useful measured comparison, not a universal hardware ranking. MiniCPM-V-4.7 is an unusual workload for a hardware shoot-out: its public model card identifies it as a multimodal MoE model in a 35B-A3B variant. The card describes single-image, multi-image, and video understanding; an early visual-token compression approach; and on-device optimization through lookahead expert scheduling, fine-grained layer-wise quantization, and DSpark. It lists the Apache 2.0 license and recommends instruct-mode sampling at temperature 0.7, top-p 0.95, and top-k 100. Our suite instead fixes temperature at zero so runs are reproducible under one benchmark recipe. Those are different goals, so the suite score should not be read as an evaluation of every intended use. Read the MiniCPM-V-4.7 model card.
What Is Edge Infrastructure and Edge AI?
To understand how hardware runtimes handle large open-weights models, we have to look closely at ai edge infrastructure. In modern computing, edge infrastructure refers to the decentralized hardware and software architectures that process data close to where it is captured or consumed, minimizing latency and bandwidth overhead compared to traditional centralized cloud datacenters.
When discussing edge intelligence, IBM's framework—frequently referenced in discussions of What Is Edge AI? | IBM—highlights the deployment of machine learning algorithms and inference directly on local edge devices, gateways, or dedicated local server boxes. This decentralization brings unique demands: hardware must deliver high throughput under constrained power envelopes while maintaining reliable output quality. Running a 35-billion parameter multimodal Mixture-of-Experts model locally tests the absolute limits of current edge infrastructure, pushing memory bandwidth, quantization efficiency, and runtime optimization to new thresholds.
The Measured Comparison
| Standard-v2 result | M4 Max / MLX | Radeon R9700 / Vulkan |
|---|---|---|
| Model and weights | MiniCPM-V-4.7 35B-A3B, mixed 4/8-bit | MiniCPM-V-4.7 35B-A3B, mixed 4/8-bit GGUF, 20.6 GiB |
| MMLU-Pro, 400 items | 90.0% | 90.0% (3 answers at cap) |
| GSM8K, 50 items | 100% | 100% (0 at cap) |
| Coding tasks | 6/6 | 5/6 |
| Single-stream decode | 66.8 tok/s | 139 tok/s |
| Suite wall time | Not recorded for the Apple run | 48 minutes |
Setup — Apple: Apple M4 Max with 128 GB unified memory, MLX through mlx-serve, tested October 2026. The measured Apple mixed-pack scorecard identifies mlx-serve 26.10.2.
Setup — AMD: AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000, Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention enabled; 131K context, ubatch 2048, four parallel slots sharing one unified KV cache. Tested October 2026.
Evaluating Throughput and Task Quality
The decode gap is about 2.1× for the R9700 in this particular comparison. It answers a narrow question: how fast did these two tested machine-and-runtime combinations generate tokens with the same model recipe while the measured task scores stayed close? It does not say that Vulkan is always twice as fast as MLX, that the AMD machine is twice as useful, or that quality is identical outside these three suite dimensions.
The test suite contains MMLU-Pro 400 items, GSM8K 50 items, and six coding tasks. Both systems use temperature 0, a 2,048-token answer cap, and parallelism four. Decode speed is a single-stream median of five short runs. A “cap” marker means an answer was truncated by the output limit; the AMD result reports these counts and the Apple summary does not, so the score percentages alone do not show response-length behavior.
The suite is also a text-oriented evaluation. It does not measure image understanding, video, tool use, or agent completion quality even though the model supports multimodal inputs. For a practitioner, this points to two separate buying questions. If the target is low-latency single-stream text generation on this model, the R9700 result is markedly faster. If the target is task quality, the scores shown here are close, and the one-task coding difference matters more than a superficial “winner” label.
The Apple comparison reports 6/6 coding; the AMD run reports 5/6. A reader should inspect the task mix and actual responses before treating either as a meaningful quality advantage. The comparison is stronger than a comparison of unrelated model families because both machines ran the same mixed 4/8-bit weights recipe. It still has limits: the published summaries do not establish an independent replication across multiple machines, and the AMD suite wall time has no corresponding Apple figure here.
We also do not have a directly paired long-context prefill measurement for this exact quantization and prompt length on both systems. See Prefill and long context: Apple MLX vs Radeon R9700 for the context data and its comparability limits.
System Bottlenecks and End-to-End Workloads
Decode is also only one component of an interactive agent turn. Prompt construction, retrieval, tool execution, and any image-processing path add their own delays. If an agent spends more time waiting on a database or running a test command than generating text, a twofold decode improvement may barely change end-to-end completion time. Conversely, a coding assistant streaming long answers may make the throughput difference visible immediately. We need a representative end-to-end task set to quantify that effect.
If you are choosing a local box, measure the workload that resembles yours. Single-stream decode is relevant to one person waiting for an answer. Parallelism four describes a different operating condition. Image or video prompts add separate preprocessing and vision-encoder work. Longer prompts shift attention toward prefill and memory use. A useful follow-up would therefore repeat the same model, same quantized weights, same prompt set, and same concurrency at multiple input lengths, then report latency distributions and answer quality alongside throughput. The present result gives a clear baseline for that follow-up without claiming to settle every use case.
This comparison is part of a practical series: Prefill and long context: Apple MLX vs Radeon R9700, Quant formats compared: MLX mixed 4/8-bit vs GGUF K-quants and IQ, Power, noise and cost per million tokens, and Picking the right box for local agents.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
