When engineers evaluate local model performance across different hardware accelerators, headline metrics can easily mislead. The available evidence shows Apple MLX MiniCPM-V-4.7 4-bit prefill of roughly 1,184 tok/s at a 4K prompt, falling to about 539 tok/s at 65K; a Radeon R9700 MiniCPM mixed 4/8-bit run measured about 4,100 tok/s around a 24K prompt. These are useful measured points, not a direct speed ratio: weights, prompt lengths, and run recipes do not match. We found no same-quantization, same-prompt-length prefill pair for these two systems.
Understanding Prefill, Decode, and AI Model Local Deployment Realities
Prefill is the work a model does to process the input before generating its first answer token. Decode is the work of generating the answer one token at a time. A short prompt can make decode speed the most visible number; a long document or codebase can make prefill time and time-to-first-token dominate the user experience. Context length also affects decode throughput as the model attends to more tokens. A useful comparison therefore keeps prompt size and model fixed instead of comparing one headline tok/s figure from each machine.
In the rapidly evolving landscape of 2025 and 2026, mastering ai model local deployment requires looking beyond synthetic speed tests to examine how token ingestion scales with prompt length, hardware memory architecture, and quantization choices.
Measured Points Available
| Machine and model | Prompt length in the recorded run | Prefill | Decode | Notes |
|---|---|---|---|---|
| M4 Max, MLX MiniCPM-V-4.7 35B-A3B 4-bit | 4.2K tokens | 1,184 tok/s | 125.6 tok/s | Needle check passed |
| M4 Max, same 4-bit model | 16.5K | 1,155 tok/s | 118.8 tok/s | Needle check passed |
| M4 Max, same 4-bit model | 33.1K | 937 tok/s | 104.4 tok/s | Needle check passed |
| M4 Max, same 4-bit model | 66.1K | 539 tok/s | 58.9 tok/s | Needle check passed |
| Radeon R9700, MiniCPM-V-4.7 mixed 4/8-bit GGUF | about 24K | about 4,100 tok/s | 121 tok/s | AMD long-context point |
Setup — Apple: Apple M4 Max with 128 GB unified memory, MLX through mlx-serve, tested October 2026. The 4-bit MLX run used mlx-serve 26.10.2. Setup — AMD: AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000, Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention enabled; 131K context, ubatch 2048, four parallel slots sharing one unified KV cache. Tested October 2026.
Context Scaling, Needle Retrieval, and Quantization Limits
The Apple points come from a separate ceiling-style context run, not the standard-v2 quality suite. Actual prompt token counts are shown because the nominal rung (for example, 16K) and the final tokenized prompt are not always identical. At each recorded rung through 65K, the needle-retrieval check passed. That check supports the claim that the test prompt could be processed and the target was recovered; it does not show that every long-context task is reliable. The next 131K prompt exceeded the recorded server context ceiling, so there is no passing 131K result for this run.
The Apple trend is clear in this run: prefill throughput drops from about 1,184 tok/s around 4K to 539 tok/s around 66K, while decode drops from 125.6 to 58.9 tok/s. Long input therefore changed both the time to ingest the prompt and the rate of generating after it. These values belong to the tested 4-bit MiniCPM package; they should not be attached to the mixed 4/8-bit Apple package used in the standard-v2 headline comparison.
The Radeon figure is a distinct recorded point for mixed 4/8-bit GGUF near a 24K input. It cannot be divided by the 16K or 32K Apple result and presented as a hardware uplift: that arithmetic would mix prompt lengths and quantization recipes. We also do not have the same MiniCPM prompt token sequence on both systems in our runs.
For contrast, a separate Apple M4 Max run of LFM2.5-24B-A2B mixed weights recorded prefill from about 2,645 tok/s at roughly 4K to 1,674 tok/s near 64K; that shows how MLX prefill falls with context for another model, but it is not an AMD comparison because we did not run LFM2.5 on the R9700. The LFM2.5-24B-A2B card describes a 24B model with 2.45B active parameters and a 131,072-token context. Its published training notes include a dedicated 300B-token long-context phase at 131K, which makes the model a useful example for context testing but does not make its measurements interchangeable with MiniCPM.
The model cards provide useful context for interpreting these measurements. MiniCPM-V-4.7 35B-A3B is a multimodal MoE model; its card documents image, multi-image, and video capabilities, and recommends instruct-mode sampling at temperature 0.7, top-p 0.95, and top-k 100. The local prefill test is a text-prompt throughput measurement, so those broader capabilities and sampling recommendations are not tested here. MiniCPM-V-4.7 model card.
How to Deploy AI Model Workloads and Answer What Is Agentic AI
When software development companies plan production architectures, they frequently ask how to deploy ai model workloads locally for security and latency reasons. Deploying models locally involves selecting a suitable serving runtime like mlx-serve or llama.cpp, configuring context limits and batch sizes, and matching quantization precision to hardware memory constraints.
Within this ecosystem, developers often encounter the question: What is Agentic AI? Software development companies define agentic AI as autonomous systems capable of executing multi-step workflows, invoking tools, and reasoning over extensive codebases or documentation without constant human intervention. Supporting these persistent agentic loops requires high prefill throughput and deep context windows so that agents can ingest entire repositories instantly.
Practical Lessons for Local Inference and Future Benchmarking
For an engineer planning a retrieval-augmented generation or coding workflow, the practical lesson is to benchmark at the prompt size you expect to serve. A short-prompt benchmark can make a system look responsive while a long context delays the first token. Record prefill and decode separately, preserve the exact prompt token count, report whether retrieval checks passed, and measure memory headroom at each rung.
If the same model is tested on multiple accelerators, align the quantized weights, prompt, server version where possible, batch settings, and concurrency. Then repeat the run enough times to report variation, not only a single point.
The missing like-for-like measurement is concrete: run the same MiniCPM mixed 4/8-bit weights on both systems at identical 4K, 16K, 32K, and 64K prompts, with the same prompt content, decoding settings, and single-request conditions. Until then, we can report a set of platform-specific observations, but not a clean Apple-versus-AMD prefill verdict.
For short-stream results see Same model, two machines: Mac Studio MLX vs Radeon R9700 Vulkan; for format trade-offs see Quant formats compared: MLX mixed 4/8-bit vs GGUF K-quants and IQ.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
