AI model local deployment: what the comparison tests
An AI model is a learned system that maps an input—such as text—to an output, using patterns encoded in its parameters. Local deployment means running those model weights on hardware you control, through a runtime that loads the model and accepts inference requests. The choice is not just about a model’s advertised context window: memory, runtime support, prompt processing, cached-prefix behavior, and the task itself can all matter.
This comparison examines two local mixture-of-experts (MoE) models on one Apple Silicon workstation. It is a four-turn, single-request growing-prefix probe, not a general speed ranking or a test of every possible workload. The configured context capacity was 131K tokens; that is a capacity setting, not a claim that any individual prompt contained 131K tokens. The successive requests were approximately 30K tokens cold, 62K with the prior prefix cached, 118K with the prior prefix cached, and a cached follow-up.
A 240-second first-token readiness budget was used as a practical boundary for these runs. Each timing below is an individual measurement, not a guarantee. In particular, reusing a prefix can make later requests behave differently from a cold request: prefix reuse does not establish that a cold full-window request will pass or meet the same latency budget.
Results: MiniCPM-V and Qwen3.8 Flash iQ
| Measure | MiniCPM-V 4.7 35B A3B | Qwen3.8-Flash-Next iQ |
|---|---|---|
| MMLU-Pro | 91.0% | 91.5% |
| GSM8K | 98% | 98% |
| Coding tasks | 6/6 | 6/6 |
| Peak memory | 23.2 GiB | 77.9 GiB |
| Growing-session TTFT, turns 1–4 | 21.4 / 36.6 / 103.2 / 3.3 s | 76.6 / 94.9 / 176.4 / 0.5 s |
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2; models: pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-4bit; ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw. Tested October 2026.
In these measured runs, MiniCPM reached first token sooner on the first three, increasingly long requests, while Flash iQ was quicker on the cached follow-up. All four reported TTFT measurements for each model were within the stated 240-second budget. The result should be read as a comparison of these exact runs and this growing-session pattern, not as proof that one model will be faster for every request. Flash’s third-turn result was 176.4 seconds; across mixed 4–8-bit Flash third-turn observations, the spread was 138.5 to 273.1 seconds. That spread is a reason not to claim repeatability or a universal timing threshold from a single sequence.
The quality figures are close on the reported suite: a half-point MMLU-Pro difference, matching GSM8K results, and the same number of coding tasks completed. These scores describe the listed evaluations only. They do not settle which model is more capable for an individual application, and benchmark scores should not be treated as interchangeable with production outcomes.
Memory is a substantial practical distinction in this test: the measured peaks were 23.2 and 77.9 GiB respectively. Memory headroom can influence what else can run alongside a model and how comfortably a long context can be accommodated. Peak memory from one run is not a universal hardware requirement; actual use depends on runtime, quantization, workload, and configuration.
Reading the depth and decode measurements
The observed TTFT pattern illustrates why session shape matters. A cold first request includes work that later requests with a shared prefix may reuse. The second and third turns add material while retaining prior context; the final request is a cached follow-up. These values therefore describe a sequence with prefix reuse, rather than four unrelated cold prompts. They cannot demonstrate that a request starting cold at the full configured capacity would succeed.
Short depth-decode samples—roughly 60 generated tokens or fewer—are indicative, not stable throughput measurements. Flash’s short plain-decode sample with MTP off measured 29.985 tokens per second; its best sample with MTP on measured 42.138 tokens per second, about 2.4 tokens per step. MTP ships with Flash. The observed gain in that short sample offers little evidence of improved behavior at depth, where the growing-session TTFT figures remain the more relevant measurements for this probe. We omit MiniCPM short-prompt prefill because it was noisy.
Neither model is an overall winner here. MiniCPM was faster at depth in this sequence and used less peak memory in the measurements, while Flash had the faster cached follow-up and an MTP feature whose short-sample result was higher. Different priorities—first-token latency, follow-up responsiveness, memory use, or a specific workload—can lead to different choices.
How to deploy an AI model locally
To deploy an AI model locally, first choose a model build and runtime that support one another and your hardware. Check the repository’s model card for supported formats, licensing, quantization, and runtime requirements. For example, the MiniCPM model repository describes a text-only MLX conversion and identifies mlx-serve v26.10.2 or later as its requirement. The Qwen repository used in this comparison is a separate MLX-served iQ conversion. These are specific artifacts, not interchangeable labels for every upstream model release.
Next, ensure that the machine has enough unified memory for the model and the intended workload, install a compatible runtime, and load the chosen repository identifier. A local runtime can expose an interface to a chat application or other client; confirm that the client and server agree on the model identifier and request format. Begin with a small prompt, then test representative prompts and session growth while observing first-token latency, peak memory, and output quality. Long-context capacity on a model card is not a substitute for testing the exact runtime and request pattern you intend to use.
For further context, see the related series: MiniCPM-V, KAT-Coder, Nemotron Bolt, Qwen3.8 Flash, four-model comparison, and MiniCPM vs Flash.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
