title: "Four Models Measured Across a Growing Context on M4 Max" metaTitle: "Four Models Across a Growing Context on M4 Max" metaDescription: "First-token waits and peak memory for four models across cached-prefix turns on an M4 Max."
What this local-model probe measured
A context limit is not a promise that every request at that length will feel usable. We tested what happens as one conversation grows, with a prior prefix available for reuse, and measured the wait for the first generated token. The distinction matters: a cached continuation and a cold request that starts with an entire long prompt are different workloads.
The probe had four turns: approximately 30K tokens, then 62K with the earlier prefix cached, then 118K with the prior prefix cached, followed by a cached follow-up. The configured context capacity was 131K. That is the capacity setting, not a 131K prompt in the test. A 240-second first-token readiness budget was used as a practical boundary. These are individual timings, not guarantees for future requests. (sourceRef: four-model measurement report)
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2 (Bolt requires experimental unreleased loader modifications); models: pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-4bit; mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit; NVIDIA Nemotron-3.5 Bolt 74B-A9B 4-bit experimental conversion; ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw; tested October 2026. The model repositories document the model variants; the measured setup and results are from the probe. (sourceRef: four-model measurement report)
Four local models, four growing-prefix turns
| Model | ~30K first token | ~62K cached-prefix | ~118K cached-prefix | Cached follow-up | Peak memory |
|---|---|---|---|---|---|
| MiniCPM-V | 21.4 s | 36.6 s | 103.2 s | 3.3 s | 23.2 GiB |
| KAT-Coder tuned | 28.1 s | 46.7 s | 127.7 s | 1.4 s | 25.4 GiB |
| Nemotron Bolt | 55.5 s | 72.6 s | 164.7 s | 1.5 s | 48.1 GiB |
| Qwen Flash iQ | 76.6 s | 94.9 s | 176.4 s | 0.5 s | 77.9 GiB |
(sourceRef: four-model measurements, including first-token timings and peak memory)
Each model stayed within the 240-second budget in these measured growing-session runs. That is a narrow statement about this test, not a service-level promise. Prefix reuse does not mean a cold full-window request passes: we did not establish that by making this probe. Nor does the configured 131K capacity mean we sent a 131K prompt. (sourceRef: four-model measurement report)
The table makes the trade-off visible. MiniCPM had the shortest first-token waits in the first three stages. Qwen’s cached follow-up was quickest, but its peak memory was also highest among these four measurements. KAT’s tuned configuration returned the cached follow-up in 1.4 seconds. Bolt also returned quickly in that step, but it depended on experimental, unreleased loader modifications and was dropped from consideration. A result can be interesting and still be too awkward to treat as a deployable option. (sourceRef: four-model measurement report; sourceRef: KAT-Coder 131K report; sourceRef: Nemotron Bolt 131K report)
Memory here means measured peak memory, not model weight size. Don’t compare the GiB column with a model’s nominal weight footprint and assume the difference is unexplained waste; the figures answer a different question, namely how much memory this run peaked at under the test setup. (sourceRef: four-model measurement report)
Task scores are a separate view
The reported MMLU-Pro scores were 91.0% for MiniCPM-V, 88.75% for KAT-Coder, 81.75% for Bolt, and 91.5% for Flash iQ. Each recorded 98% on GSM8K and 6/6 on the coding sample. These outcomes are useful context, but the small coding set is not a broad estimate of programming ability. They also do not answer the serving question: a model can score well and still have a long wait before it starts responding at depth. The model variants are documented in the MiniCPM-V, KAT-Coder, Nemotron Bolt, and Qwen Flash Next iQ model cards; the scores here are reported test results, not independently validated benchmark claims. (sourceRef: four-model benchmark results)
Short depth-decode samples, roughly 60 generated tokens or fewer, are indicative rather than stable throughput measurements. Treat them as a glimpse, not a dependable ranking of sustained generation speed. (sourceRef: four-model measurement report)
Tuning findings, and what they do not establish
On the measured KAT and Bolt configurations, 8-bit KV was faster than KV off. That evidence is not established for all four models. MiniCPM and Flash iQ lacked a clean A/B comparison, so extending the result to them would overstate what we measured. (sourceRef: KAT-Coder 131K report; sourceRef: Nemotron Bolt 131K report; sourceRef: four-model measurement report)
For KAT, disabling PLD helped on novel text in a one-run comparison. Its SSM checkpoint stride change had a much larger observed follow-up effect: the cached follow-up moved from 13.5 seconds to 1.4 seconds, with a growth-latency trade-off. This is a measured configuration-specific result, not a universal setting recommendation. Bolt’s chunk size of 4096 was best among the sizes tested, though the loader requirement remains a substantial caveat. (sourceRef: KAT-Coder 131K report; sourceRef: Nemotron Bolt 131K report)
Multi-token prediction (MTP) is relevant only when a model ships with a trained, compatible head. For Flash iQ, a short-prompt sample measured 42.138 tok/s with MTP versus 29.985 tok/s plain, at about 2.4 tokens per step. It helped little at depth. In the mixed t3 result, Flash’s spread was 138.5 versus 273.1 seconds. That spread is not evidence of reliability; it is a warning not to turn a handful of measurements into a stability claim. (sourceRef: Qwen Flash Next iQ 131K report)
Related measurements
Related series articles include MiniCPM-V 4.7 35B A3B, KAT-Coder v2.5, Nemotron Bolt 74B, Qwen3.8 Flash Next iQ, the four-model MoE roundup, and the MiniCPM vs. Qwen3.8 Flash Next comparison. For a live sibling measurement, see MiniCPM and Qwen3.8 Flash at 131K on M4 Max.
These related articles provide measurement context; the timings, peak-memory readings, benchmark samples, and tuning observations in this article are results from the stated probe and should be interpreted with its setup and caveats. (sourceRef: four-model measurement report)
Practical conclusion
If the workload is a long-running local conversation, measure the continuation path, not only an isolated prompt. Keep cold-start and cache behavior separate in your notes. And don’t plan around a single successful deep request as if it were a latency guarantee. In this run, every model met the chosen 240-second first-token budget across the growing-prefix turns, but the models differed sharply in time, memory, and practical setup friction. (sourceRef: four-model measurement report)
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
