ProBackend
News & Analysis
Oct 19, 20265 min read

Four Models Measured Across a Growing Context on M4 Max

Four Models Measured Across a Growing Context on M4 Max

A four-model M4 Max probe reports first-token times and peak memory across growing cached sessions.

What this local-model probe measured

A context limit is not a promise that every request at that length will feel usable. We tested what happens as one conversation grows, with a prior prefix available for reuse, and measured the wait for the first generated token. The distinction matters: a cached continuation and a cold request that starts with an entire long prompt are different workloads.

The probe had four turns: approximately 30K tokens, then 62K with the earlier prefix cached, then 118K with the prior prefix cached, followed by a cached follow-up. The configured context capacity was 131K. That is the capacity setting, not a 131K prompt in the test. A 240-second first-token readiness budget was used as a practical boundary. These are individual timings, not guarantees for future requests.

Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2; models: pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-4bit; mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit; NVIDIA Nemotron-3.5 Bolt 74B-A9B 4-bit experimental conversion; ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw; tested October 2026. The model repositories document the model variants; the measured setup and results are from the probe.

Four local models, four growing-prefix turns

Model~30K first token~62K cached-prefix~118K cached-prefixCached follow-upPeak memory
MiniCPM-V21.4 s36.6 s103.2 s3.3 s23.2 GiB
KAT-Coder tuned28.1 s46.7 s127.7 s1.4 s25.4 GiB
Nemotron Bolt55.5 s72.6 s164.7 s1.5 s48.1 GiB
Qwen Flash iQ76.6 s94.9 s176.4 s0.5 s77.9 GiB

Each model stayed within the 240-second budget in these measured growing-session runs. That is a narrow statement about this test, not a service-level promise. Prefix reuse does not mean a cold full-window request passes: we did not establish that by making this probe. Nor does the configured 131K capacity mean we sent a 131K prompt.

The table makes the trade-off visible. MiniCPM had the shortest first-token waits in the first three stages. Qwen’s cached follow-up was quickest, but its peak memory was also highest among these four measurements. KAT’s tuned configuration returned the cached follow-up in 1.4 seconds. Bolt also returned quickly in that step, but it needed a custom loader, so it was benchmarked and then dropped from our stack. A result can be interesting and still be too awkward to treat as a deployable option.

Memory here means measured peak memory, not model weight size. Don’t compare the GiB column with a model’s nominal weight footprint and assume the difference is unexplained waste; the figures answer a different question, namely how much memory this run peaked at under the test setup.

Task scores are a separate view

The reported MMLU-Pro scores were 91.0% for MiniCPM-V, 88.75% for KAT-Coder, 81.75% for Bolt, and 91.5% for Flash iQ. Each recorded 98% on GSM8K and 6/6 on the coding sample. These outcomes are useful context, but the small coding set is not a broad estimate of programming ability. They also do not answer the serving question: a model can score well and still have a long wait before it starts responding at depth. The model variants are documented in the MiniCPM-V, KAT-Coder, Nemotron Bolt, and Qwen Flash Next iQ model cards; the scores here are reported test results, not independently validated benchmark claims.

Short depth-decode samples, roughly 60 generated tokens or fewer, are indicative rather than stable throughput measurements. Treat them as a glimpse, not a dependable ranking of sustained generation speed.

Tuning findings, and what they do not establish

On the measured KAT and Bolt configurations, 8-bit KV was faster than KV off. That evidence is not established for all four models. MiniCPM and Flash iQ lacked a clean A/B comparison, so extending the result to them would overstate what we measured.

For KAT, disabling PLD helped on novel text in a one-run comparison. Its SSM checkpoint stride change had a much larger observed follow-up effect: the cached follow-up moved from 13.5 seconds to 1.4 seconds, with a growth-latency trade-off. This is a measured configuration-specific result, not a universal setting recommendation. Bolt’s chunk size of 4096 was best among the sizes tested, though the loader requirement remains a substantial caveat.

Multi-token prediction (MTP) is relevant only when a model ships with a trained, compatible head. For Flash iQ, a short-prompt sample measured 42.138 tok/s with MTP versus 29.985 tok/s plain, at about 2.4 tokens per step. It helped little at depth. In the mixed t3 result, Flash’s spread was 138.5 versus 273.1 seconds. That spread is not evidence of reliability; it is a warning not to turn a handful of measurements into a stability claim.

Related series articles include MiniCPM-V 4.7 35B A3B, KAT-Coder v2.5, Nemotron Bolt 74B, Qwen3.8 Flash Next iQ, the four-model MoE roundup, and the MiniCPM vs. Qwen3.8 Flash Next comparison. For a live sibling measurement, see MiniCPM and Qwen3.8 Flash at 131K on M4 Max.

These related articles provide measurement context; the timings, peak-memory readings, benchmark samples, and tuning observations in this article are results from the stated probe and should be interpreted with its setup and caveats.

Practical conclusion

If the workload is a long-running local conversation, measure the continuation path, not only an isolated prompt. Keep cold-start and cache behavior separate in your notes. And don’t plan around a single successful deep request as if it were a latency guarantee. In this run, every model met the chosen 240-second first-token budget across the growing-prefix turns, but the models differed sharply in time, memory, and practical setup friction.

Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.

this local-model probe measuredthis local-model probe measured

Abstract visualization of layered data blocks expanding toward a localAbstract visualization of layered data blocks expanding toward a local