metaTitle: "Four Local Models in a Growing 131K-Context Session" ogTitle: "Four Local Models in a Growing 131K-Context Session" metaDescription: "Measured four-model growing-prefix benchmark on an M4 Max, with first-token latency and cached follow-ups. 131K is context capacity, not prompt length." ogDescription: "A measured comparison of four local models as one session grows toward 118K tokens and reuses cached prefixes. The 131K setting is context capacity, not prompt length."
AI edge infrastructure: what a growing local-model session reveals
Edge infrastructure is the compute, storage, networking, and software deployed near the people or devices that use a service, rather than relying entirely on distant centralized systems. Edge AI means running AI inference on that nearby infrastructure. IBM’s explainer, What Is Edge AI?, describes the broader concept: local processing can support lower-latency or connectivity-sensitive applications, while memory and compute remain deployment constraints. This four-model test is a narrow measurement of local inference behavior, not a general assessment of edge readiness.
The probe followed one request across four turns: a cold prompt of approximately 30K tokens, then growing prefixes of approximately 62K and 118K tokens with prior prefix content cached, and finally a cached follow-up. The configured context capacity was 131K tokens; this is capacity, not a measured prompt length. Successful prefix reuse also does not establish cold-request performance at the full window.
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2 (Bolt was benchmarked using a custom loader build and then dropped from our stack); models: pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-4bit; mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit; NVIDIA Nemotron-3.5 Bolt 74B-A9B 4-bit experimental conversion; ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw. Tested October 2026.
Evidence note: The figures and tuning observations are the authors’ measurements. Benchmark logs, evaluation outputs, and configuration files are not publicly available or linked; independent replication is therefore not implied.
First-token times across the growing prefix
The table summarizes individual-run first-token measurements for the four turns described above.
| Model | ~30K, cold | ~62K, cached prefix | ~118K, cached prefix | Cached follow-up |
|---|---|---|---|---|
| MiniCPM-V | 21.4 s | 36.6 s | 103.2 s | 3.3 s |
| KAT-Coder tuned | 28.1 s | 46.7 s | 127.7 s | 1.4 s |
| Nemotron Bolt | 55.5 s | 72.6 s | 164.7 s | 1.5 s |
| Qwen Flash iQ | 76.6 s | 94.9 s | 176.4 s | 0.5 s |
All four were within the 240-second first-token readiness threshold in these individual growing-session runs. The threshold is a test criterion recorded for this experiment, not a service-level guarantee. First-token time rose with prefix size in these observations, while the cached follow-up was shorter.
The model names identify distinct public repositories, but this is not a controlled ranking of model quality. In particular, Nemotron Bolt was an experimental conversion loaded with a custom loader build, measured, and then dropped from the stack. The measurements describe these particular configurations on this machine; they do not establish that the models are interchangeable or have equivalent capabilities. Related individual evaluations include MiniCPM-V 4.7 on Apple Silicon and Nemotron Bolt 74B-A9B.
Quality checks and memory footprint
The authors’ evaluation outputs for the measured set report MMLU-Pro results of 91.0% for MiniCPM-V, 88.75% for KAT-Coder, 81.75% for Nemotron Bolt, and 91.5% for Qwen Flash iQ. The same outputs report GSM8K at 98% for each model and a score of 6/6 on the coding check for each. These are limited checks, not comprehensive measures of general capability, and differences need not transfer to other tasks.
The workload measurements recorded peak memory of 23.2 GiB for MiniCPM-V, 25.4 GiB for KAT-Coder, 48.1 GiB for Bolt, and 77.9 GiB for Flash iQ. These are observed peaks for the tested workloads and configurations, not estimates derived from model weight size. Context, cache, and runtime behavior affect the memory a workstation must accommodate.
This test illustrates a practical tension: prefix caching made the follow-up much faster, while first-token times increased across the growing-prefix turns. The measurements cover one machine, software build, and request sequence; they do not establish performance with multiple users, other hardware, or production workloads, or whether local AI deployment is ready in general.
What tuning changed—and what remains uncertain
The experiment configuration used an 8-bit KV cache and a 4 GB prefix cache for all four models. KAT-Coder additionally used SSM checkpoint stride 2048 and a maximum of 64 checkpoints, with PLD disabled. MiniCPM-V, Bolt, and Flash iQ used the common cache settings; Flash iQ also used MTP. These are experiment settings, not universal recommendations.
Eight-bit KV was faster than KV off in the measured KAT-Coder and Bolt comparisons; that result was not established for all four models. MiniCPM-V and Flash iQ did not have clean A/B comparisons. Disabling PLD helped KAT-Coder on novel text in a one-run comparison, which is suggestive rather than conclusive. Changing KAT-Coder’s SSM checkpoint stride fixed a slow follow-up measurement, reducing it from 13.5 seconds to 1.4 seconds, with a growth-latency trade-off. For Bolt, chunk size 4096 was fastest among the sizes tested; this does not establish that it is best beyond those trials.
MTP should be considered only when a trained, compatible head ships with a model. In the Flash iQ short-prompt sample, MTP reached 42.138 tokens per second versus 29.985 for plain decoding, at about 2.4 tokens per step. It showed little benefit at depth. These short depth-decode samples—responses of roughly 60 tokens or fewer—are indicative, not stable throughput benchmarks. A mixed t3 Flash result ranged from 138.5 to 273.1 seconds in the authors’ runs; this spread is an observation from that test, not evidence of reliability.
For readers asking what edge infrastructure means in practice, it is more than a device running a model: it includes the nearby compute platform and the software and operating conditions needed to serve a workload. This benchmark addresses only one part of that picture, first-token latency and memory for a growing, cached request on one Apple Silicon workstation. It does not test network behavior, multiple simultaneous users, power consumption, production availability, or other hardware.
Disclaimer. These results come from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
