What an AI model local deployment probe measures
An AI model is a learned system that maps an input to an output. Local deployment means running that model on hardware under your control, rather than treating a hosted endpoint as the only way to use it. But “it runs locally” is not a useful performance claim by itself. Context length, time to first token, cache behavior and output length all change what a user experiences.
This probe followed a single request across four turns, growing the prefix and reusing the prior prefix cache. It is a narrow test of that pattern—not a claim that every long request will behave the same way. In particular, a 131K configured context capacity is not a 131K prompt, and a cached continuation is not evidence that a cold request of the full window will pass.
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2 (Bolt was benchmarked using a custom loader build and then dropped from our stack); models: ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw; tested October 2026.
AI model local deployment: four growing-prefix turns
The point of the test is the shape of the request. The first turn begins cold at roughly 30K tokens; subsequent turns carry forward the prior prefix, growing to about 62K and then 118K before a cached follow-up. The results below are measurements from this run, not service-level guarantees.
| Turn | Request pattern | Approximate prefix | Time to first token |
|---|---|---|---|
| 1 | Cold first request | 30K | 76.6 s |
| 2 | Prior prefix cached | 62K | 94.9 s |
| 3 | Prior prefix cached | 118K | 176.4 s |
| 4 | Cached follow-up | Reused prefix | 0.5 s |
The team’s readiness budget for first token was 240 seconds. The listed ~118K cached-prefix runs—138.5 seconds for mixed 4–8 and 176.4 seconds for iQ—were below that budget, while the 273.1-second mixed-configuration rerun exceeded it. That contrast matters more than a neat pass/fail headline: one passing run is not robust evidence of repeatable readiness.
The 131K figure is the configured context capacity, not the length of a single prompt in this test. Prefix reuse helped the measured follow-up, but it does not imply a cold full-window request passes. Those are different workloads, and conflating them turns a cache result into a claim the measurement did not make.
Comparing the two KV configurations
The mixed 4–8 baseline recorded time-to-first-token figures of 75.8, 93.5, 138.5 and 0.6 seconds across the same four-turn progression. The iQ 8-bit KV probe recorded 76.6, 94.9, 176.4 and 0.5 seconds. The first two turns are close; at the third, the mixed run was faster in the initial measurement. The follow-up times were both under a second.
There was also a same-configuration mixed 4–8 rerun at turn three: 273.1 seconds, versus 138.5 seconds in the earlier run. That spread is a warning, not a footnote. A single passing run cannot establish stable performance, and the iQ figures do not show a universal speed advantage. Different runs can vary; we should resist promoting one favorable number as the behavior to expect.
Quality and memory observations
The recorded evaluation results were 91.5% on MMLU-Pro and 98% on GSM8K. Coding evaluation passed 6 of 6 tasks. Peak memory was 77.9 GiB. These are results from the reported setup and test suite, not a broad ranking against other models or a guarantee about performance under a different prompt mix.
The figures answer different questions. Benchmark scores describe the evaluated tasks; peak memory describes this run’s observed footprint; time to first token describes the wait before output began. None can stand in for the others. For local deployment decisions, the intended workload matters: a long prefix-heavy session is not the same as a short interactive prompt, and neither is fully captured by one score.
Short decode samples need a narrow reading
On a short prompt, the plain cell with MTP off measured 29.985 tokens per second. The best cell with MTP on measured 42.138 tokens per second. Best prefill was 187.334 tokens per second, and the observed rate was about 2.4 tokens per step.
These are short depth-decode samples, roughly 60-token responses or fewer. They are indicative, not stable throughput measurements. MTP helped on short prompts in these cells, with little help at depth. Don’t turn that into a blanket claim about long-form generation: the sample is deliberately small, and the measured advantage belongs to those conditions.
How to deploy an AI model, and what this test answers
A practical local deployment starts by choosing a model build and inference stack compatible with the target hardware, then testing the workload you actually expect: cold starts, growing conversations, cached follow-ups, and the output lengths your users need. Record the settings alongside the timings. Otherwise, a table can look precise while leaving out the conditions that explain it.
Here, the iQ model card is publicly available on Hugging Face. The measurements are not a general endorsement of that model, nor proof that a 131K prompt is practical just because the configured capacity is 131K. The cold start and cached continuation should be evaluated separately, especially when first-token readiness has a hard budget.
For context across this local-model series, see MiniCPM-V 4.7 at 131K, KAT-Coder V2.5 at 131K, Nemotron Bolt at 131K, four local MoE models at 131K, and MiniCPM 3.5B vs Qwen Flash Next. A separate Qwen Flash Next model page gives the public repository details.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.