What "local deployment" is actually measuring here
Start with the thing people skip. An AI model is, at the level that matters for a laptop, a frozen set of numbers — weights — plus an attention mechanism that reads your prompt token by token and predicts the next token until it stops. Quantizing it to 4 bits trades a slice of precision for a model small enough to sit in memory. Deploying it locally just means running the inference loop on your own silicon instead of someone else's GPU in a datacenter, and pointing an OpenAI-compatible client at it. Nothing exotic.
What is genuinely interesting to measure, once the obvious stuff works, is how the box behaves as the context window fills. Small prompts are easy. The hard part is the long, growing conversation: the kind a coding agent builds up over many turns, where most of the tokens are already-known prefix and only the new tail is novel. That is exactly what this KAT-Coder V2.5 probe stresses.
The model under test is KAT-Coder-V2.5-Dev-OptiQ-4bit from mlx-community — a Mixture-of-Experts code model (qwen3_5_moe architecture, roughly 36B parameters with about 3B active per forward pass) under an Apache 2.0 license. MoE is the whole reason a model this size is feasible on a laptop: only a small slice of the weights lights up for any given token.
How to deploy an AI model locally on an M4 Max
The mechanics are short. The MLX ecosystem ships mlx-lm, and the same weights run as a local OpenAI-compatible server, so any client that speaks that protocol — coding agents included, can point at localhost and treat your Mac like a cloud endpoint. The model card on Hugging Face shows the one-liner shape: load the quantized checkpoint, apply the chat template, generate. For a persistent server you bring up the MLX server against the model id and leave it listening.
That is the how-to-deploy-an-AI-model part in two paragraphs. The part worth reading is the timing.
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2 (Bolt requires experimental unreleased loader modifications); models: mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit; tested October 2026.
The four-turn growing-prefix probe
This is a single-request, four-turn probe where the prompt keeps growing and each turn reuses the prefix the previous turn already paid for. One clarification up front, because it trips people: 131K is the configured context capacity, not a 131K-token prompt. No request here ever sent a full 131K window. The turns climb to roughly 30K on a cold first request, 62K and 118K with the prior prefix cached, and a final short cached follow-up.
I held a 240-second first-token readiness budget. Every individual timing below is a measurement, not a guarantee, one observation on one machine on one day.
| Turn | Approx. context | State | Tuned TTFT (s) | Baseline TTFT (s) |
|---|---|---|---|---|
| 1 | ~30K | cold, full prefill | 28.1 | 21.4 |
| 2 | ~62K | prior prefix cached | 46.7 | 36.6 |
| 3 | ~118K | prior prefix cached | 127.7 | 113.7 |
| 4 | short follow-up | fully cached | 1.4 | 13.5 |
Read it carefully, because the tuned column is not "faster" across the board. On cold and growing turns the tuned configuration is slower, 28.1 versus 21.4 on the cold 30K request, 46.7 versus 36.6 at 62K, 127.7 versus 113.7 at 118K. Then on the fully cached follow-up it collapses from 13.5 seconds down to 1.4. The tuning buys you one very specific thing.
Tuned versus baseline: what the 8-bit KV cache changed
The baseline uses an 8-bit KV cache. The tuned run uses 8-bit KV too, but with the checkpoint stride and PLD (the draft-verify speculative decode path) disabled on novel text. The "118K passes" line means the third turn completed inside the 240-second budget, it does not mean 118K is the model's true ceiling or that it ran comfortably; 127.7 seconds on a budget of 240 is a pass with real room, but a slow pass.
Here is the honest interpretation, and it is the part I'd want if I were reading this cold: prefix reuse is doing the heavy lifting. When most of the window is a cached prefix, you only prefill the new tail, which is why a 118K request can land at all on a laptop. But that mechanism is about reuse. It tells you nothing reassuring about a cold full-window request, which would pay the entire attention cost up front. Do not read "it handled 118K cached" as "it would handle a 118K cold prompt." Those are different problems.
Checkpoint stride, PLD, and the cached follow-up
The single most dramatic number is the follow-up: 13.5 seconds baseline versus 1.4 seconds tuned. Restoring the SSM checkpoint stride is what brought the cached follow-up from 13.5 back down to 1.4. That is roughly a tenfold swing on a small, fully-cached request, and it is the clearest signal in the whole run that these cache settings are not cosmetic.
The catch is that same tuning is what made the cold and growing turns cost more. So you are looking at a genuine trade-off, not a free win: tune the cache and stride for agents that sit in long cached conversations issuing small follow-ups, and you pay a little on the turn where you first pour in the big prefix. There is no configuration in this test that wins on every turn. I'd take the trade for a coding-agent workload without much hesitation, but that's my call, and your workload may not look like mine.
A note on decode speed. These are short depth-decode samples, responses of roughly 60 tokens or fewer, so treat the decode figures as indicative, not stable sustained throughput. A short response never settles into a steady rate.
Accuracy and decode speed in the same window
The quality results, from the same machine:
- MMLU-Pro-400: 88.75%
- GSM8K-50: 98%
- Coding suite: 6/6
- Plain decode: 48.068 tok/s
- Peak memory: 25.4 GiB
The coding score is the one that fits the model's purpose, it is a code-tuned MoE, and 6/6 on a small coding set is encouraging without being a benchmark-wide verdict. GSM8K-50 at 98% on a fifty-item math slice is a small sample; fine as a sanity check, not a leaderboard claim.
Two structural notes behind the decode number. First, peak memory of 25.4 GiB against 128 GB of unified memory is why a long context is even plausible here, the weights leave a lot of room for the KV cache to grow into. Second, plain decode at 48.068 tok/s is genuine plain decode: the model has no native MTP (multi-token prediction) head, so there is no hardware-friendly speculative head to lean on for free speedups. What you measured is the model actually generating tokens one at a time.
Reading these numbers honestly
Pull the lessons apart and they are simple. MoE plus 4-bit quantization puts a 36B-parameter coding model in 25.4 GiB on a laptop, and that is the real headline. Prefix caching is what lets a laptop survive long, growing conversations, because it only prefills what is new. Cache and stride tuning is a trade between the cold-pour turn and the cached-follow-up turn, and you pick a side. And none of this generalizes past this box.
Related models in this series
- MiniCPM-V 4.7 (35B, A3B) at 131K on the M4 Max
- KAT-Coder V2.5 at 131K on the M4 Max
- Nemotron Bolt 74B at 131K on the M4 Max
- Qwen3.8 Flash-Next (IQ) at 131K on the M4 Max
- Four local MoE models at 131K on the M4 Max
- MiniCPM 35B vs Qwen3.8 Flash-Next at 131K
Model repositories tested or referenced:
- KAT-Coder V2.5: https://huggingface.co/mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit
- MiniCPM-V 4.7: https://huggingface.co/pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-4bit
- Nemotron 3.5 Bolt 74B: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
- Qwen3.8 Flash-Next (IQ): https://huggingface.co/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
