title: "Qwen3.8-Flash-Next Cached-Prefix Probe: 118K Context on M4 Max" metaTitle: "Qwen3.8-Flash-Next Cached-Prefix Probe on M4 Max"
A local model deployment is not established by a context-window number on a model card. It has to load, accept the intended prompt, and return useful output within a practical latency and memory budget. This article reports a four-turn growing-prefix probe of Qwen3.8-Flash-Next in an Apple Silicon setup. The probe measured approximately 30K tokens on a cold request, then approximately 62K and 118K with the previous prefix cached, followed by a cached follow-up. The configured context capacity was 131K; that is a capacity setting, not evidence that a 131K prompt was submitted.
This is one machine and one test suite, not a general ranking. The available public model repository identifies the model artifact and its serving-oriented packaging; it does not independently verify the latency or benchmark measurements. The measurement values below are reported as test results, with their limits stated directly.
What the four-turn cached-prefix probe measured
The test used a growing-prefix sequence: each subsequent turn reused the accumulated earlier context rather than treating every request as a new cold prompt. In that setting, TTFT (time to first token) is affected by the request’s new work as well as prefix reuse. It is therefore useful to distinguish cold prompt processing, cached-prefix processing, and the follow-up after the long prefix has already been established.
| Turn | Probe condition | Approximate input position | iQ 8-bit KV TTFT |
|---|---|---|---|
| 1 | Cold request | 30K | 76.6 s |
| 2 | Prior prefix cached | 62K | 94.9 s |
| 3 | Prior prefix cached | 118K | 176.4 s |
| 4 | Cached follow-up | Follow-up | 0.5 s |
These are individual observed timings, not service-level guarantees. The configured context capacity was 131K, but this probe’s measured long-prefix point was about 118K. Prefix reuse can make a follow-up request behave very differently from a cold request; it does not show that a cold full-window request will pass. For operational planning, the first-token readiness budget was 240 seconds. That budget is a test/acceptance boundary, not a promise that all prompts will complete within it.
Setup. Apple Silicon workstation (M4 Max, 128 GB unified memory); mlx-serve v26.10.2 (Bolt requires experimental unreleased loader modifications); models: ddalcu/Qwen3.8-Flash-Next-MLX-Serve-iQ-MLX-4.7bpw; tested October 2026.
Comparing the two KV configurations without overclaiming
The mixed 4–8 baseline recorded TTFT values of 75.8, 93.5, 138.5, and 0.6 seconds across the corresponding four turns. The iQ 8-bit KV probe recorded 76.6, 94.9, 176.4, and 0.5 seconds. On these observations, the first two turns were close, the third-turn iQ result was slower, and the cached follow-ups were both sub-second. These figures do not establish that iQ is universally faster or slower.
The mixed baseline’s third-turn result has an important qualification: a same-configuration rerun measured 273.1 seconds, rather than 138.5 seconds. A single passing run is not robust evidence of repeatable performance. The spread illustrates why a deployment decision should be based on repeated trials and the actual workload, rather than selecting the best observed timing. In particular, it would be misleading to present the 138.5-second result as a stable expectation or to imply that KV format alone explains every difference.
The purpose of the comparison is practical: cache behavior and request history can materially change observed latency at depth. If an application receives independent long prompts, it should not assume the benefit seen on a cached continuation. If it builds a conversation or agent context incrementally, prefix reuse may matter—but the exact behavior still depends on serving implementation, memory pressure, prompt structure, and workload.
Short decoding results are not a throughput guarantee
Short depth-decode samples—roughly 60-token or fewer responses—were indicative, not stable throughput measurements. In the plain cell with MTP off, a short prompt measured 29.985 tokens per second. The best observed cell, with MTP on, measured 42.138 tokens per second. The best prefill measurement was 187.334 tokens per second, and the reported decoding behavior was about 2.4 tokens per step.
These numbers describe specific short samples and settings. They should not be extrapolated into sustained generation throughput or long-answer completion time. MTP helped on short prompts in these samples, with little help at depth. The result is a useful clue for testing an interactive workload, not a universal performance claim. A serious deployment evaluation should include the response lengths and concurrency patterns expected in production, not only short decode snippets.
Peak reported memory was 77.9 GiB. That figure is relevant to a 128 GB unified-memory workstation but should not be treated as a universal footprint: model loading, cache growth, system activity, and serving settings all affect available headroom. Likewise, a configured 131K context is not equivalent to a guarantee that every request near that limit can be admitted and processed within a desired latency.
Quality indicators and what they do—and do not, say
The reported evaluation included MMLU-Pro at 91.5%, GSM8K at 98%, and coding at 6/6. These are results from the stated test suite, not a complete characterization of model quality. They do not imply that the model is the best choice for a particular application, nor do they replace task-specific evaluation, safety review, or verification of generated output. Small benchmark sets and broad benchmark scores can conceal weaknesses in the domain an application actually needs.
The public model repository is useful for identifying the artifact and its intended MLX serving context. It includes model-use information and interoperability examples. That repository is not the source of the machine-specific TTFT, memory, or benchmark observations in this report. For serving-version context, consult the mlx-serve releases; release notes likewise do not independently reproduce the measurements described here. The measurement claims are the results of the specified probe and should be interpreted with the setup and caveats in this article, rather than inferred from repository documentation.
How this Qwen cached-prefix probe informs local deployment
A useful local deployment process begins with the intended workload. Identify typical and worst-case prompt lengths, whether requests are continuations that can reuse a prefix, desired first-token latency, output lengths, and memory headroom. Then choose a model artifact and serving stack compatible with the target hardware. This probe used an MLX-oriented model on Apple Silicon; its findings should not be generalized to other accelerators or serving software.
Next, run a controlled test sequence that separates cold requests from cached continuations. Record actual prompt sizes, TTFT, output length, memory, and serving configuration. Repeat long-context measurements: the mixed baseline rerun demonstrates how one unusually favorable result can distort conclusions. Test both the first request and subsequent requests, since an application may encounter either. Keep a fixed readiness budget, such as the 240-second first-token budget used here, but report observed latency separately from that budget.
For developers exploring agentic systems, this kind of local inference can be one component of an agent workflow, but it does not by itself constitute Agentic AI. An agentic application typically combines a model with instructions, tools or actions, state, and control logic. The model’s context capacity and short-answer speed are only parts of that system; tool reliability, error handling, privacy, and the consequences of actions need separate evaluation. Local deployment may support data-control goals, but the operational design still determines what is logged, exposed, or acted upon.
Start with a small end-to-end task, test realistic prompts, and measure repeated runs before depending on a latency target. A developer choosing between local and hosted inference should weigh privacy and control against hardware cost, model quality, maintenance, and actual response-time needs. Results from a single M4 Max can inform that investigation, not settle it for other teams or workloads.
Related local model measurements
This Qwen probe belongs to a broader 131K-context local-model series. Related articles: MiniCPM-V 4.7 35B A3B, KAT-Coder v2.5, Nemotron Bolt 74B, Qwen3.8 Flash Next iQ, the four-model MoE roundup, and the MiniCPM vs. Qwen3.8 Flash Next comparison. Another related measurement is MiniCPM and Qwen3.8 Flash at 131K on M4 Max.
Public model repositories for the related systems: MiniCPM-V 4.7, KAT Coder V2.5, Nemotron 3.5 Bolt 74B, and Qwen3.8 Flash Next iQ. These links identify public model repositories; they are not evidence for the machine-specific measurements summarized above.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
Sources and evidence notes
- Model identity and public artifact: Qwen3.8-Flash-Next MLX Serve iQ model repository.
- Serving release context: mlx-serve releases.
- Original model family reference: Qwen3.8-Flash-Next.
The source links above support artifact and software context; they do not independently substantiate the measured TTFT, decode, memory, or benchmark values. Those values are reported from the specified test results and retain the single-machine and repeatability caveats throughout.
