Key takeaway: Quantization shifts the balance among memory footprint, speed, and task scores, but these MLX and Radeon results are not one controlled quant sweep. The headline is not that one format wins. It’s that you need to know exactly what was measured before choosing what to test next.
Quantization stores model weights at lower numerical precision to reduce their memory footprint and data movement. “4-bit” alone tells you very little about the complete package. Grouping schemes, tensor exclusions, calibration, mixed-precision layers, backend kernels, and speculative decoding can all affect results.
GGUF K-quants and IQ formats are llama.cpp-compatible quantization families. MLX packs use Apple’s MLX format and may mix bit widths across layers or tensors. A 4.7-bpw IQ pack and a mixed 4/8-bit pack are not interchangeable just because their average bits per weight look close.
AI model local deployment: measured results
| Platform and measured pack | MMLU-Pro | GSM8K | Coding | Decode tok/s | What is comparable |
|---|---|---|---|---|---|
| Apple MLX MiniCPM-V-4.7 35B-A3B mixed 4/8-bit | 90.0% | 100% | 6/6 | 66.8 | Same model family as the Apple 4-bit row below |
| Apple MLX MiniCPM-V-4.7 35B-A3B tuned 4-bit | 91.0% | 98% | 6/6 | 50.6 | Same suite; tuning and quant differ from mixed pack |
| AMD GGUF MiniCPM-V-4.7 mixed 4/8-bit | 90.0% (3 at cap) | 100% (0 at cap) | 5/6 | 139 | Same mixed weights recipe as Apple headline, different runtime |
| AMD GGUF Qwen3.8-27B UD-Q4_K_XL + MTP depth 2 | 86.75% (31 at cap) | 94% (1 at cap) | 5/6 | 62 | One member of the AMD Qwen sweep |
| AMD GGUF Qwen3.8-27B UD-Q4_K_XL + MTP depth 4 | 87.5% (30 at cap) | 92% (3 at cap) | 4/6 | 56 | Draft depth changes too |
| AMD GGUF Qwen3.8-27B UD-Q6_K_XL + MTP depth 4 | 86.25% (32 at cap) | 96% (1 at cap) | 4/6 | 51 | Higher K quant, same listed draft depth as Q4 row |
| AMD GGUF Qwen3.8-27B GSQ-RCO IQ3_S + built-in MTP depth 2 | 85.25% (39 at cap) | 92% (3 at cap) | 5/6 | 57 | Different IQ pack and built-in draft |
| Apple MLX Qwen3.8-Flash iQ-MLX 4.7 bpw tuned | 91.5% | 98% | 6/6 | 42.1 | Related Qwen family, different model variant and suite pack |
Setup — Apple: Apple M4 Max with 128 GB unified memory, MLX through mlx-serve, tested October 2026. The listed MLX standard-v2 records used mlx-serve 26.10.2.
Setup — AMD: AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000, Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention enabled; 131K context, ubatch 2048, four parallel slots sharing one unified KV cache. Tested October 2026.
The table is useful as a map of candidates, not a podium. Rows vary by model, weight recipe, and draft settings. Decode rate is only one part of a local deployment decision; these measurements don’t report a universal memory-versus-quality curve.
The cleanest comparison is still limited
The closest format-related comparison is within Apple’s MiniCPM rows. The mixed 4/8-bit package generated at 66.8 tok/s and reached 90.0% on MMLU-Pro, 100% on GSM8K, and 6/6 coding. The tuned 4-bit package measured 50.6 tok/s with 91.0%, 98%, and 6/6.
In this suite, the mixed package was faster; the tuned 4-bit package scored one point higher on MMLU-Pro. That’s a measured trade-off between two prepared packages, not an isolated bit-width experiment. Tuning differs, and there’s no evidence every other weight and runtime detail was held constant.
The AMD MiniCPM result is another useful point of reference, but not a clean cross-platform verdict. It uses the same mixed-weights recipe as the Apple headline and a different runtime. Its measured 139 tok/s and task scores belong to that specific setup; they don’t establish that AMD or GGUF will always be faster.
Reading the AMD Qwen sweep
Within the AMD Qwen3.8-27B sweep, matching the listed draft depth makes the K-quant contrast more informative. At MTP depth four, Q4_K_XL recorded 56 tok/s and Q6_K_XL 51 tok/s. Scores don’t move in one direction: Q4_K_XL scored 87.5% MMLU-Pro, 92% GSM8K, and 4/6 coding; Q6_K_XL scored 86.25%, 96%, and 4/6.
The measured Q6 row is slower and has higher GSM8K; the MMLU-Pro scores are close. This is not a universal Q4-versus-Q6 rule. It describes these packs, prompts, and settings.
IQ3_S reached 57 tok/s with MTP depth two. Comparing that with Q4_K_XL MTP2 at 62 tok/s is not just a quant-format test: the weights differ, and the Q4 entry uses a separate stock drafter while IQ3_S uses built-in MTP. Draft acceptance and compute overhead can affect throughput. The draft method belongs in the result label, not in a footnote you forget to read.
Model identity and task settings matter
The Apple Qwen row is not the exact Qwen3.8-27B model. It is Qwen3.8-Flash at iQ-MLX 4.7 bpw. The official Qwen3.8-27B model card describes a 27B causal language model with a vision encoder, 64 layers, Apache 2.0 licensing, and a native 262,144-token context. It recommends temperature 1.0, top-p 0.95, and top-k 20 in thinking mode, or temperature 0.7, top-p 0.80, and top-k 20 in instruct mode.
Our suite fixes temperature at zero. Those card details and recommended settings do not automatically apply to every related variant, and scores from different model variants should not be treated as a direct quantization comparison.
How to deploy an AI model locally
Start with the model and workload, not the smallest-looking quant label. Select a base checkpoint and a task set that reflects the work you need the model to do. Then export candidate formats from that same source and record calibration, layer overrides, runtime, and draft method. Keep context and draft settings fixed across candidates wherever possible.
Measure more than decode tokens per second. A practical comparison should include peak memory, prefill, decode, and first-token latency, alongside the same task scores. Include a long prompt and verify that the intended context fits without memory spill. A fast result that doesn’t fit the target machine—or answers the task poorly—isn’t the right deployment choice.
Our R9700 sweeps found that MTP depth two outperformed depth four on every model in its reload sweep. For example, GSQ IQ3_S prompts measured 66/61 tok/s at depth two versus 57/44 at depth four. These are separate code/prose reload results, not the standard-v2 table above. Treat them as a tuning clue, not rates to combine with the table.
What we can conclude
First, a lower nominal bit width does not promise the same quality or higher speed. Second, tok/s alone misses whether a pack fits the memory budget and whether its answers remain useful. Third, draft settings matter enough to state alongside the quantization format.
The current MLX reports provide useful standalone results; the AMD table offers several GGUF candidates. We still need paired scores for exact same model revisions and matched quant recipes before attributing a difference to MLX mixed weights, K-quants, or IQ alone.
For related context, see Same model, two machines: Mac Studio MLX vs Radeon R9700 Vulkan for the matched-weights decode headline, and Prefill and long context: Apple MLX vs Radeon R9700 for prompt-length behavior. Both call out controls still needed before drawing broader conclusions.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
