Key takeaway: Every model in this test set could be run at 131K context on the 32 GB R9700 with ubatch 2048, but fitting a model is not the same as leaving comfortable headroom. The largest tested configuration occupied about 27.6 GiB, while a 27.9 GiB Flint Q5 build spilled into system memory. A model-size label is only a starting point for estimating memory. The actual allocation includes quantized weights, runtime buffers, attention state, and KV cache. Context length and concurrent requests also matter.
How to Deploy an AI Model Locally: Fit Your Workload
Our practical question was simple: can these local models run at the advertised long context, and what configuration changed the margin?
Setup
Setup. AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000; Windows 11; llama.cpp llama-server, Vulkan backend, build b11836; flash attention on; 131K context unless noted; ubatch 2048; four parallel slots sharing one unified KV cache. Tested October 2026. The Qwen3.8-27B model card documents a dense 27B vision-language model with native context of 262,144 tokens, extensible further. The Flint card describes a decoder-only MoE with 39.8B total and approximately 2.5B active parameters per token, plus a documented 262,144-token context. These public model specifications do not tell us the exact runtime allocation in GGUF; our memory observations below are tied to the tested builds and server configuration.
Measured fit
| Model/configuration | Measured or noted memory | At 131K context? |
|---|---|---|
| MiniCPM-V-4.7 35B-A3B, mixed 4/8-bit | ~24.4 GiB | Fits |
| Flint 40B-A2.5B, mixed 4/8-bit | ~27.6 GiB | Fits, limited headroom |
| Flint Q5 variant | 27.9 GiB model build | Spilled to system memory |
The two explicit 131K figures demonstrate that large context is feasible on this card with these settings. They do not imply that any model file of similar size will fit. An image projector, architecture-specific buffers, runtime version, KV-cache type, and concurrency affect the result. We did not record a full per-layer VRAM trace for every quantization, so the table should be read as tested outcomes, not a universal capacity calculator. A useful distinction is total parameter count versus active parameters. In a mixture-of-experts model, only some experts are selected per token, which helps compute cost; the full expert weights still need to be available to the runtime. Therefore a 40B-A2.5B label does not mean the model occupies memory like a 2.5B dense model. The Flint model card makes this distinction clear; it does not replace measuring the chosen quantization.
Context is not a free setting
KV cache grows with active context and the number of sequences. Four parallel slots sharing a unified cache use the available budget differently from one long conversation. The benchmark used 131K context and four slots, but actual workloads may have different prompt lengths and concurrency. A workstation used for single-person chat can reduce slots to reserve more practical room for long prompts. The benchmark setup used flash attention and a unified KV cache. Changing cache precision, parallelism, or server build can affect both fit and quality. Because we did not measure per-model KV bytes per token or a full occupancy curve, we do not estimate an exact number of simultaneous 131K conversations. Treat that as a workload-specific measurement still to collect.
ubatch changes the experience
Memory planning is not only about the final context. Prompt ingestion also uses temporary resources and can be dramatically slower with an undersized ubatch. In our MiniCPM-V check, ubatch 512/1024 cut prefill from about 3,850 to 830 tokens per second and decode from 103 to 61 tokens per second. At ubatch 2048, both recovered. For Flint, long-context prefill was 4,330 tokens per second at ubatch 2048 versus 3,050 at 512 for the measured 22.5K prompt. These figures support choosing a batch large enough for the intended workload, but they are not a recommendation to increase it without checking memory. The correct choice is a tested point where prompt throughput improves and the workload still fits with the desired context and concurrency. We did not record a complete sweep of memory usage across every batch size.
Windows mmap behavior and spillover
A Windows-specific observation caught our attention: memory-mapped loading kept the whole GGUF resident in system RAM after upload to VRAM. For MiniCPM-V that meant 17.7 GB remained in system memory. Loading without mmap reduced that amount to about 1 GB in our observed run. This is host-memory residency behavior; it does not mean the model’s GPU footprint disappeared or that a specific mmap setting will always consume the same amount on another machine. System RAM can mask an undersized VRAM budget by accepting overflow. That may let a model start, but moving data across the host/device boundary can change performance and stability. The tested 27.9 GiB Flint Q5 variant spilled into system memory; we did not measure its resulting throughput, so we avoid assigning a speed penalty. If your operating system reports high RAM use after a model upload, compare memory-mapped and non-mapped loading on the same server build and observe both GPU and host memory.
How to decide whether a model fits your use
Use the model file’s actual quantization size as a first filter, then test the server at the context and concurrency you need. Include the prompt batch you intend to keep. Observe whether memory remains on the GPU, whether the operating system begins paging or the runtime moves tensors to system memory, and whether prefill and decode remain acceptable. Repeat after a clean server restart to avoid confusing cached allocations with the active model. The practical answer is encouraging but bounded: 32 GB held the listed mixed 4/8-bit models at 131K in the measured profile, and the largest named fit left less margin than the smaller one. A slightly larger quantization spilled. The follow-up articles compare their quality and speed, then examine the Qwen quantization tradeoff directly. Related articles: Flint on the R9700: AI Model Local Deployment Performance and Reasoning Realities; Local AI Model Deployment on the Radeon AI PRO R9700: Speed and Tradeoffs. Unmeasured points remain: exact per-model VRAM traces, context-memory curves, and the throughput of the spilled Flint Q5 variant. Power and cost per token were also not measured. These are good candidates for a follow-up data collection rather than a basis for estimation.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, model file, and load. They are not a general ranking or a recommendation.