How to deploy an AI model for a local agent
Choose around the workload and software stack you intend to maintain. In our measured MiniCPM-V-4.7 run, the Apple M4 Max with MLX produced 66.8 tok/s; the Radeon R9700 with Vulkan produced 139 tok/s. Both reached the same MMLU-Pro and GSM8K scores, while coding results were 6/6 and 5/6 respectively. Useful evidence, yes. A universal buying answer, no.
A local agent repeatedly asks a model to plan, call a tool, inspect the result, then continue. Its feel depends on more than answer-generation speed: tool-call reliability, context length, first-token delay, model compatibility, memory pressure, and the application around the model all matter. A fast decoder that won't run your tool template or context size is the wrong fit. A slower model in a familiar, low-friction setup may be better for one developer.
Start with the tasks you need
Before comparing computers, write down the agent's shape: average and maximum prompt length, usual answer length, concurrent users or jobs, tool-call pattern, model family, and whether image or video input is required. Then test representative tasks with the same model, quantized weights, system prompt, tool schema, and sampling settings on each candidate.
Capture more than tokens per second. Record time to first token, completed tasks, invalid tool calls, context failures, and memory use. A generic chat benchmark can screen candidates, but it won't replace the agent's own task set.
This series' standard-v2 suite used 400 MMLU-Pro questions, 50 GSM8K questions, and six coding tasks, with temperature zero, a 2,048-token answer cap, and parallelism four. On MiniCPM-V-4.7 mixed 4/8-bit, Apple scored 90.0% on MMLU-Pro, 100% on GSM8K, and 6/6 on coding at 66.8 tok/s. The Radeon run matched the first two percentages and scored 5/6 on coding at 139 tok/s. That suggests higher single-stream decode throughput on the R9700 system for this tested setup, with close task quality. It says nothing about tool calls or an end-to-end agent loop.
Setup. — Apple: Apple M4 Max with 128 GB unified memory, MLX through mlx-serve, tested October 2026. The measured mixed-pack run used mlx-serve 26.10.2.
Setup. — AMD: AMD Radeon AI PRO R9700 (32 GB GDDR6); desktop AM5 platform, DDR5-6000, Windows 11; llama.cpp server, Vulkan backend, build b11836; flash attention enabled; 131K context, ubatch 2048, four parallel slots sharing one unified KV cache. Tested October 2026.
When the Mac Studio fits
The Apple M4 Max configuration here has 128 GB of unified memory and ran the tested models through MLX. Consider this path if your local workflow already targets Apple Silicon, if a large shared memory pool helps fit a model and its context, or if a single desktop platform is appealing. Those are selection criteria, not measured claims about power, noise, or total ownership cost; this series did not measure those.
Quantization changed the measured result. The MiniCPM mixed 4/8-bit pack decoded at 66.8 tok/s; a tuned 4-bit pack scored 50.6 tok/s and had slightly different task scores. Apple reports also include longer-context prefill measurements for a different MiniCPM 4-bit run. That isn't a direct comparison with the Radeon mixed package. If your agent handles large codebases or documents, test your actual prompt sizes and record prefill separately from decode. See the related Navigating AI Model Local Deployment: What Two MiniCPM Runs Can—and Can't—Tell Us article for the separate long-context observations.
When the Radeon system fits
The Radeon setup measured 139 tok/s for MiniCPM mixed 4/8-bit in a Vulkan-backed llama.cpp server. It may suit you if your models and serving stack work well on that path. The setup used four parallel slots sharing one KV cache and 32 GB of GPU memory.
Configuration matters. On this measured system, ubatch 2048 avoided a large prefill and decode drop seen at lower settings. A desktop rendering workload also reduced measured decode at 16K context until the integrated GPU was used for display. So “R9700 speed” isn't a property of the accelerator alone; it depends on a correctly tuned, configured host. Check model support, available quantization, multimodal paths, context size, and maintenance needs in the runtime you actually plan to use.
The official MiniCPM-V-4.7 model card describes a 35B-A3B multimodal MoE model, Apache 2.0 licensing, and support for several inference frameworks. But an individual quant pack may not offer the same features or behavior as the original weights.
A practical deployment decision
For a single user, compare prompt-to-first-token time and output speed on your usual prompt length. For a multi-user service, test sustained concurrency and queueing; a single-stream median won't answer that question. For tool-heavy agents, count completed tasks and successful tool invocations. For long histories, run repeated long-context tasks and watch memory headroom. If the agent processes images or video, include representative media inputs.
Keep one suite for quality and another workload for latency and throughput if combining them would hide a trade-off. Then account for the full experience: how much time will you spend converting model formats, matching quant packs to supported kernels, and debugging updates? Which programming environment will you use? Does the model fit alongside the context and concurrency you need? Can the host stay stable through long sessions?
Those questions often matter more than a peak token rate in isolation. We don't have measured power draw or noise for either setup, so those need direct measurement, not assumptions. The evidence supports a narrow claim: one shared MiniCPM weights recipe reached 139 tok/s on the R9700 and 66.8 tok/s on the M4 Max, with close but not identical suite outcomes. To make the choice specific, benchmark your agent's task loop on both systems, record a matched long-context run, and add wall-power and acoustic measurements. Until then, use these results as a starting point for a local proof of fit, not a universal buying verdict.
Continue with Same model, two machines: Mac Studio MLX vs Radeon R9700 Vulkan, Quant formats compared: MLX mixed 4/8-bit vs GGUF K-quants and IQ, and Power, noise and cost per million tokens: Mac Studio vs R9700 for supporting measurements and explicit gaps. For a related comparison of long prompts and different MiniCPM runs, consult that comparison without a link.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
