What this pack actually is
pgmharikrishnan/grug-35b-v2-MLX-4bit is a 4-bit MLX conversion on Hugging Face built for local serving. The model card tags it qwen3_5_moe, Mixture of Experts, 4-bit precision, text-only, conversational, Apache-2.0. Read those tags carefully because they tell you exactly what you are buying and, just as usefully, what you are not.
This is a text-only Qwen3.5-MoE-family derivative. It does not see images. Any expectation of vision here is a category error — the pack never claims multimodal input and this benchmark never tested for it. What it does claim is a Mixture of Experts layout with a 4-bit default quantization, plus audited overrides that lift the router gate and the shared-expert gate to 8-bit. That mixed-precision detail is the whole story of why a 4-bit MoE behaves the way it does, and it's the reason this model earns a dedicated scorecard instead of a one-line entry in a leaderboard table.
If you are new to running these packs on Apple silicon, start with the primer on what mlx-serve adds on top of MLX. If you are wondering how a Hugging Face repo turns into something a server can load, the walkthrough on converting a Hugging Face model for mlx-serve covers the mechanics end to end.
Inside the quantization
Here is the part most 4-bit posts gloss over. Quantizing everything to 4 bits is cheap to say and, for MoE models, quietly dangerous to do. The expert weights dominate parameter count, so they carry the bulk of the memory savings when you compress them hard. But the router gate makes the dispatch decision — it picks which experts a token visits — and a coarse gate misroutes tokens in ways that compound across layers. The shared-expert gate sits in the residual path on every token, so its precision has even less room to hide.
This pack's documented design matches what that reasoning would predict: 4-bit as the default storage across the model, with audited 8-bit overrides specifically on the gate and shared-expert gate. You are paying a small memory premium on a tiny slice of tensors to protect the routing signal that everything else depends on. For the intuition behind choosing precision per tensor role, see the guide on MLX quantization, 4-bit and 8-bit mixed precision, and for the router mechanics themselves, the deep dive on MoE models, router precision, and local inference.
How to deploy this model in five minutes
"How to deploy an ai model" locally, for this pack, is a short list once the server is installed. Point mlx-serve at the repo identifier, let it pull the safetensors, and talk to it over an OpenAI-compatible endpoint. No cluster, no container registry, no orchestration layer. The MoE architecture means only a slice of parameters activates per token, which is what lets a 35B-class model fit and run responsively on a single workstation rather than a rack.
Setup. Apple M4 Max workstation; mlx-serve v26.10.2; pgmharikrishnan/grug-35b-v2-MLX-4bit 4-bit; tested October 2026.
Those numbers below come from the standard-v2 harness. The full methodology — how prompts are batched, how decode and prefill are sampled, what "plain" means — is documented in how we benchmark local models, so you can argue with the numbers on the merits rather than on mystery.
The quality scorecard
The reasoning and knowledge block, MMLU-Pro-400, landed at 294/400, a 73.5% hit rate. For a 4-bit MoE pack that is a respectable floor: strong enough to hold a coherent multi-step answer on most graduate-level topics, with clear misses where a question needs exact recall the quantization has rounded away.
Math held up better than general knowledge. GSM8K-50 came in at 46/50, or 92.0%. Word problems with a clean arithmetic spine are exactly the kind of task where a preserved routing signal pays off — the model can walk the chain of operations without losing the thread.
Coding was the standout. Six of six tasks passed, a 100.0% on the coding block. That is a small sample, and I will not pretend otherwise: six passes is encouraging, not a guarantee. But a flawless block on a 4-bit text-only pack is the kind of result that makes you actually try your real work on it.
The speed and memory scorecard
Plain decode measured 49.848 tokens per second. That sits comfortably past the speed most people read at, so for a single user typing and waiting, the output stream never feels like it's stalling. Plain prefill ran at 410.688 tokens per second — useful context ingestion for longer prompts and pasted documents.
Peak memory hit 26.55 GB (24.7 GiB). That is the headline number for anyone with a smaller M4 Max unified-memory configuration. A 4-bit MoE is a memory-efficient design, but 35B-class still wants room, and you want headroom above that peak for the KV cache as context grows. This is not a model to size your purchase around a single tight number; budget above 26.55 GB and you will be comfortable.
Fidelity: a gap, not a zero
One honest blank. Fidelity was not measured for this run. I am writing that deliberately because an absent measurement is not a zero score, and a scorecard that quietly drops a column looks worse than one that admits the gap. Whether the audited 8-bit gate overrides actually preserve the fp16 logits the way the design implies is the single most interesting open question on this pack, and it is the one thing I have not closed. Treat every quality number above as a real but un-fidelity-checked result.
Where this fits in 2026
The MoE-on-Apple-silicon niche is crowded in 2026, and Grug-35B-v2 does not win it on raw speed alone. Its case is narrower and more specific: a text-only conversational pack whose audited gate precision gives the router the headroom it needs, paired with a coding block that came back clean and a decode rate that feels instant at a single desk. If you can feed it text and you have the memory headroom, it earns its slot. If you wanted vision, this is the wrong pack and you should keep looking.
The takeaway for anyone working through how to deploy an ai model on their own hardware is simpler than the spec sheet suggests. Match the precision design to the workload, give the model more memory than its measured peak, and benchmark on your own machine before you trust anyone's table — including this one.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
