Building a benchmark, not a leaderboard
This page documents a local-model benchmark suite that is still under development in 2025–2026. It is a method write-up, not a results page. As more teams start comparing engines for AI model local deployment, the most useful thing we can publish early is not a single speed number but the protocol behind any number we might later publish: what we hold fixed, what we measure separately, what evidence we keep, and why we deliberately do not reduce all of it to one "winner".
A bare throughput claim is easy to make and almost impossible to trust. "Model X hits 40 tok/s" is meaningless without the chip, the unified-memory size, the server build, the quantisation, the prompt length, and the load pattern behind it. Our goal is the opposite of a leaderboard headline: a repeatable procedure where two people reading the same description, on the same class of machine, could re-run the test and land in the same place.
What "repeatable" means here
Repeatable does not mean "fast". It means a run is reproducible in its setup and verifiable in its evidence. We borrow vocabulary from a published methodology draft for LLM-serving benchmarks — an individual Internet-Draft (draft-gaikwad-llm-benchmarking-methodology, 2026) that describes test setup parameters, load-generator requirements, reference workloads, warm-up rules, and reporting formats. That draft is explicitly work-in-progress and not an endorsed standard; we cite it as a useful framing, not as authority.
Concretely, repeatability for our suite rests on four habits:
- Fixed workload and setup parameters. Each test declares its reference workload (prompt and output-length profile), sampling settings, and concurrency up front. We state whether load is generated open-loop (requests arrive on a schedule regardless of responses) or closed-loop (a fixed number of workers), because the same server reports different latency under each.
- A defined warm-up versus cold-start policy. The first request after a model loads behaves differently from steady state. We separate cold-start measurement from warmed steady-state numbers rather than blending them.
- Tokenizer-aware accounting. Throughput and context behaviour are meaningless unless token counts come from the model's own tokenizer, so the tokenizer is part of the recorded setup, not an afterthought.
- Retained per-run evidence. Every run keeps its raw trace — per-request timings, the recorded settings, and the build identifier — alongside the summary. The summary you might see published is derived from kept evidence, not typed in by hand.
It is important to distinguish this plan from implemented features. Some of the items above are wired into the current development harness; others are specified but not yet automated. When a run is still manual, we say so.
Separating the dimensions of an AI model local deployment benchmark
The suite treats four things as different questions, measured independently and reported independently.
Latency answers "how long until I see something?" — time to first token and the spacing between streamed tokens. It is what a person typing at a keyboard actually feels.
Throughput answers "how much work does this machine do per second?" — output tokens over the wall-clock window. It is what matters when you fan out many requests, and it trades directly against latency.
Quality answers "is the output any good?" — task correctness on a held-out task set. We refuse to infer quality from speed. A faster run is not a better model, and a 4-bit run of a model is a different data point from its 8-bit version, which is why quantisation is recorded for every quality claim (see MLX quantisation: 4-bit, 8-bit, and mixed precision).
Context behaviour answers "what happens as the window grows?" — how latency and memory shift as prompts and generations lengthen, and how cache policies behave. The MLX tooling exposes exactly the levers that matter here: a rotating fixed-size key–value cache, prompt caching, and a configurable prefill step size. These are knobs that change the answer, so they belong in the recorded setup rather than being silently defaulted. Understanding Apple MLX unified memory and local model serving is the background for why the same weights can behave very differently across unified-memory sizes.
Measuring them separately is the whole point: a configuration can win on throughput and lose on first-token latency, and that is information, not a contradiction.
Why there is no composite winner
You will not find a single score on these pages, and that is intentional. Quality, throughput, latency, and context behaviour are not commensurable — there is no honest arithmetic that adds "tokens per second" to "task accuracy" into one number without smuggling in hidden weights. A weighted total would silently encode a use case (chat? batch summarisation? long-document agents?) and then present it as a universal verdict.
It also matters for agentic workloads. People asking "What is Agentic AI?" are usually imagining multi-step, tool-using sessions where a model runs many short calls in a loop. Those stress first-token latency and context retention differently from a single long generation, so a benchmark tuned for one will mislead you about the other. Multi-step agent traces are out of scope for the current suite; we list them on the roadmap rather than pretending a single-turn score covers them.
So every comparison here is dimension-by-dimension and configuration-by-configuration. It is not a ranking, and it is not a recommendation.
The first results discussion: how we will report them
When a development run is recorded and promoted to public reporting, the first thing published is an explicit setup header so the reader can judge transferability before reading a single number. Our reporting format is:
Setup. Apple Silicon workstation (chip family and unified-memory capacity stated only when supported by the recorded result); mlx-serve release version; public model ID and quantisation; tested month and year.
If a recorded result does not contain a chip family, a memory size, or a server version, those fields stay unstated — we do not fill them in from assumption. The model ID points at a public release, the quantisation is named exactly (for example the mixed 4/8-bit treatment described in our LFM2.5-24B-A2B mixed 4/8-bit mlx-serve write-up), and the date is the date the run was recorded.
If you want to deploy an AI model locally
A fair question behind a benchmark question is simply "how to deploy ai model". The local path these tests target is: pick a public model, convert it for the runtime (converting Hugging Face models for mlx-serve), choose a quantisation, then serve it and read its real behaviour on your own hardware rather than trusting a borrowed number. Start with what mlx-serve is and local MLX inference. That hands-on step is exactly why our suite insists on recording the machine, build, and quantisation: the only figure worth trusting for your deployment is one measured on a setup you can recognise as yours.
Roadmap for the suite
What remains under development: automating open-loop load generation and clock-stamped per-run traces, standardising the held-out quality task set, adding long-context sweeps as first-class reports, and — later — multi-step agent traces. Until each piece exists, the suite ships its protocol ahead of its numbers, because a method you can inspect is more useful than a result you cannot reproduce.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
