What AI server hardware architecture means in practice
AI server hardware architecture is the interaction between a model’s computation and the machine that executes it: memory movement, kernels, dispatch overhead, batching, and the accelerator’s strengths all matter. It isn’t simply a list of chips or a leaderboard. This comparison makes that concrete. Inkling-2-Mini has an unusual looped architecture, and two very different backends put the same pressure point in view: decoding can be limited by launching work rather than by arithmetic throughput.
That is a useful engineering finding, not a ranking win. The measurements below are supplied results from the stated November 2026 runs; the linked public materials provide model identity and architecture context, not independent verification of these benchmark numbers.
The architecture: 48 layer visits from 24 layers
Inkling-2-Mini has 26.4 billion total parameters, with 3.4–5.4 billion active in its looped mixture-of-experts configuration. It has 24 physical layers, but the middle layers 6–17 are run three times with shared weights. The result is 48 layer visits per token, not 24. That distinction matters for serving: physical depth describes stored layers, while visits describe how often the token traverses computation. The model card describes this looped structure and the active-parameter range. Model card
The attention and residual design is also specific. BlockAttentionResidual is used across every six layers. On the global arm, attention uses RelativeLogits bias with d_rel 16 and rel_extent 1024 rather than RoPE; each layer has four short convolutions with kernel size 4, and sliding-window layers use a 1024-token window. These details help explain why the implementation is not just a conventional transformer with a different parameter count. The linked model card provides architecture context. Model card
Its feed-forward path uses a softcapped SwiGLU MoE with 128 routed experts and top-8 routing. The route scale is 8.0, gate softcap 4.0 and up softcap 25.0; QK and relative-logit values are L2-clipped, and the final logit softcap is 30.0. In other words, sparsity doesn’t make the model a single small dense network: each token is routed through selected experts, while the loop revisits the core. Model card
There is one thinking control: reasoning_effort, from 0.0 to 0.99. Zero is the floor, and still produces brief reasoning. There is no enable_thinking toggle and no option to omit the reasoning channel. That matters when evaluating output length and failures; reasoning wasn’t disabled in the run by setting effort to zero.
Setup and first results
Setup. Apple Silicon workstation (Apple M4 Max, 128 GB unified memory; mlx-serve v26.10.3 public fork branch inkling2-loop-support; pgmharikrishnan/Inkling-2-Mini-MLX-Serve-4bit) and AMD Radeon AI PRO R9700 (32 GB; llama.cpp Vulkan public build b12600; IQ4_NL quantisation); tested November 2026.
| Platform and path | Non-speculative decode | Prefill / attention observation |
|---|---|---|
| Apple M4 Max, 4-bit MLX-Serve | 39.1–39.3 tok/s | Prefill 204.6–206.3 tok/s; 10,809 unfused operations per forward |
| Radeon AI PRO R9700, IQ4_NL | 30.8 tok/s, flash attention off; 31.2 tok/s on | Near-zero flash-attention delta |
The Apple result is striking less for its absolute rate than for the execution profile: the forward pass is unfused at 10,809 operations per forward and is host-dispatch bound. Non-speculative decode measured 39.1–39.3 tokens per second, while prefill reached 204.6–206.3 tokens per second. This is a useful illustration of why AI server hardware architecture cannot be inferred from peak compute alone. A workload with repeated layer visits can spend a meaningful share of time coordinating small operations, even when the hardware has substantial memory bandwidth and compute capacity.
On the 32 GB Radeon AI PRO R9700, a public llama.cpp Vulkan build (b12600) ran the IQ4_NL quantisation at 17.6 GB. Stock decode was 30.8 tokens per second with flash attention off and 31.2 with it on, a difference of only 0.4 tok/s. With 48 layer visits per token, that near-zero delta is evidence consistent with dispatch-bound decoding on this build, rather than a major gain from that attention switch. The results do not establish a universal hardware ranking.
Apple’s tested pack is the public 4-bit MLX-Serve model, and the implementation branch adds loop support and the native draft head. Radeon results use a public Vulkan build identity. These links document artifacts and implementation context only; none is the source of the measured rates. 4-bit pack · mlx-serve loop-support branch · llama.cpp releases · AMD Radeon AI PRO
Speculation must fit the backend
Speculative decoding asks a drafter to propose upcoming tokens so the main model can verify several at once. The catch is that verification only helps if a batch of proposed tokens is cheaper than running those forwards one by one. The same model can therefore prefer different speculation strategies on different engines.
On Apple Silicon, the model’s own DSpark native draft head used rnn_hidden, block_size 8 and markov_rank 256. It achieved 1.78–3.44 accepted drafts per round, with 3.44 on code prompts. In this environment the trained drafter is a practical match: accepted proposals can amortize verification work. These are supplied run measurements, not figures reported on the linked model or implementation pages.
On Radeon, the same native drafter lost. Across draft lengths one through four, decode fell to 18.1–24.2 tok/s, and acceptance was only about 0.32–0.44. The reason is operational: a four-token verification batch costs roughly four single-token forwards on this Vulkan path, so the verification work consumed the potential gain. Adding a trained draft model does not guarantee faster output when the target backend makes wide verification proportionally expensive.
The winner on Radeon was lossless ngram-mod lookup speculation, which needs no draft model. It raised decode from 31.2 tok/s to 39.1–52.1 tok/s, about 1.3× to 1.7×, with draft acceptance of 0.72–0.97. This is the clearest contrast in the test: a native trained draft head fits better where verification is cheap, while a lossless lookup drafter fits better where a wide verify batch costs a proportional number of forwards. Neither is universally right. Choosing speculation per backend is the engineering answer.
Quality context: useful output, with a real gap
The Radeon quality run used the same 32 GB workstation, IQ4_NL at 17.6 GB, reasoning_effort 0.0, the standard-v2 suite and ngram-mod with n-max 8. MMLU-Pro, a multiple-choice benchmark intended to probe broad expert-level knowledge and reasoning, scored 341/400 (85.25%, Wilson 95% interval 81.4–88.4). GSM8K, a set of grade-school math word problems, scored 48/50 (96%). The six-task coding set, which samples coding problem-solving across six tasks, scored 5/6. Across MMLU-Pro and GSM8K, the run produced 605,682 output tokens.
This score was below the top pick on the same box and suite: Qwen3.6-35B-A3B scored 362/400 (90.5%) at about 144 tok/s. The paired MMLU-Pro comparison against the 362/400 Qwen3.8-27B cell had 30 questions wrong-to-right and 9 right-to-wrong, p = 0.0011. That gap is real and should not be called equal. Nor does it make Inkling-2-Mini a failure. It describes what a day-one engine for a new looped architecture delivered and where it stood in this test.
There was also a concrete failure mode: 35 of 400 MMLU-Pro items produced empty answers because the model stayed inside its reasoning channel until the 8,192-token cap. They were scored wrong under the same rule applied to every cell. The reasoning_effort floor did not disable reasoning, and this behavior should be reported rather than hidden.
These are one machine, one engine build, one suite and one date per platform. The platforms were not run on identical suites, and their cross-platform speed numbers are not a quality comparison. The public model card and serving material establish architecture context; the rates, quality scores, paired comparison and failure count above are supplied measurements from the described runs, not claims attributed to those pages. Model card
The practical takeaway
A model with 24 physical layers can still make 48 visits per token. That architecture puts repeated scheduling and dispatch on the critical path, and the two backends expose it in different ways. Apple’s native draft head found room to help; Radeon’s verification cost made that same head a drag, while lookup speculation paid off. The useful result is not that one accelerator wins. It is that backend-aware speculation is part of implementing the architecture, not an afterthought.
Disclaimer. These are measurements from one machine, one server build, and one test suite on the stated date. Results vary with hardware, software version, settings, and load. They are not a general ranking or a recommendation.
ai server hardware architecture means in practice