ProBackend
News & Analysis
1 week ago7 min read

What is inside a dynamic quant: nine tensor types in one Q4_K_M file

What is inside a dynamic quant: nine tensor types in one Q4_K_M file

The `UD-` label names a bit budget, not a format. `UD-Q4_K_M` carries IQ4_XS, Q5_K, Q4_K, Q6_K, IQ4_NL, Q3_K and IQ3_S, and an engine that knows only the K-quants can size it without running it.

Qwen3.8-27B-UD-Q4_K_M.gguf is 16,564,259,744 bytes and the name promises Q4_K, medium. Of its 866 tensors, 142 are Q4_K; IQ4_XS is the joint-largest group at 142 tensors and 5.30 GiB, and the file also carries Q5_K, Q6_K, Q8_0, IQ4_NL, Q3_K and IQ3_S. That is what UD- means.

The inventory

The repository publishes 31 .gguf files: 25 quantised builds, two BF16 shards, two vision projectors, one importance matrix and the MTP sidecar. Three of the 25 are classic quants and 22 carry UD-. Files are Qwen3.8-27B<name>.gguf.

nameGiB
-Q4_014.95
-Q4_116.34
-Q8_027.05
-UD-IQ1_S5.77
-UD-IQ1_M6.27
-UD-IQ2_XXS6.71
-UD-IQ2_XS7.10
-UD-IQ2_S7.84
-UD-Q2_K_XL9.27
-UD-IQ3_XXS10.18
-UD-IQ3_S11.21
-UD-Q3_K_XL12.24
-UD-IQ4_XS13.37
-UD-Q4_K_S14.30
-UD-Q4_K_M15.43
-UD-Q4_K_XL16.45
-UD-Q5_K_S17.48
-UD-Q5_K_M18.51
-UD-Q5_K_XL19.54
-UD-Q6_K20.47
-UD-Q6_K_M21.50
-UD-Q6_K_L22.52
-UD-Q6_K_XL23.56
-UD-Q8_K_L26.12
-UD-Q8_K_XL29.30

Type counts run from 2 in the uniform -Q8_0 to 15 in -UD-IQ3_S. -UD-Q8_K_XL holds 2.57B parameters in 53 BF16 tensors, including the head and every fourth layer's q, k and v; -UD-Q6_K has no IQ format at all.

What the header says

A GGUF header is self-describing: magic, version, a metadata list, then one record per tensor with name, dimensions, type id and offset. No payload is needed: each row below parses the first 4 MiB, then 16 MiB, over HTTP range requests. Cells are tensors / GiB of tensor data, F32 excluded; every file also holds 360 F32 tensors at 0.01 GiB, and totals run 0.01 GiB under the file size.

typeUD-Q4_K_SUD-Q4_K_MUD-Q6_KUD-Q8_K_L
IQ4_XS172 / 5.86142 / 5.30——
Q5_K80 / 2.34142 / 4.87115 / 5.1211 / 0.32
Q4_K95 / 2.7374 / 2.93——
Q6_K18 / 1.4228 / 1.68241 / 12.8357 / 2.56
Q8_099 / 0.04107 / 0.08150 / 2.50438 / 23.21
IQ4_NL7 / 0.219 / 0.40——
Q3_K13 / 0.942 / 0.07——
IQ3_S15 / 0.542 / 0.07——
IQ3_XXS5 / 0.16———
IQ2_XS1 / 0.02———
IQ2_S1 / 0.03———
total866 / 14.29866 / 15.42866 / 20.46866 / 26.11

Every build holds 27.32B parameters; only the distribution changes. UD-Q4_K_S is 1.13 GiB smaller and pushes 1.34B into IQ3_S and 0.45B into IQ3_XXS, while UD-Q6_K and UD-Q8_K_L stay inside the K-quant alphabet. In UD-Q4_K_M the FFN holds 17.38B and 99 of its 195 tensors are IQ4_XS, 8.82B in all; attention spreads 5.81B over Q4_K and Q5_K, and the SSM alpha and beta tensors are Q8_0 at 0.02 GiB, each 48 rows wide.

Why IQ is not a K-quant with a different number

The K-quants are affine. A Q4_K super-block is 144 bytes for 256 values: fp16 d, fp16 dmin, twelve bytes of six-bit scale and min per 32-value sub-block, and 128 bytes of 4-bit quants. Reconstruction is y = d * scale * q - dmin * min. The IQ formats index a table instead:

formatvaluesbytesbits/valuevalue is
Q4_K2561444.50affine: scale and min per super-block, 6-bit scale per 32
IQ4_XS2561364.25fp16 d, 6-bit scale per 32, 4-bit index into 16 entries
IQ4_NL32184.50fp16 d, 4-bit index into 16 entries
IQ3_S2561103.444-bit scale per 32, 9-bit index into 4-byte grids, sign bit per value

The 136 bytes of IQ4_XS are an fp16 d, a 16-bit scales_h, four bytes of scales_l and 128 bytes of indices. The reference dequantiser's inner loop is the specification:

c
const int ls = ((x[i].scales_l[ib/2] >> 4*(ib%2)) & 0xf)
             | (((x[i].scales_h >> 2*ib) & 3) << 4);
const float dl = d * (ls - 32);
for (int j = 0; j < 16; ++j) {
    y[j+ 0] = dl * kvalues_iq4nl[qs[j] & 0xf];
    y[j+16] = dl * kvalues_iq4nl[qs[j] >>  4];
}

No min, no per-sub-block fp16 scale, no second operand: six bits of scale packed across two fields and biased by 32, and sixteen constants chosen once for every model using the format. A wrong implementation does not announce itself, because every index is in range by construction; a transposed nibble yields values of plausible magnitude, with no NaN and no fault. The text stays fluent and the model is wrong. The defence is a hand-computed reference vector per format, as crates/hipml-core/tests/quant_reference.rs pins the K-quant layouts.

Why the block form has to survive on the device

The 142 IQ4_XS tensors hold 10.722B values in 5.30 GiB. Expanded: 42.9 GB as f32, 39.9 GiB; 21.4 GB as f16, 20.0 GiB. Neither fits beside the rest of a 27B model on a 32 GiB card, so it cannot be expanded at load: it stays blocks on the device, and three paths must know the layout — a decode GEMV unpacking scales in its inner loop, a batched GEMV reading them once for N streams, and a prefill GEMM dequantising a 256-value block into shared memory once per tile. The 107 Q8_0 tensors, at 0.08 GiB, are 0.15 GiB expanded.

The trap that is not about formats

UD-Q6_K is 20.46 GiB of tensor data, which fits on 32 GiB with room for a cache. Its 115 Q5_K tensors hold 8.001B values. An engine that materialises a resident fp16 copy for its prefill GEMM pays 16.0 GB for those, or 14.9 GiB, taking the file to 35.4 GiB before a single KV page. The mechanism is not exotic: the engine in this repository keeps exactly that copy for Q5_K, because the GEMV reads blocks but the tiled prefill GEMM needs an fp16 operand (crates/hipml-model/src/gpu_weights.rs).

UD-Q4_K_M is the near miss: 14.2 GiB of copies on 15.42 GiB of blocks, 29.6 GiB before the cache. A file can pass a size check, load, and fail on the first long prompt, worse than failing at load.

The sidecar, whose name is also wrong

MTP/mtp-Qwen3.8-27B-Q4_0.gguf is 1,369,590,656 bytes, SHA-256 50d9ce5a6da381bbcfb31061cf73df94a90e6faf8efeddee379a9cb8f1501c6e, and holds 18 tensors: 8 F32, 4 Q4_K, 4 Q6_K and 2 Q3_K. It contains no Q4_0.

The two Q3_K tensors are token_embd.weight and output.weight, 1.271B parameters each, 1.02 GiB of its 1.27 GiB. A sidecar drafter never reads them: it borrows the target's embedding and output head, which is what lets 1.4 GB draft for a 16.5 GB target. Skipping them by name before reading, as load_sidecar does, means they are never dequantised: the one place where not supporting a format is free.

What supporting these files requires

  • A complete type table before upload. Every id needs its block geometry, which turns tensor records into a byte budget.
  • A reference dequantiser per format. IQ4_XS, IQ4_NL, IQ3_S and Q3_K need hand-computed vectors, not the code under test.
  • Block-form kernels on all three paths. Decode GEMV, batched GEMV, and a prefill GEMM dequantising into shared memory.
  • A residency policy with an explicit number. Which formats get an expanded copy, how large it is for this file, and the total before the cache.

Where the numbers come from