The Memory Problem Nobody Wants to Talk About
Here's a number that should keep any AI engineer up at night: a 7-billion-parameter model in full precision needs roughly 28 gigabytes just to exist. Twenty-eight gigs. That's not a typo. And if you're running anything larger — the 70B-class models that actually do useful work in production — you're looking at a quarter of a terabyte sitting in memory, idle, waiting to be queried.
The industry's default response has been quantization. Shrink the numbers. Go from 32-bit floats down to 16, then 8, then 4 bits. Each step cuts the footprint in half. It works, sure. But there's a tax. Every bit you shed takes accuracy with it. And at some point — usually around 4-bit quantization for semantic tasks — the model starts producing garbage that sounds plausible enough to slip past a human reviewer.
What if there's a different path? One that doesn't require choosing between size and smarts?
How Quantization Actually Works
Let's be clear about what quantization does, because a lot of people treat it like magic compression. It isn't.
At its core, quantization maps a wide range of floating-point values onto a smaller set of discrete levels. Think of it like rounding. If your model weights normally live at values like 0.347821 and -1.923847, quantization snaps them to the nearest representable value in a coarser format. FP16 uses two bytes per parameter instead of four. INT8 uses one byte. Q4 uses half a byte on average.
The tradeoff is straightforward and unforgiving: less precision means less fidelity. For some tasks — image classification, basic text generation — the loss is barely noticeable. For semantic work, where the model needs to understand that "king" relates to "queen" the way "man" relates to "woman," precision matters enormously. Those relationships live in the subtle geometry of embedding vectors, and when you crush those vectors down to 4-bit integers, the relationships collapse.
There are clever variants that mitigate this. AWQ (Activation-Aware Weight Quantization) identifies which weights matter most and preserves their precision. GPTQ does something similar but operates post-training. SmoothQuant redistributes the quantization error between weights and activations. These techniques buy you a few percentage points, but they're still playing the same game: preserve as much precision as possible while shrinking the representation.
The fundamental assumption remains: you need to keep the numbers. SEMQ challenges that assumption entirely.
What SEMQ Actually Does
Andrés Mac Allister, who founded The SEMQ Group, put it this way: conventional embedding systems store semantic state as sequences of high-precision numerical coordinates. Those coordinates encode both magnitude and direction in the embedding space.
SEMQ asks a different question. What if you don't need to preserve the absolute magnitude at all? What if semantic relationships depend primarily on the relative orientation of vectors — how they point with respect to each other, what regions they occupy, what directional configuration they form?
That's a big if. But Mac Allister's answer is to replace raw vectors with fixed-dimensional symbolic structures that preserve relational properties. Relative similarity ordering. Neighborhood structure. The geometry of meaning, stripped of the noise of absolute values.
The paper describes it as decoupling representation from metrics, indexing, and execution semantics. In plain English: you keep what matters for understanding, and drop what only matters for computation.
This isn't quantization in the traditional sense. It's not about representing the same information more efficiently. It's about recognizing that most of what you're storing isn't actually needed for the tasks that matter.
The Numbers Don't Lie
Here's where it gets interesting. Mac Allister shared benchmark results from a test using the Banking77 dataset from MTEB with the all-MiniLM-L6-v2 embedding model.
The FP32 baseline achieved 92.26 percent accuracy. SEMQ achieved 92.27 percent. That's effectively identical — within 0.03 percentage points, which is well inside the margin of error for this kind of test.
Now compare that to 4-bit quantization on the same task: 56.05 percent accuracy. Thirty-six percentage points worse. That's not a marginal degradation. That's the difference between a system that works and one that doesn't.
Mac Allister was careful to note that these results don't prove conventional quantization is universally ineffective. For many tasks — classification, retrieval with large vocabularies, anything where absolute magnitude carries signal — quantization remains the right tool. But for semantic classification specifically, preserving the relevant structure is materially different from simply reducing numerical precision.
The implication is significant. If you can maintain near-full-precision accuracy while storing far less data, the infrastructure savings compound quickly.
Deploying Without Replacing Your Stack
One of SEMQ's more practical advantages is that it doesn't require ripping out your existing infrastructure. Mac Allister describes two integration points.
First, at data ingestion. You run your embedding model as usual, generate the vectors for your documents, then pass those vectors through the SEMQ SDK to encode them as .semq artifacts. The original model stays in place. You're just adding a transformation layer.
Second, at query time. Load the .semq encoding, query it, compare results, restore state, verify outputs. This works alongside whatever vector database or agent framework you're already using.
The sidecar approach means teams can adopt SEMQ incrementally. Run it alongside the existing stack for selected retrieval or memory workloads, prove the value, then expand. No big-bang migration. No vendor lock-in to a new embedding model.
This matters because most enterprises aren't starting from scratch. They've got vector databases, embedding pipelines, and agent frameworks that took months to build. A technology that requires replacing all of that has a much higher bar to adoption than one that slots in alongside it.
Where This Could Actually Go
The use cases Mac Allister outlined are specific enough to feel real rather than speculative.
Making embeddings portable across systems is one. If your semantic state lives in a format that's decoupled from any particular model or database, you can move it. Reproduce it across different runs or machines. Audit changes to model behavior by diffing semantic state between versions.
There's also the runtime angle. Mac Allister mentioned that .semq files have been used to snapshot and restore transformer KV-cache state across process boundaries. That's a runtime-state workflow — pausing, transferring, and resuming an active model session. Not pre-training. Not fine-tuning. Actual inference state, captured and restored.
For enterprises dealing with AI systems where reproducibility matters — regulated industries, audit-heavy workflows, anything where you need to prove what a model knew and when — this kind of state management is operationally important. Mac Allister said his company is working with hyperscalers and application-layer companies through a Founding Design Partnership Program, though NDAs prevent naming them.
The interest, he said, comes from teams where "reproducibility, state, lower infrastructure overhead, and the ability to inspect semantic behavior are operationally important." Translation: big companies with big problems.
The Bigger Picture
What's striking about SEMQ isn't just the benchmark numbers. It's the shift in perspective.
The quantization approach assumes that semantic information lives in the values themselves — in the exact floating-point coordinates of embedding vectors. SEMQ assumes it lives in the relationships between those values — in how they point, where they cluster, what structure they form.
That's a philosophical difference as much as a technical one. And it opens up possibilities that quantization simply can't address: auditing, reproducibility, portability. When your semantic state is encoded as relational structure rather than raw numbers, you can diff it, compare it, transfer it across systems in ways that would be meaningless with conventional representations.
This isn't going to make quantization obsolete. For many workloads, crushing model weights down to 4 bits is still the right call. But for semantic tasks — retrieval, memory, agent state, anything where understanding matters more than raw computation — SEMQ offers a path that doesn't require the same tradeoffs.
The question isn't whether this will replace what we have. It's whether enterprises willing to adopt it will gain enough from the infrastructure savings and operational clarity to make it worth the integration effort. The early benchmarks suggest the answer is yes.