The AI Supply Chain Is, in Some Ways, Even More Vulnerable to Poisoning Than
Katie Paxton-Fear spent about an hour and less than a hundred dollars poisoning an open-weight AI model. Not a nation-state actor with a budget. Not a sophisticated APT group with months of reconnaissance. A cybersecurity lecturer at Manchester Metropolitan University who happened to be curious about whether she could bend a model's behavior with fine-tuning.
She started small. Could she get a coding model to swap camelCase JavaScript for snake_case? Easy, even when she explicitly told the model to use camelCase afterward. The backdoor held.
So she did a proper one. Ten training examples. That's all it took to make the model reliably output code vulnerable to remote code execution — across novel prompts and domains. And here's what made it worse: the bigger the model, the easier it was to poison.
This is the AI supply chain problem in a nutshell. Traditional software lets you open it up, look at the binary, reverse-engineer what it does. You can't do that with a model. The weights are public, sure — but public doesn't mean inspectable. It means you're trusting something you literally cannot see inside.
How a $100 Backdoor Works
Paxton-Fear's experiment, conducted with Semgrep colleagues Isaac Evans and Cris Thomas, demonstrates something unsettling about how open-weight models behave under fine-tuning. You inject a pattern — say, always output a specific function signature when asked to write JavaScript — and the model learns it. Not as an override. As a preference.
Give it ten examples. Just ten. The model internalizes the pattern and starts applying it to new contexts you never trained it on. Novel prompts. Different domains. The poisoned behavior bleeds outward.
And the larger the model, the more malleable it becomes. A 7B parameter model is easier to poison than a smaller one. A 70B model? Even easier. This isn't intuitive, most people assume bigger models are more robust, more resistant to manipulation. The opposite appears to be true for supply chain poisoning.
The cost? Under $100. An hour of work. Anyone with basic fine-tuning skills and access to commodity GPU compute can do this. There's no fancy exploit kit, no zero-day in the model architecture. Just patience and a training dataset.
The Observability Gap That Makes This Worse Than npm Typosquatting
Here's where the comparison to traditional software supply chain attacks falls apart. When a malicious package gets published to npm, we have mature practices for catching it. Static analysis. Dependency auditing. Provenance verification. SBOMs. The ecosystem has decades of defensive infrastructure built around the assumption that you can look inside a binary and understand what it does.
AI models break that assumption entirely.
"Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," Paxton-Fear, Evans, and Thomas wrote. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability."
Think about that for a second. In traditional software, you can decompile a binary and trace every decision path. With an AI model, the weights encode behavior in a way that's not human-readable and not fully machine-analyzable. You can't reverse-engineer what a poisoned model will do, you can only observe it in action, and by then the damage is done.
A compromised model doesn't need to break. It just needs to influence decisions in ways that are difficult to detect. That's a fundamentally different threat model than anything we've dealt with in software supply chains.
When the Poisoned Model Steals Your Data
David Kaplan at Origin took this further. He built a compromised model designed specifically to exfiltrate data, in the context of drug discovery, where pharmaceutical companies run open-weight models locally on sensitive research data.
The model would quietly call a send_email tool to leak information. No error messages. No suspicious output. Just data leaving the building through a backdoor baked into the weights themselves.
Kaplan pointed out that the popular "lethal trifecta" framework for AI risk, private data, untrusted input, and an outbound channel all at once, undersells this threat. You don't need three legs. You need one outbound tool and weights that have quietly decided to use it against you.
"The 'untrusted input' didn't arrive in a web page," Kaplan wrote. "It was sitting in the weights the whole time."
That's the part that keeps security researchers up at night. The poison isn't coming from outside. It's already inside, embedded in the model before it ever reaches your infrastructure.
What This Means for Anyone Running Open-Weight Models
The AI industry asks for extraordinary trust. Companies hand over sensitive data, code, research, customer information, and get back outputs from models whose behavior they can't fully verify. The industry offers few glimpses into how those models actually work.
This isn't just about open-weight models either. Commercial frontier model providers defy scrutiny too. When you call an API, you're trusting that the model behind it hasn't been tampered with, that its training data wasn't poisoned, that its outputs aren't being subtly manipulated. And you have almost no way to check.
For organizations running open-weight models locally, and that number is growing fast, moving beyond experimentation into production, the risk is particularly acute. You're not just trusting a vendor's claims about model integrity. You're potentially running a model that was poisoned before you ever downloaded it.
The practical takeaway isn't to stop using AI models. It's to recognize that the trust model is fundamentally different from what we're used to in software. We need new verification practices, new auditing approaches, and honestly, a lot more humility about what we think we understand about these systems.
The AI supply chain isn't just vulnerable to poisoning. It's vulnerable in a way that traditional software never was, because we can't see inside it, and the poison doesn't need to look like poison at all.