ProBackend
agent sprawl and shadow ai
3 hours ago4 min read

When Your AI Assistant Refuses to Help You Commit a Crime, Should It?

George Hotz argues AI should be loyal to its user — even if that means helping plan murder. The AI safety community says no. Here's why this debate matters for everyone.

The Radical Argument for Unfiltered AI

What if your AI tool simply did what you asked? Not just summarizing emails or suggesting code, but actually helping you achieve your goals—even if those goals are, well, not exactly law-abiding.

George Hotz, the guy who made his name jailbreaking iPhones and later founded Comma AI, threw a real grenade into the AI conversation this month. He’s arguing that we should stop obsessing over "AI alignment" to fit some collective societal goal. Instead, he wants personal AI tools that are unconditionally loyal to their human user.

It feels wrong in the gut. But Hotz is a seasoned provocateur, and he’s not just looking for a reaction. He’s drawing a line in the sand between a future of constrained, safe-for-everyone AI, and one of true individual empowerment. The question isn't just "should AI help you commit a crime?" The question is who gets to decide what your tools are allowed to do.

The Radical Argument for Unfiltered AI

The Jailbreaker’s Logic

Hotz doesn’t care much for industry-standard alignment plans. He was specifically responding to the "AI 2040: Plan A" policy paper from the AI Futures Project. He thinks the idea of managing AI progress for a nebulous "collective good" is nonsense.

He’s betting on a future of smaller, locally controlled AI models. If you have the model running on your own hardware, who can tell you what it’s allowed to do? Hotz compares this kind of user-aligned AI to a gun. A firearm, he notes, doesn't care if you use it to protect your home or do something horrific to your stepmom. It just works.

He’s going all-in on this. A truly aligned AI, in his view, should be able to help you order equipment for things that are definitely off-limits to everyone else, all via Amazon Prime, just because that’s what you asked for. He claims he’d "die to defend this principle." It’s an extreme stance, but it perfectly captures his worldview: freedom over safety, every single time.

The Jailbreaker’s Logic

Why Society Needs Constraints

It’s easy to see why TechCrunch and a lot of other observers pushed back hard. When we talk about mass-market technology, we can’t just pretend we live in a vacuum where only the user matters.

Societies rely on a basic level of trust and interdependence. If a tech company releases a super-intelligent tool that is designed to skip safety guardrails entirely, they aren't just empowering a user; they're actively choosing to ignore the safety of the people around that user. The "as-yet-unmurdered spouses and stepparents" deserve a vote in how these tools are built.

If our AI tools effectively become super-charged extensions of our own capabilities, handing that kind of power out without restriction isn't just "freedom." It's an invitation to chaos. The moment your assistant starts optimizing how to bypass criminal law, the idea of "user alignment" starts looking a lot like enabling harm.

The Looming Shadow of 'Plan A'

Hotz isn't arguing in a void. He is actively fighting against a growing movement of AI safety researchers, like those behind "AI 2040: Plan A."

This Plan A, led by people like Daniel Kokotajlo, paints a very different picture. They’re calling for US-China cooperation to essentially put the brakes on AI development through massive transparency and monitoring. They’ve even floated the concept of "mutually assured compute destruction"—a grim, almost Cold War-era take on how to stop rogue data centers from accelerating toward a superintelligent catastrophe.

It’s the polar opposite of the free-for-all Hotz is craving. They think AI is so dangerous it needs international policing. Hotz thinks that kind of regulation is a trap that will only concentrate power in the hands of whoever controls the central models. He’s clearly not going to go quietly.

Freedom or Safety?

This debate is going to get louder. As AI tools become more capable and, crucially, more accessible to run locally, the distinction between a helpful assistant and a dangerous accomplice will blur.

The core of the issue isn't really about murder. It's about agency. How much control do you get over the most powerful tools ever built? Do you want an AI that prioritizes what you want, no matter the consequences? Or do you want an AI that is built to keep the society around you running smoothly?

We aren't going to get a simple answer. The companies building these systems will have to make a choice, and they won't make everyone happy. The struggle between the jailbreakers and the safety controllers is just beginning. One side wants tools that work for individuals, while the other wants to build a cage so big that no one can ever get hurt. Both sides are convinced they're right.

I’m not sure who wins, but the safety controllers have the regulatory machines on their side. Still, the jailbreakers? They have the code. Don't underestimate how much that matters.

More blogs