The Ghost Window
Here's the thing nobody tells you when they hand you an API key and say "rotate it if it leaks": deletion isn't a kill switch. It's more like pulling the plug on a machine that's already mid-request. The current operation finishes. Maybe a few more finish after that, too — queued up in some distributed buffer you can't see. And then, eventually, the whole thing goes dark.
That gap between clicking "delete" and the key actually dying? That's what Joe Leon at Aikido Security calls a ghost window. And it's not theoretical. It's measurable. It's minutes to hours long, depending on how the key was being used and where it lived in Google's infrastructure. For an attacker who caught that key before you rotated it, those minutes are everything.
I've been thinking about this for a while now. We treat credential lifecycle like it's binary — active or dead, on or off. But the reality is messier, and messier means more dangerous.
What Leon Actually Found
Leon's research, reported by Dark Reading, documents a specific behavior in Google's API key infrastructure: when you delete a key through the Cloud Console or via the IAM API, the deletion propagates asynchronously. The key doesn't vanish from every serving layer at the same instant. Some layers honor the deletion immediately. Others keep honoring it for a window that can stretch from minutes to hours.
This isn't a bug in the traditional sense. It's a consequence of how Google distributes authentication state across its global infrastructure. API keys get cached at edge nodes, replicated through regional identity services, and validated against multiple backend systems that each have their own consistency model. When you delete a key, the deletion signal has to fan out to all of those systems. And fan-out takes time.
The practical implication is brutal: if an attacker has your key and you delete it at 2:00 PM, they might still be able to use it at 2:47 PM. Or 3:15 PM. Or later, depending on which part of the infrastructure they're hitting.
That's not a small window. That's an attack surface.
Why This Matters More Than You Think
Most teams I talk to treat API key rotation as a checklist item. "Leak detected? Delete the old one, generate a new one, move on." But Leon's finding exposes a fundamental flaw in that mental model: you're assuming the old key is dead the moment it's gone from your console. It isn't.
Consider what an attacker does with a stolen API key. They don't just fire one request and call it done. They probe. They enumerate. They try different endpoints, different scopes, different service accounts attached to that key. And they do it fast — usually within minutes of acquisition. But here's where the ghost window becomes a problem: if you delete that key after they've already stolen it, you're not actually stopping them. You're just making the key disappear from your own view while it keeps working on theirs.
This is especially dangerous for teams that use API keys with broad permissions. A key with access to Cloud Storage, BigQuery, and Compute Engine isn't just a minor credential — it's a skeleton key. And if that key stays active for an hour after you delete it, the attacker has a full hour to do whatever they want with those services.
I've seen teams panic-delete keys during incidents and then sit back, convinced the threat was neutralized. Leon's research shows that confidence is premature.
The Mechanics of the Delay
So why does this happen? It comes down to distributed systems 101: consistency versus availability.
Google's API key validation happens at multiple layers. When a request comes in, it hits an edge node first. That node checks its local cache of valid keys. If the key is in the cache, the request proceeds. If it's not, the node queries a regional identity service, which in turn may query a central directory. Each of these layers has its own TTL (time-to-live) for cached entries.
When you delete a key, the deletion signal propagates from the central directory outward. But each layer has to expire its local cache before it stops honoring the deleted key. Edge nodes might have TTLs of minutes. Regional services might cache for longer. And there's no guarantee that every node in the fleet receives the deletion signal at the same moment.
The result is a cascading delay. Some requests fail immediately after deletion. Others succeed for minutes or longer, depending on which layer is validating them and how fresh that layer's cache is.
This isn't unique to Google, by the way. AWS has similar behavior with IAM credentials. Azure has it too. But Leon's work is notable because he actually measured the window and made it concrete instead of leaving it as an abstract "eventual consistency" caveat.
What Teams Should Actually Do
So what's the fix? Here's where I get opinionated, because I think most teams are doing this wrong.
First, stop relying on deletion as your primary incident response. When you detect a leak, don't just delete the key. Immediately revoke its permissions at the project or organization level. Rotate the associated service account credentials if it's tied to one. And monitor for anomalous usage of that key during the ghost window — because it will still work.
Second, scope your keys aggressively. The narrower the permissions on each key, the smaller the blast radius when a ghost window exists. A key that can only read from one specific Cloud Storage bucket is annoying but not catastrophic if it lingers. A key with project-wide editor access? That's a disaster waiting to happen.
Third, implement real-time alerting on API key usage. If you're not monitoring who's calling your APIs, when they're calling, and from where, you're flying blind during the ghost window. Set up alerts for unusual patterns — sudden spikes in requests, calls from unexpected IP ranges, access to endpoints that aren't part of normal traffic.
Fourth, consider short-lived credentials instead of long-lived API keys. OAuth tokens with expiration times, IAM workload identity federation, short-lived service account tokens — these all self-expire. There's no deletion to wait on. The credential just stops working when its TTL runs out.
I know API keys are convenient. They're simple. You paste them into a config file and forget about them. But convenience isn't free, and the cost here is that ghost window — a window where your "deleted" key is still very much alive in the hands of someone who shouldn't have it.
The Bigger Picture
Leon's research is a reminder that security isn't about individual controls — it's about the gaps between them. The ghost window exists because of how distributed systems work, not because Google was negligent. But it exists anyway, and that's what matters.
We keep building security postures around the assumption that controls are instantaneous. Delete a user? They're locked out now. Revoke a certificate? It's invalid immediately. Block an IP? Traffic stops.
The reality is that most of these controls have latency. Some of it is network latency, which we accept as normal. But some of it is architectural — the time it takes for state to propagate through a distributed system. And that latency is where attackers live.
The teams that will survive the next incident aren't the ones with the best tools. They're the ones who understand that their controls have gaps, and they've built detection and response around those gaps instead of pretending they don't exist.
Google's API key deletion isn't broken. It's just honest about how distributed systems actually work. The question is whether we're honest about what that means for our security.
Key Takeaways
- API key deletion in Google Cloud is asynchronous — deleted keys can remain active for minutes to hours.
- The delay comes from distributed cache propagation across edge nodes, regional services, and central directories.
- An attacker who has already stolen a key isn't stopped by your deletion — they keep working during the ghost window.
- Scope keys narrowly, monitor usage in real time, and prefer short-lived credentials over long-lived API keys.
- Security controls have latency. Build your response around the gaps, not just the controls themselves.