Cloudflare’s Workers AI leak came from a KV cache handoff failure

Cloudflare says a tenant isolation bug in its GLM-5.3 Workers AI deployment exposed data from other customers’ requests in 155 responses between Aug. 27 and Sep. 9, 2026. The leak was not tied to a broad platform outage or a model jailbreak. It came from a specific failure in how Cloudflare and SGLang handled KV cache transfer for very long prompts.

The key technical detail is stark: prompts longer than 681,280 tokens could pass through a path where the request was rejected by prefill, but the decode side did not learn that rejection correctly. Instead of failing cleanly, the system handed decode a cache it believed belonged to the current request. Decode then read stale KV cache pages that still held fragments of previous customers’ prompts and generations.

What happened

Cloudflare says it was alerted on Sep. 8 by researcher Gadi Evron of Knostic through HackerOne. Cloudflare reproduced the issue the next morning, took the model offline within 40 minutes of declaring an incident, deployed a corrected configuration and patched inference engine, and restored service on Sep. 9.

The affected model was @cf/zai-org/glm-5.3 on Workers AI. Cloudflare says no other model hosted on Workers AI was affected.

The exposure window began when GLM-5.3 launched on Aug. 27 with a 1,048,576-token context window but a prefill KV cache pool sized for only 681,280 tokens. Cloudflare says it knew about the mismatch before launch, but expected the engine to return an error for requests it could not process. In testing, long-context requests returned successful responses instead.

That assumption was wrong.

The failure mechanism

Cloudflare’s postmortem says the incident required two overlapping issues:

1. A configuration mismatch: the KV cache pool on prefill GPUs was too small for the advertised context window.
2. An SGLang bug in disaggregated prefill/decode mode: when a request exceeded the pool, prefill marked it as rejected and replaced it with a one-token placeholder, but did not stop the request cleanly.

Instead, prefill generated a KV cache for the placeholder prompt and transferred it to decode. Decode received a “transfer complete” signal and assumed the current request’s KV cache had been fully populated. In reality, most of the prior cache state remained intact. When decode generated the answer, it attended over pages that still contained data from earlier requests.

Cloudflare says the result was responses that could contain other users’ content, and that every affected response came from prompts above the 681,280-token threshold.

Why the bug was security-relevant

This was not a model hallucination or prompt injection issue. It was a memory ownership failure in a distributed inference stack.

Cloudflare describes the KV cache as a shared pool of fixed-size pages in GPU memory. Those pages are allocated to a request when it is admitted and released when it finishes. They are not zeroed between requests; the next request that gets a page overwrites it. That is a normal performance tradeoff in inference systems, but it makes the engine responsible for tracking ownership correctly.

In this case, ownership tracking broke at the boundary between prefill and decode.

What Cloudflare changed

Cloudflare says it fixed the incident before restoring service by doing three things:

• Matched the pool to the context window, so the prefill KV pool could hold the full GLM-5.3 context.
• Patched the engine so a request carrying a pending rejection is retired at the admission queue with its original error before bootstrap, allocation, or KV transfer happens.
• Added verification and hardening around the handoff, including a check that the number of tokens transferred matches the number reserved.

Cloudflare says it proposed the key engine fix upstream as #38704. It also says a related fix had already landed on SGLang main in #36988 on Sep. 6, and that maintainers later merged #38935, which rejects these requests earlier.

Cloudflare validated the changes against the researcher’s reproduction script and its own canary-marker tests before restoring the model.

Why this matters for edge AI builders

The practical lesson is not “shared caches are bad.” The lesson is narrower and more useful: performance shortcuts in multi-tenant inference need explicit ownership checks at every handoff.

Cloudflare says the issue was specific to GLM-5.3, but it also notes that other SGLang-dependent operators should update to v0.5.22 or later. The company further says it has applied the engine fix and related hardening to every model it serves with SGLang.

For builders running inference at the edge, the failure mode is easy to miss because it looks like a throughput optimization, not a security feature. But the KV cache is part of the trust boundary. If a request can be rejected in one stage and still proceed to another, the system needs a hard fail, not an assumed handoff.

That implies a concrete engineering checklist:

• reject over-limit requests before any cache transfer begins;
• verify transfer size against reservation size;
• track page ownership across prefill, decode, and abort paths;
• test the exact over-limit path, not just successful prompts;
• assume stale cache state is sensitive until proven otherwise.

Cloudflare’s own report also shows why reproductions matter. The leak was not discovered by a generic scanner or a policy audit. It was caught through a researcher report and confirmed with a reproduction script.

The strongest reading of the incident

The strongest reading is also the simplest: this was a narrow implementation bug with a large privacy consequence.

Cloudflare says the affected surface was limited to one model path, one size threshold, and 155 responses. That does not make the problem trivial. It makes it diagnosable. The bug lived at the seam between an inference engine’s performance assumptions and its isolation guarantees.

That seam is where edge AI systems will keep failing if operators treat cache ownership as an internal detail rather than a security requirement.

Sources

• Cloudflare postmortem: https://blog.cloudflare.com/kv-cache-exposure-remediation/
• Related incident timeline and status page: https://www.cloudflarestatus.com/incidents/jsvvrtk5hvym
• SGLang fixes referenced by Cloudflare: https://github.com/sgl-project/sglang/pull/38704 , https://github.com/sgl-project/sglang/pull/36988 , https://github.com/sgl-project/sglang/pull/38935