vLLM probe bug can quietly inflate serving benchmarks

A small benchmark flag in vLLM can make a supposedly single-token probe do much more work than it should, which means some serving tests may be measuring the wrong thing.

In a Nov. 7 GitHub issue, a contributor reported that vllm bench serve --probe-request-rate can let probe requests inherit generation-length overrides from the main workload. That breaks the feature’s promise: the docs say probes are meant to be single-token, text-only requests that run alongside the main workload and report their own latency separately. If the probe picks up max_tokens, max_completion_tokens, or max_output_tokens from the benchmark setup, it is no longer a minimal “unrelated request” test. Instead, it can execute multiple decoding steps and inflate the probe’s own latency in a way that blurs interference from the main workload with the probe’s own generation cost. The documented behavior is described here: https://docs.vllm.ai/en/latest/benchmarking/cli/.

The bug report says the issue was reproduced on vLLM source commit c06ddca81c07ce941119a29513f670ebe409b237, with the problem visible in three benchmark backends: OpenAI completions, chat completions and Responses. According to the report, a main-workload override of 64 tokens caused probe payloads to request 64 tokens as well, even though the probe loop set output_len=1. The issue also states that when no override is supplied, the probe request stays at 1 token. Source: https://github.com/vllm-project/vllm/issues/66246.

That distinction matters because bench serve is the benchmark many teams use to compare throughput, latency and concurrency behavior on serving stacks. A probe that silently becomes a longer generation task can still complete successfully, so the error is easy to miss. But it changes the benchmark’s meaning: a measured probe percentile no longer cleanly reflects only the interference a busy server imposes on a lightweight request.

The issue author included a self-contained reproduction that does not need a model or GPU. It uses the unchanged nested probe_loop from benchmark(), a local aiohttp server, and the benchmark endpoint request functions to capture outgoing JSON payloads. In the reproduction, the captured probe requests contain max_tokens: 64, max_completion_tokens: 64 or max_output_tokens: 64 when the main workload sets those overrides, even though the probe was supposed to send one token. That makes the problem a CLI/request-construction bug rather than a model-runtime artifact. Source: https://github.com/vllm-project/vllm/issues/66246.

The same issue also links the bug to the request-building path in vllm/benchmarks/serve.py and vllm/benchmarks/lib/endpoint_request_func.py. The report says probe_loop creates a copy of the main request with output_len=1, but the endpoint helper later applies payload.update(request_func_input.extra_body), restoring the main workload’s generation limit. In other words, the code path that prepares the probe appears to enforce the one-token setting first and then overwrite it with inherited request parameters. Source links embedded in the issue point to the relevant files and lines: https://github.com/vllm-project/vllm/blob/c06ddca81c07ce941119a29513f670ebe409b237/vllm/benchmarks/serve.py#L1022 and https://github.com/vllm-project/vllm/blob/c06ddca81c07ce941119a29513f670ebe409b237/vllm/benchmarks/lib/endpoint_request_func.py#L156.

The bug’s practical impact is narrower than a full benchmarking catastrophe, but broader than a one-off typo. It does not mean every vllm bench serve result is wrong. It does mean any test that combined --probe-request-rate with a generation-length override in extra_body may have been benchmarking a heavier probe than intended. That can skew latency readings, especially if users interpret probe percentiles as a pure measure of interference from the main workload rather than a blend of interference plus extra decoding work on the probe itself.

The issue author said they planned to fix the bug by giving probes an independent copy of the extra request parameters and enforcing a one-token generation limit after inherited settings are applied, while preserving the main workload configuration. The issue also records a commit reference tied to the report, e64ee2aaff7c466ee080fe73d621be301207a4c4, indicating the bug had already drawn a code change on Nov. 7. Source: https://github.com/vllm-project/vllm/issues/66246.

There is already precedent for benchmark features in vLLM needing careful interpretation. The project’s docs note that --probe-request-rate sends single-token probes alongside the main workload, and that probe latency is reported separately. That design is supposed to isolate interference. When a probe inherits a longer output limit, the measurement no longer isolates that variable cleanly.

For teams using vLLM benchmark numbers in capacity planning, regression tracking or vendor comparisons, the practical takeaway is simple: rerun any serving tests that used --probe-request-rate together with extra_body generation-length overrides. The safest assumption is that older probe latency numbers may have mixed in extra decode time unless you verified the outgoing probe payloads. If your benchmark used only the default one-token probe path, the issue report says those control cases still sent a limit of 1.

For readers building LLM serving tests, the bug is a reminder that benchmark harnesses can silently drift from the behavior they advertise. In this case, the advertised probe was a one-token check for interference; the actual request could become a multi-token generation. That is enough to change what the numbers mean, even if the server itself is functioning normally.