One Request, Traced

llama.cpp or vLLM?

The same chat request, sent to both servers on one RTX 4090, followed through their source and timed.

  1. How far does vLLM pull ahead as more people chat?
  2. Who reuses your prompt better?
  3. Where does your request cross a thread or process boundary?
  4. What happens when the KV cache runs out of room?

So which one?

How this was measured

Both servers run the same model on the same card and get the same requests.

Hardware
One RTX 4090, 24 GB
Model
Llama 3.1 8B Instruct, 16-bit
Versions
Server settings
Both servers get the same 32,768-token KV cache and restart before every level; llama.cpp runs one slot per user.
Show details

llama.cpp serves an F16 GGUF with 64 slots and a 32,768-token context (-np 64 -c 32768, 512 tokens per slot), and 8 slots for the prompt reuse test. vLLM serves BF16 weights with 2,048 cache blocks, which is the same 32,768 tokens, and a maximum model length of 512 tokens (4,096 for prompt reuse, 2,048 for the KV cache test). The KV cache test gives llama.cpp one shared 32,768-token pool (--kv-unified) to match. Each server restarts before every level. Before each measured level, 4 short requests are sent and discarded so first-request start-up work is not timed; the prompt reuse chart comes from a control run whose 4 warm-up requests use unrelated prompts, because without them the first message also paid start-up costs (202 ms on llama.cpp and 380 ms on vLLM). Each level runs 30 seconds of warm-up, then a 60-second measured window. llama.cpp at 256 tokens in the KV cache test completed only 7 to 18 requests in its window, a small sample. llama.cpp ran with one slot per user; with fewer slots, published results show a much wider gap. Every flag is in NOTES.md.

Against published numbers
Within about 7% of published results for one user on llama.cpp and for many users on vLLM, on the same card.
Show details

One user on llama.cpp: llama-bench reports 54.3 tokens per second for Llama 3 8B F16 on an RTX 4090, against 50.4 here. That is about 7% lower, as expected through the HTTP server with sampling, and 50.4 is about 80% of the physical ceiling of about 63 (16 GB of weights read per token at 1,008 GB/s). vLLM with many users: 2,285 to 2,374 output tokens per second for an 8B Llama-architecture model (DeepSeek-R1-Distill-Llama-8B, 16-bit) on one RTX 4090 with 300 requests at once, against 2,209 here at 64 users. llama.cpp prompt processing: 6,500 to 9,400 tokens per second published for 512 to 4,096 token prompts (same llama-bench source), against about 9,200 here (the cold prompt of about 1,500 tokens with its chat template, in 166 ms). The gap at 64 users: Red Hat measured about 44 times on an H200 without stating llama.cpp's slot count, against 2.5 times here with one slot per user.

Method
Each level is a closed loop: every simulated user sends the next request when the previous one finishes, so requests tend to arrive in groups. Three repeats per level, medians shown. Rerun instructions in bench/README.md; raw results and caveats in NOTES.md.

More engines will be added, each as a data file, and each gets the same rows.