One Request, Traced

Tracers and code

How each engine handles one chat request: an interactive map of each codebase, then all four stages side by side, each with its code path and features.

The tracers

Interactive maps of the real code. Press Trace and the request walks itself, one function at a time, each linked to its source line.

Close-up of the llama.cpp tracer on step 13, Pick a slot, with the step card explaining it

llama.cpp tracer

C++, 53 steps. One process, one inference thread. Watch the request get a slot, run through ggml on the GPU and stream back.

Open the tracer Close-up of the llama.cpp inference tracer on step 11, Gate, up and SwiGLU, at 2.31 ms and 40% of a decode step

llama.cpp inference tracer

Zoomed in, 17 steps. One token through Llama 3.1 8B on the GPU, op by op, with each step's measured CUDA kernels.

Open the tracer Close-up of the weight tape with the playhead on layer 0's gate matrix, and the real values it produced

llama.cpp through the weights

The same pass as a 4.4 GB tape of weights, 19 steps. Each matrix in the order it is read, with the real numbers that came out.

Open the tape Close-up of the attention map of layer 10, head 27, with the prompt tokens on both axes

llama.cpp attention maps

Where all 1,024 heads look while reading “Why is the sky blue?”. A grid of every head, and a 41 × 41 map for each.

Open the maps Close-up of the logit lens table: layers down, prompt tokens across, each cell the word the model would say from there

llama.cpp logit lens

What the model would say from every layer and position, how “The” forms in the last two layers, and which heads and feed-forward blocks wrote it.

Open the lens Close-up of the vLLM tracer on step 18, Schedule, next to Admit waiting request

vLLM tracer

Python, 59 steps. Two processes over ZMQ, a token-budget scheduler and a paged KV cache.

Open the tracer Close-up of the vLLM module graph on step 4, ZMQ hand-off to the engine core

vLLM module map

Zoomed out, 18 steps. The same trip drawn across vLLM's 164 modules.

Open the map

Step with the arrow keys, click any box for details. Best on a laptop or bigger.

Every stage

Each engine's steps, time and features for every stage. Pick a stage below, or open a row, to see its code path.