The major LLM inference servers in one table, then the tools that trace requests from the app side. This site traces and benchmarks two of the engines; everything else is here for context, from each project's docs, not tested on this site.
The servers that load the weights and run the model. Grouped by what each is for, what hardware it runs on, which models it takes, how it serves, and the project behind it.
llama.cpp and vLLM are traced and benchmarked hereAs of October 2026. Engines move fast; check the repo before relying on a cell.
| Category | vLLMTraced and benchmarked | SGLang | TensorRT-LLM | LMDeploy | TGIHugging Face Text Generation Inference. Archived | llama.cppTraced and benchmarked | Ollama | LM StudioDesktop app | MLX LM | MLC LLM |
|---|---|---|---|---|---|---|---|---|---|---|
| What it is for | ||||||||||
| Built for | Serving many users at once from datacentre GPUs | Serving many users at once, especially agents and multi-call programs | The highest throughput on NVIDIA datacentre GPUs | Serving many users at once on NVIDIA and Huawei Ascend | Production serving (archived) | Running models anywhere: laptops, desktops, CPUs, phones, any GPU | Running models locally with one command | Running models locally in a desktop app, no terminal needed | Running and fine-tuning models on a Mac | Compiling a model to run natively on phones, in browsers and on any GPU |
| Users at once | Many; built for high concurrency | Many | Many | Many | Many | A few: a fixed number of parallel slots | A few per loaded model | A few: 4 parallel requests by default | One or a few | Mostly one, on a device |
| Setup | pip or Docker on Linux with a supported GPU | pip or Docker on Linux with a supported GPU | The most work: NVIDIA containers and per-model tuning | pip or Docker | Docker | One binary: download a build or compile for your GPU | An installer, then ollama run |
An installer with a GUI; the easiest here | pip on a Mac | Compile each model for each target device |
| Good fit for | High-throughput serving on datacentre GPUs; the widest model support | Agents and multi-turn chat with lots of shared prefix; large MoE models | The last bit of speed on recent NVIDIA GPUs, if you accept the setup | Fast 4-bit serving; InternLM and vision-language models | Existing deployments only. Archived in March 2026 after a maintenance-mode period; Hugging Face recommends vLLM and SGLang, or llama.cpp and MLX locally | Local and edge: laptops, CPUs, mixed CPU and GPU, small footprint | The easiest local setup: one person, one machine | Trying models on your own PC with a GUI; AMD and Intel GPUs through Vulkan | Running and fine-tuning models on a Mac | Shipping a model inside a phone app or a web page |
| Hardware | ||||||||||
| NVIDIA (CUDA) | Yes | Yes | Yes, the only target | Yes | Yes | Yes | Yes | Yes | No | Yes |
| AMD (ROCm) | Yes | Yes (Instinct) | No | Not listed | Yes | Yes (HIP) | Yes | Yes, through its ROCm runtime | No | Yes |
| Vulkan (any GPU vendor) | No | No | No | No | No | Yes, a full backend: AMD, Intel, NVIDIA and mobile GPUs | Yes, on by default on Windows and Linux when the Vulkan backend is installed | Yes, through its llama.cpp Vulkan runtime | No | Yes, its main path for AMD and Intel GPUs |
| Apple (Metal) | No | Apple silicon listed as supported | No | No | No | Yes | Yes | Yes, with llama.cpp or MLX | Yes, the only target | Yes, Mac and iOS |
| CPU | Yes (x86, Arm) | Yes (Intel CPUs) | No | Not listed | Not listed | Yes, a first-class target | Yes | Yes | No | Not listed |
| Other | Intel GPUs, Google TPU; Ascend, AWS Neuron and more through plugins | Google TPU, Intel GPUs, Huawei Ascend, Moore Threads | None | Huawei Ascend and others through its PyTorch engine | Intel GPUs and Gaudi, Google TPU, AWS Inferentia | SYCL (Intel), CANN (Ascend), OpenCL (Adreno) and more | None | None | None | WebGPU in the browser; Android through OpenCL |
| Models | ||||||||||
| Model format | Hugging Face safetensors; GGUF experimental | Hugging Face safetensors | Hugging Face checkpoints, or compiled TensorRT engines (older path) | Hugging Face safetensors | Hugging Face safetensors | GGUF (converted from Hugging Face) | GGUF from its model library; can import safetensors | GGUF, and MLX on Macs | MLX safetensors (converted from Hugging Face) | MLC-compiled weights (converted from Hugging Face) |
| Quantization | FP8, AWQ, GPTQ, INT8, bitsandbytes, compressed-tensors | FP8, FP4, AWQ, GPTQ, INT8 | FP8, NVFP4, INT4 AWQ, INT8 SmoothQuant (via NVIDIA ModelOpt) | 4-bit AWQ, W8A8, FP8; INT4 and INT8 KV cache | AWQ, GPTQ, Marlin, FP8, bitsandbytes | 1.5 to 8-bit k-quants and i-quants; quantized KV cache | GGUF quants (4-bit by default); quantized KV cache | GGUF quants; MLX quants on Macs | 2 to 8-bit group quantization | 3 and 4-bit group quantization, or FP16 |
| Serving | ||||||||||
| Batching | Continuous batching under a token budget per step; chunked prefill | Continuous batching; CPU scheduling overlapped with the GPU step; chunked prefill | In-flight (continuous) batching | Persistent (continuous) batching | Continuous batching in the Rust router | A fixed number of parallel slots, batched together each step | A few parallel requests per loaded model | Continuous batching from llama.cpp since 0.4, and for MLX since 0.4.2 | Batch generation API; the server batches concurrent requests, except with a quantized KV cache | Continuous batching in its server |
| KV cache and prefix reuse | Paged KV blocks (PagedAttention); automatic prefix caching by block hash | RadixAttention: a radix tree of cached prefixes shared across requests | Paged KV cache; KV reuse across requests; offload to host memory | Blocked KV cache; prefix caching | Paged attention; prefix caching | One KV buffer split across slots; each slot reuses its longest matching prefix | Inherits the ggml engine's per-request prompt cache | Inherits llama.cpp and MLX caching; a unified KV cache by default | Rotating KV cache; prompt caches you can save to disk and reload | Paged KV cache |
| Multi-GPU | Tensor, pipeline, data and expert parallel; multi-node | Tensor, pipeline, data and expert parallel; prefill and decode on separate machines | Tensor, pipeline and expert parallel; multi-node; disaggregated serving | Tensor parallel; data and expert parallel for MoE models | Tensor parallel | Split by layer or by row; RPC backend for several machines | Spreads layers across the GPUs it finds | Splits layers across GPUs (llama.cpp) | Distributed inference over mx.distributed |
Tensor parallel across GPUs |
| Structured output | JSON schema, regex, grammars (XGrammar, Guidance) | JSON schema, regex, EBNF (XGrammar by default) | JSON schema and grammars (XGrammar) | JSON schema and regex | JSON schema and regex; tool calls | GBNF grammars; JSON schema converted to a grammar | JSON schema through the format field |
JSON schema | None built in | JSON schema (XGrammar comes from the same team) |
| Project | ||||||||||
| Made by | Started at UC Berkeley; now a PyTorch Foundation project | Hosted by LMSYS, a non-profit | NVIDIA | InternLM team, Shanghai AI Lab | Hugging Face | ggml-org community | Ollama | LM Studio | Apple (ml-explore) | The mlc-ai community, started at Carnegie Mellon |
| Written in | Python, with C++ and CUDA kernels | Python, with its own CUDA kernel library | Python and C++, CUDA kernels | Python; C++ and CUDA in the TurboMind engine | Rust router, Python model server | C and C++ (ggml) | Go, on a ggml-based engine | A desktop app over the llama.cpp and MLX engines | Python on MLX (C++ and Metal) | Python and C++ on Apache TVM |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 | MIT | MIT | Proprietary app, free at home and at work; open-source SDKs | MIT | Apache 2.0 |
| API | OpenAI-compatible server; offline Python API | OpenAI-compatible server; a Python frontend language for multi-call programs | OpenAI-compatible trtllm-serve; Triton backend |
OpenAI-compatible server; Python pipeline | Its own /generate plus an OpenAI-compatible Messages API |
OpenAI-compatible llama-server with a web UI; C library |
Its own REST API plus OpenAI-compatible; CLI and desktop app | OpenAI-compatible server, its own REST API, Python and TypeScript SDKs, lms CLI; headless as llmster |
OpenAI-style server; Python API; LoRA fine-tuning in the same package | OpenAI-compatible API over REST, Python, JavaScript, iOS and Android |
Want numbers rather than features? The comparison page benchmarks llama.cpp against vLLM on one RTX 4090.
Tools that trace requests from the app side: every model call your app makes, with its prompt, response, latency and cost. They see the call from outside; the tracers on this site show what happens inside the server during one.
| Category | Langfuse | LangSmith | Arize Phoenix | Opik | HeliconeMaintenance mode | W&B Weave | Braintrust | OpenLLMetryInstrumentation only, no backend |
|---|---|---|---|---|---|---|---|---|
| Made by | Langfuse, part of ClickHouse since January 2026 | LangChain | Arize AI | Comet | Helicone; the team joined Mintlify in 2026 | Weights & Biases | Braintrust | Traceloop |
| License | MIT (core) | Proprietary platform; open-source SDKs | Elastic License 2.0 (source-available) | Apache 2.0 | Apache 2.0 | Apache 2.0 SDK; the backend is the W&B platform | Proprietary platform; open-source SDKs | Apache 2.0 |
| Self-host | Yes, free (Docker, Kubernetes) | Enterprise plan only | Yes, free (local, Docker, Kubernetes) | Yes, free (Docker Compose, Kubernetes) | Yes (Docker Compose) | Through W&B's self-managed or dedicated deployments | Hybrid: data plane in your cloud, on enterprise plans | Not applicable: sends data to the backend you choose |
| How data gets in | Python and JS SDKs, OpenTelemetry, framework integrations | SDKs (tight LangChain and LangGraph integration), OpenTelemetry | OpenTelemetry, with OpenInference auto-instrumentation | Python and TypeScript SDKs, OpenTelemetry | A proxy gateway (change the base URL), or async logging | Python and TypeScript SDKs (a decorator on your functions) | SDKs, a proxy, OpenTelemetry | OpenTelemetry instrumentation for LLM providers and vector DBs |
| Evals | LLM-as-judge, datasets, experiments, human annotation | Datasets, experiments, LLM-as-judge, annotation queues | LLM-based evals, versioned datasets, experiments | Datasets, experiments, LLM-as-judge, online evals, PyTest | Basic scoring | Scorers, datasets, evaluation comparisons | The core of the product: datasets, scorers, experiments | None (it only produces traces) |
| Prompt management | Yes, versioned, with a playground | Yes, a prompt hub and playground | Yes, versioned and tagged | Yes, with a playground | Yes, versioned, deployable through the gateway | Yes, prompts as versioned objects | Yes, with a playground | No |
| Good fit for | The default open-source pick; self-host it for free | Teams already building on LangChain or LangGraph | Local debugging on your own machine; OpenTelemetry-first teams | A fully open-source, self-hosted alternative to Langfuse | Existing users. No new features; it gets fixes only | Teams already on W&B for training runs | Eval-heavy teams that want a hosted product | Sending LLM traces to an observability stack you already run (Datadog, Honeycomb, Grafana) |