Benchmarks

One client drives both engines and times them the same way: wall clock from the request write to the first streamed token, then the gap between tokens. No engine’s self-reported timer is used. Every figure names the machine it came from and the date it was taken.

A model larger than the card and the host memory

2026-08-22

openai/gpt-oss-120b, MXFP4, 36 layers, 128 experts, top-4. 24 GiB of VRAM plus 32 GiB of host memory against a 60.8 GB model: every token reads from the drive on both sides, and there is no configuration of either engine on this machine where it does not.

Decode, tokens per second — 512-token prompt, 64 tokens generated, batch 1

llama.cpp -ngl 99 -ncmoe 36 0.08

llama.cpp -ngl 99 -ncmoe 30 0.09

empanel default 8.3

empanel --io-workers 64 10.7

llama.cpp’s 0.08 tok/s is twelve seconds a token, roughly one pass over the file at the bandwidth the drive gives a single stream.

Prefill, tokens per second

llama.cpp -ngl 99 -ncmoe 36 31.7

llama.cpp -ngl 99 -ncmoe 30 47.2

empanel default 125

empanel --io-workers 64 175

The machine
GPU	RTX 4090, 24 GiB, driver 570.133.20
CPU	AMD EPYC 7B13, 256 vCPU
Host memory	capped at 32 GiB, page cache included
Drive	NVMe: 2.8 GB/s one reader, 16.95 GB/s at 16
Sizes	empanel 61.07 GB store; llama.cpp 59.02 GiB GGUF

Why the gap is what it is

llama.cpp's mmap path cannot know which experts a token will route to, so its working set is the whole 59 GiB and every token faults most of the model back in. Empanel reads the records the router asked for and nothing else: 629 MB a token at about 6.7 GB/s, starting from 28.6% of expert records resident.

What this does not show

256 EPYC cores make llama.cpp's -ncmoe path far faster than it would be on the 8 or 16 cores a machine with one 4090 usually has. The comparison is conservative in empanel's favour.
llama.cpp's -lm dio direct-I/O mode was tried and cannot work here: it turns mmap off and reads tensors into buffers, so the whole model must fit in memory. It was killed by the OOM killer before loading.
One machine. Identical GPUs have measured more than 2x apart in this project.

The full run

A model that fits in memory

2026-07-30

Qwen3-30B-A3B, int4 store on an M1 Max, against ollama 0.32.5. This is the expected result. The weights are already resident, so streaming pays for reads a resident engine never makes.

Decode by context length, tokens per second

empanel ollama

319 tok 60.9 74.5

1095 tok 44.8 69.7

3238 tok 32.5 63.2

7608 tok 19.8 51.7

End to end
workload	empanel	ollama
256 in / 256 out	46.3 tok/s	71.8 tok/s
8k in / 128 out	17.2 tok/s	50.3 tok/s

The full run

Footprint

The store is streamed rather than loaded, so what the process holds does not scale with the model. Same machine and session as the run above.

	empanel	ollama
Process start to listening	3.7 s	loads lazily on first request
First request, cold	67.6 s	2.9 s
Steady-state resident set	551 MB	20.7 GB
On disk	15.4 GB store + 2.9 GB model dir	18 GB blob

gpt-oss-120B on a laptop

The same model as the headline run, on an M1 Max with 32 GB and no discrete card. Nothing else on that machine loads it at all.

Decode	4.6 to 5.4 tok/s
Read per token	about 490 MB
Records resident	81%

Release readiness

Measure your own

empanel bench drives the server over HTTP and reports TTFT, TPOT, inter-token latency and end-to-end, each with mean, median and p99, plus store bytes read per token — the number throughput divides into.

Method and every experiment Hardware guide