empanel
Install

Architecture

Empanel keeps the whole model on NVMe and streams only the active experts each token.

throughput = delivery bandwidth cold bytes per token

Cold bytes per token depends only on the active parameters, so total model size becomes a storage cost rather than a speed wall.

Conversion splits the checkpoint

The resident file

Everything used on every token: attention, whether multi-head latent or grouped-query, the router, the layer norms, the shared experts and the embeddings. Small enough to live in memory.

The expert store

The routed experts, quantized to int4 or int8 and packed into one flat file addressed by (layer, expert). This is the part that streams. Its contents are hashed against the digest the metadata carries before a byte is used.

Experts are dequantized on the GPU from the packed bytes with fused int4 kernels, so the cache holds two to four times more experts per byte than it would if it cached dequantized weights.

Every token, three tiers

The router picks a handful of experts per layer. Each is served from the fastest tier that already has it, and only a miss touches the drive. On a default install there are two tiers, not three: the host tier is opt-in.

router picks the top-k VRAM cache hit: stays on the device miss pinned host RAM the hot layers miss NVMe store every routed expert MoE GEMM then the next token
How far a token falls depends on what is already cached, and it differs from token to token. Averaged over the measured gpt-oss-120B run, 81% of records were already resident and about 490 MB was read per token.
  1. 1

    Device expert cache --cache-gb on by default

    Packed expert bytes held on the device, carved out of one allocation as fixed-size slots. A hit never leaves the card. Left at 0 it sizes itself from what the device reports free.

  2. 2

    Host memory --ram-gb / --pin-gb off by default

    Two separate opt-ins, both 0 by default. --ram-gb holds leading MoE layers in ordinary host memory; --pin-gb holds the first N GB page-locked, so those records reach the device without a read or a staging copy.

  3. 3

    NVMe store --store always

    Every routed expert, read with pread on a worker pool that splits a batch by bytes rather than by record, so a two-expert miss can still use every worker.

The host tier is a cliff on unified memory

Where the GPU shares memory with the CPU, the host tier and the device cache are the same physical memory, so asking for both holds every record twice and the second copy swaps. Measured on a 32 GB M1 Max against the 14 GB Qwen3-30B-A3B store: --ram-gb 18 served 7.36 tok/s where the same build without it served 54.82, with 6.5 GB in swap. The engine prints a warning rather than leaving the number to explain itself.

What a token does

  1. Attention and the router run from the resident weights, giving the top-k experts for each layer.
  2. Those experts are fetched through the tiers as one batched, concurrent read, dequantized on the GPU, and applied.
  3. The engine can run the router a layer or more early, so a layer's choices are known before the layers between are computed. It is causal: the router runs on the residual stream as it stands.

Why the early router does not prefetch by default

Prefetching never reduces the bytes read. A prefetcher fetching p experts at recall r still moves p+k ( 1−r ) ≥k . It only converts idle drive time into useful drive time, so over-fetching is not a free hedge but directly negative throughput. The guess therefore orders eviction by default, which costs nothing and keeps what the next layers want. Fetching on it is opt-in behind --fetch-guesses 1.

What a miss costs

Qwen3-30B-A3B on an M1 Max with --page-cache drop, so the drive is the thing being measured. As the cache shrinks, residency falls and the token slows — but the read is not what is wrong.

Decode, tokens per second, by VRAM cache size
16 GB 100% resident 53
8 GB 94% resident 15.5
4 GB 64% resident 5.5
2 GB 54% resident 4.6
cacheresidentread a tokencompute a token
16 GB 100% — 19 ms
8 GB 94% 16.8 ms 47.7 ms
4 GB 64% 67.6 ms 114 ms
2 GB 54% 80.6 ms 136 ms

At 54% resident a token misses about 443 MB and moves it in 80.6 ms, which is 5.5 GB/s against the 7.02 this drive gives at queue depth 16 — 78% of it. What is wrong is that a streaming token computes in 47 to 136 ms where a resident one computes in 19.

The streaming pass

Why MoE-native matters

Layer-by-layer offloaders move every expert of a layer each token, throwing away the sparsity that makes a Mixture-of-Experts model worth running. Empanel moves only the experts actually selected.

Depth is the other half. Saturating a fast RAID needs many reads in flight, and one layer of lookahead cannot supply them. Batching reaches further than prediction: the chance a layer's expert is wanted by some token in a step is 1− ( 1−kE ) T , which at a few tens of tokens approaches one — so a whole-layer fetch becomes nearly all useful with no predictor at all. Tokens in flight share expert reads, and aggregate throughput climbs with load until the drive saturates.

Full architecture notes On-disk format Kernels Benchmarks