empanel
Install

Run the largest open models
on the machine you already own.

Empanel streams Mixture-of-Experts weights from NVMe and loads only the experts each token actually needs.

Quickstart
$curl -fsSL https://empanel.ai/install.sh | sh
held in cache used this token read from NVMe Illustration. One column per expert, one row per layer.

Bigger than memory

The whole model sits on the drive. What your GPU holds decides how much stays cached, not which models you can load at all. Pick a pair and see the split.

cached on the GPU
warm in host memory
read from the drive

How the store is split

Measured on real hardware

Two runs on two machines. Each row is what that box produced, not a projection.

modelmachine tokens/secread per token
gpt-oss-120B61 GB, MXFP4 M1 Max, 32 GBMetal 4.6–5.4 490 MB
Qwen3-30B-A3B14 GB, int4 RTX 3070 Ti, 8 GBCUDA 13.3 296 MB

Sources: release readiness (gpt-oss-120B, Metal) and the CUDA suite (Qwen3-30B-A3B, 8 GB card). Both greedy and repeatable.

See the benchmarks

Every token, three tiers

The router picks a handful of experts per layer. Each is served from the fastest tier that already has it, and only a miss touches the drive.

router picks the top-k VRAM cache hit: stays on the device miss pinned host RAM the hot layers miss NVMe store every routed expert MoE GEMM then the next token
How far a token falls depends on what is already cached, and it differs from token to token. Averaged over the measured gpt-oss-120B run, 81% of records were already resident and about 490 MB was read per token.
What a miss costs

Keep your setup

An OpenAI-compatible API on port 6863, plus ollama’s /api/* over the same engine. Point what you already use at it and change nothing else.

  • Open WebUIOpenAI connection
  • ContinueapiBase
  • ZedOpenAI provider
  • Aider--openai-api-base
  • LangChainChatOpenAI
  • LlamaIndexOpenAILike
  • Vercel AI SDKcreateOpenAI
  • Prometheus/metrics
See all integrations