Models / Gemma 4
Gemma 4
empanel run gemma-4
empanel serve gemma-4 --port 6863curl http://localhost:6863/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma-4",
"messages": [{"role": "user", "content": "Name three prime numbers."}]
}'from openai import OpenAI
client = OpenAI(base_url="http://localhost:6863/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="gemma-4",
messages=[{"role": "user", "content": "Name three prime numbers."}],
)
print(resp.choices[0].message.content)Applications
- Open WebUI
OpenAI connection → http://localhost:6863/v1 - Continue
apiBase: http://localhost:6863/v1 - Zed
api_url: http://localhost:6863/v1 - Aider
--openai-api-base http://localhost:6863/v1 - LangChain
ChatOpenAI(base_url="http://localhost:6863/v1", api_key="not-needed") - LlamaIndex
OpenAILike(api_base="http://localhost:6863/v1", api_key="not-needed") - Vercel AI SDK
createOpenAI({ baseURL: "http://localhost:6863/v1", apiKey: "not-needed" })
Notes
The converter reads a local checkpoint and never the network, requantizing it into the store’s own grouped-affine int4 or int8. A sample of experts is dequantized and checked against the source weights before the run is called a success.
The server binds 127.0.0.1 and does not authenticate. Put a reverse proxy
in front of it to expose it.