A 35B-class sparse Mixture-of-Experts LLM that streams experts from SSD on demand — so it runs at interactive speed in just ~3 GiB of active memory, in 4-bit.
Interactive simulation of a single forward pass under the edge0 framework: expert weights stream from storage only as routed, while a trained prerouter pre-loads next step's experts. Hit run and watch a token decode.
Full 4-bit weights stay on storage. Only the experts the router touches — the active set — are streamed into RAM, so peak memory is bounded by activity, not parameter count.
All 256 expert blocks live on disk. The router fetches the 4 it needs per token just in time, from NVMe or internal flash, so RAM never holds the full model.
A trained head predicts next-step routing one token ahead, so expert loads overlap compute instead of stalling it — like instruction prefetch for an LLM.
The int4 base is frozen; LoRA adapters are distilled from the FP16 teacher to claw back quantization loss. One read-only base serves many adapters, unmerged.
OpenCompass, identical settings, max score 100. The edge0 int4 + adapters pipeline loses 3.9 points on average versus the fp16 base.
| Decode | Prefill cold / warm | Peak active memory* | |
|---|---|---|---|
| edge0-35b | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB |
*Short contexts; long contexts add KV cache. Experts stream from SSD and are not resident.
| Item | Value |
|---|---|
| Parameter count | 35B total, A3B active |
| Adapter contents | lora_edge0_35b.safetensors · prerouter_edge0_35b.safetensors |
| Chat template | thinking-mode enabled (multilingual reasoning) |
| Backend | edge0 streaming framework · MLX (Apple Silicon) |
The repo is a complete, ready-to-run model directory — base, LoRA and prerouter co-located and loaded automatically.
1 · Install edge0
# MLX backend — requires Apple Silicon pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
2 · Download the model
huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview
3 · Chat or serve
# Interactive chat export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview edge0 chat --name edge0-35b --prompt "Introduce yourself" # OpenAI-compatible HTTP server edge0 serve --name edge0-35b --port 8085
Connect anything that speaks OpenAI
mlx_lm.server / Pi / Hermes / OpenClaw at http://127.0.0.1:8080/v1