EdgeRock-4B
Rock Matrix, with HKUST and CMU, open-sourced EdgeRock-4B, a Qwen3.5-4B conversion that holds its KV cache at 72 MiB at 256K context instead of 8 GiB.
The team converted Qwen3.5-4B in stages instead of pretraining. It kept the 24 Gated DeltaNet layers and turned the 8 full-attention layers into 1,024-token sliding-window attention. Rock Memory Anchors keep the top 1,024 tokens from every 256 candidates per KV head group, so the model retains information that leaves the window.
- Date
- Tuesday 27 October 2026
- Lab
- Rock Matrix
- Kind
- open-weights
- Access
- open weights (restricted license)
Figures
| Measure | Value | Measured by |
|---|---|---|
| KV cache at 256K context | 72 MiB against 8 GiB for Qwen3.5-4B, a 99.1% cut | company |
| Throughput on one 80 GB Hopper GPU | 12,124 tokens/s at 512 streams 6.42 times Qwen3.5-4B's 1,889 tokens/s at its batch limit of 64 | company |
| AIME score retained vs Qwen3.5-4B | 96.3% 97.5% retained on MATH-500 | company |
| Retrieval ablation | 94.1% with anchors, 0.4% without | company |
Released under the EdgeRock model license, which the team says allows commercial use. Larger sizes are planned.
Sources
This record was checked against its sources on 27 October 2026. How we check