Story MoE
A mixture-of-experts language model I built from scratch.
I wrote and trained a decoder-only transformer in PyTorch with sparse top-2 expert routing, rotary position embeddings, and KV-cached text generation. I also wrote tests to check causal attention, routing gradients, and that cached and uncached generation match.
*Measured on TinyStories. A dense model trained on the same number of tokens scored 10.77.
Two experts selected for each token

