Roshan Bhandari
Team Lead · Software Engineer · DevOps
Available for Projects
Run a 28.9M parameter LLM locally on an ESP32-S3 microcontroller with no internet connection. Uses Per-Layer Embeddings to fit large models in tiny flash memory.
Imagine running a language model with nearly 30 million parameters on a $8 microcontroller. No internet connection, no cloud servers, no expensive GPUs. Just pure AI inference happening locally on hardware that fits in your palm. This isn't science fiction—it's now reality thanks to a clever approach that reimagines how we store and access neural network weights.
This project demonstrates how to run a 28.9 million parameter language model directly on an ESP32-S3 microcontroller. Unlike traditional AI deployments that require powerful servers or cloud connectivity, everything happens locally on the chip itself. The model generates text at approximately 9.5 tokens per second and writes output directly to a small attached screen.
What makes this particularly remarkable is the memory constraint. The ESP32-S3 has only 512KB of SRAM (fast memory) and 8MB of PSRAM (medium-speed memory), yet the model successfully operates by storing most parameters in the 16MB flash storage. This approach breaks the typical limitation where models were restricted to just 260 thousand parameters on similar hardware.
The model was trained on TinyStories, meaning it excels at generating short, simple narratives while maintaining coherence. Think of it as a creative writing companion that lives entirely on your microcontroller, always ready to spin up a tale without ever sending data over the network.
| Specification | Details |
|---|---|
| Parameters | 28.9M stored (25M in flash lookup table) |
| Chip | ESP32-S3 (~$8, 512KB SRAM, 8MB PSRAM, 16MB flash) |
| Speed | ~9.5 tokens/second end-to-end |
| Connectivity | None required—fully offline operation |
| Model Size | 14.9MB at 4-bit quantization |
Developers can leverage this for several compelling applications:
This project has captured significant attention because it pushes the boundaries of what's possible with edge computing. The previous state-of-the-art for microcontrollers like the ESP32 topped out at around 260,000 parameters—a hundred times smaller than this breakthrough.
The secret sauce lies in Per-Layer Embeddings, a technique pioneered by Google's Gemma models. Instead of loading the entire embedding table (which contains most model parameters) into fast memory, the system stores it in flash and retrieves only the necessary rows for each token generation. This requires reading just ~450 bytes per token from flash while keeping the active computation core in SRAM.
The memory architecture works like this: the small "thinking" core