Memory Hierarchy: Why Fitting a Model Is Not the Same as Making It Capable
Small devices force visible tradeoffs that are easy to hide on larger computers.
Goal
Explain how a model can be made to fit within limited memory without assuming that fitting the model makes it intelligent or useful for every task.
Activity
- Review the ESP32-AI project as a case study.
- Draw three storage tiers: fast/small working memory, larger/slower external RAM, and nonvolatile flash/storage.
- Place frequently accessed state, model weights, temporary buffers, and persistent files where you think they belong.
- Explain what quantization changes and what it does not change.
- List three capability questions that parameter count alone cannot answer.
Deliverable
Submit the memory diagram and a paragraph answering: Why can an engineering achievement in model placement still produce a model with limited practical capability?
Extension
Compare the same problem with a desktop local-LLM system. Which constraints disappear, and which remain?
Source material
First spotted in PTIR: August 9, 2026, Evening Briefing.
ESP32-AI demonstrates a 28.9-million-parameter, 4-bit model running entirely on an ESP32-S3 by treating SRAM, PSRAM, and flash as distinct memory tiers. The project’s own documentation is especially useful because it separates the engineering accomplishment of fitting and running the model from claims about model usefulness or intelligence. That distinction became the core of this lab.