Quantization

Unsloth Dynamic GGUF 3.0 is the latest iteration of Unsloth’s quantization format for running large models locally. It delivers more than 10% better top-1% accuracy at the same file size as every other GGUF provider, works with llama.cpp and Unsloth Desktop, and makes a 27B-parameter model runnable on as little as 7-8GB of RAM. This review explains what changed, how the quality claims are measured, and exactly which quant to pick for your hardware. ...

Introduction — Why Run a Language Model on an $8 Microcontroller? The idea of running a language model on a microcontroller that costs less than a cup of coffee sounds improbable, but the ESP32-S3 makes it a reality. With dual-core Xtensa LX7 processors running at up to 240 MHz, 512 KB of internal SRAM, and support for up to 8 MB of external octal SPI PSRAM, this $6–8 chip can execute small language models (SLMs) entirely offline — no cloud, no WiFi dependency, no API keys required. ...