I'm making LLMs work on small and old devices
smolmaxxing is a llama.cpp patch set that aims to run the largest possible language model on low-end hardware such as TV boxes, old tablets and single-board computers. Using 1-bit/2-bit/ternary kernels for Cortex-A55 and A7, trimmed vocabulary output rows, and MoE experts paged from flash on demand, it runs 8B and 16B MoE models in about 1 GB of RAM, with all numbers measured on-device.
What this means for indie developers：If you plan on-device inference in an app or on old Android hardware, its kernel and flash-streaming techniques are directly reusable, though you must build the patched llama.cpp yourself.