Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU

Key Points:

• FreeToken, developed by UC Berkeley and UT Austin researchers, enables efficient local serving of large open-weight AI models by treating personal machines as unified inference platforms, allowing models like 35B to run interactively on an 8 GB laptop GPU and 753B GLM-5.2 on a single workstation card.
• The system overcomes key limitations of existing engines by adaptively splitting model computation between GPU and CPU based on measured bandwidth, maintaining bit-exact expert outputs without router modifications or precision loss, and employing semantic-aware caching and elastic memory management.
• FreeToken achieves 1.5–2.3× faster decode throughput than leading baselines such as llama.cpp, Ollama, and KTransformers, with tail token-to-first-token latency under 44 seconds, supporting practical deployment on consumer hardware including laptops and desktops.
• It is open-source under Apache-2.0 license, available on PyPI and as a one-click desktop app for Windows and Linux, targeting solo developers, startups, and SMBs, particularly in regulated or data-sensitive industries like healthcare, finance, and defense.
• Typical applications include local coding agents, private code review, offline contract analysis, and synthetic data generation, enabling affordable and private AI inference without reliance on datacenter-class GPU clusters.