GitHub radar
Run 80B LLMs on Your Mac or iPhone
A Swift & Metal runtime for Qwen3-Next/Qwen3.5/3.6 Mixture-of-Experts family models that loads a 35B or 80B Qwen MoE model from disk as needed, using just 2–3 GB of memory for offline inference on an M-series Mac or iPhone 17.
Swiftlet is an open source Swift & Metal implementation of inference runtime for Qwen3-Next/Qwen3.5/3.6 Mixture-of-Experts family models released by Alibaba’s Qwen team under Apache 2.0 license. With the ability to load only small part of dense weight into memory while streaming sparse experts from disk, we are able to run a 35B model with only 2.6GB memory on M5 Mac at 7–11 tokens/sec. And on iPhone 17, we can run the same model with ~2.5GB memory at about 1 token/sec. Swiftlet became popular because of its capability to stream large MoE weight from disk rather than loading them all into memory.
Why a vibe-coder should care
If you are interested in local deployment of AI apps without paying any API fee nor worrying about data security issues, now you could deploy powerful large language models to your existing Mac devices without any subscription and offline requirement. Meanwhile, this repo provides a good starting point for building privacy preserving AI inference system on Apple platform.
▌ More finds