GitHub radar
Moonshot AI’s 2.78T Parameter Model on Apple Silicon Kimi-K3 MLX Port
Moonshot AI’s Kimi-K3, a 2.78 Trillion parameter multimodal model with 1M token context length, has been ported to Apple’s MLX framework (Apple Silicon) by a community developer. Details on the new Kimi Delta Attention, Attention Residuals and other architectural components have also been explored.
PipeNetwork has ported Moonshot AI’s Kimi-K3 to Apple’s MLX framework (Apple Silicon). Kimi-K3 is a 2.78 trillion parameter multimodal MoE model (104 billion parameters activated per token) developed by Chinese AI firm Moonshot AI. It has a 1M token context length. This port includes a complete implementation of the text tower (Kimi Delta Attention, Attention Residuals), the 3D aware video vision tower MoonViT, and a multimodal wrapper, which were all numerically tested for parity with the original PyTorch implementation.
Why a vibe-coder should care
The Kimi-K3 model requires at least 870GB of RAM, which exceeds the 512GB limit of the world’s most powerful Mac computer, meaning this implementation is not runnable on any one device at the moment. However, it is a huge step towards complete transparency of the Kimi-K3 model architecture on Apple ML stack. Complete transparency means that API services and cloud deployment for K3 models will be available sooner.
▌ More finds