roofline brings Mixture-of-Experts (MoE) to the edge!

MoE powers many of today's strongest LLMs. By activating only the experts each token actually needs, it packs far more performance into the same compute budget, a limited resource on the edge. But this only pays off if your deployment stack can handle the dynamic expert routing MoE depends on, and until now, edge AI compilers couldn't.

‍roofline closes this gap by bringing native MoE support to its IREE-based deployment stack.

In our latest case study, we compile and run MoE models end-to-end, from IBM's Granite 3.1 1B-A400M, through Liquid AI's LFM2-8B-A1B, up to Qwen-REAP-15B-A3B. The case study walks through the operators that were missing, how we enabled them, and includes a demo of the compiled models running end-to-end: https://lnkd.in/eqqtneKV

‍‍#EdgeAI#AIDeployment#AICompiler#MLIR#MoE#Roofline#CoFundedByTheEU

‍