New roofline paper: "Scalable Packed Layouts for Vector-Length-Agnostic ML Code Generation"

Modern CPUs accelerate AI inference using SIMD (Single Instruction, Multiple Data) instructions. Arm’s Scalable Vector Extension (SVE) is a modern SIMD extension that introduces a vector-length-agnostic programming model, removing the assumption of a fixed vector length at compile time. However, this flexibility introduces substantial compiler challenges in vectorization, tiling, and data layout strategies.

Expanding on the earlier joint Arm x roofline case study on SVE support, this paper lays the groundwork for data layout strategies for scalable vector ISAs and evaluates these on ARM SVE.

Three key findings:

1. We enabled data-tiled Arm SVE support in MLIR/IREE by making data layouts and tile sizes parametric in the hardware vector length, rather than fixed at compile time. This means a single compiled AI model runs efficiently across Arm CPUs with different vector lengths.

2. Our SVE code generation matches and often outperforms IREE's existing NEON code generation (up to 1.5× speedup), confirming that scalable data layouts do not regress performance despite the added compiler complexity. It also outperforms PyTorch ecosystem frameworks, with speedups of up to 1.7× over ExecuTorch and 6× over TorchInductor

3. In a cycle-level simulator study, our generated code scales with increasing SVE vector length, reaching up to 3.4× speedup when moving from 128-bit to 512-bit SVE on compute-bound workloads.

Great work by Ege Beysel, Maximilian Bartel, and Jan Moritz Joseph, and thanks to Andrzej Warzyński for the continuously great collaboration.

Read the full paper: https://lnkd.in/dTwt__ar

‍‍#MLIR #LLVM #AICompiler #EdgeAI #AIDeployment #Roofline

‍