Massive move by Apple for local inference. They just launched M6 and M5 Ultra to move more AI inference onto Mac desktops.

The M5 Ultra Mac Studio is configurable upto 256GB of unified memory and 512 GB coming in Oct.

M6 debuts in Mac mini as Apple's first 2nm chip, with a 12-core CPU, 12-core GPU and Dual 16-core Neural Engine.

The smaller process increases transistor density, while GPU Neural Accelerators and 170GB/s memory bandwidth give local models more compute and faster access to data.

M5 Ultra takes the approach further by joining two dual-die M5 Max chips through UltraFusion into Apple's first quad-die SoC.

It scales to 36 CPU cores, 80 GPU cores, 512GB unified memory and 1.2TB/s bandwidth, 50% above M3 Ultra.

The memory capacity changes AI use because Apple says it can hold LLMs with hundreds of billions of parameters entirely on-device.

Apple also supports Thunderbolt 5 clustering with RDMA, so it will let multiple Mac Studio systems pool memory for distributed inference.