How Leaders in Tech Cut On-Device AI KV-Cache Memory by 37.5%

Apple introduced a compact, approximately 3-billion-parameter on-device model and a mixture-of-experts server model with a novel architecture tailored for Private Cloud Compute.

Apple’s on-device model shares KV caches between its two blocks, which reduces KV-cache memory usage by 37.5% and significantly improves time-to-first-token. Decoder weights are compressed to 2 bits per weight for the on-device model and 3.56 for the server model. After compression and adapter recovery, the on-device model shows about 4.6% regression on MGSM and 1.5% improvement on MMLU. The server model outperforms Qwen-2.5-VL in human evaluation of image responses at less than half the inference FLOPS.

37.5%

reduction in KV-cache memory usage for the on-device model, which also significantly improves time-to-first-token

87.5%

of synchronization overhead removed in Apple’s example with track block depth D = 4 (2L with tensor parallelism to L/D)

2-bpw

decoder weights of the on-device model via quantization-aware training (server model: 3.56-bpw via ASTC)

< half

the inference FLOPS of Qwen-2.5-VL for the server model, which outperforms it in human evaluation of image responses

Figures come from the source below. Anything from another place is attributed in the text.