Edge Inference

Edge inference is the execution of LLM and multimodal model inference entirely on a user-owned device, without round-trips to a hyperscaler datacenter.

Why It Matters

Edge inference changes the privacy, latency, and unit-economics story for any AI feature that today requires a server — and reopens hardware cycles for Apple, Qualcomm, and AMD.

Timeline

1. 2024-06

   Apple Intelligence announced — on-device 3B model.

2. 2024-Q3

   Copilot+ PCs launch with 40+ TOPS NPU baseline.

• 12 monthshigh confidence

  A 7B-class model runs at >30 tokens/sec on a mainstream consumer phone.

Related Trends

• Inference Cost Collapse

  Mainstream

  Token prices falling 80–95% per year while quality keeps rising.

• Open-Weights Frontier

  Rising

  Open-weight models closing the gap with closed frontier labs to under 6 months.