Hi HN, Roman and Henry here from Cactus (https://github.com/cactus-compute/cactus).We just shipped the biggest upgrade to our on-device inference platform:- Built-in model confidence-based routing to hand off inference runs to the cloud - Converter for any PyTorch model - Lossless 4-bit quantization (evals on our GitHub README) - GPU acceleration on compatible devices (starting with Apple Metal) - Minimal RAM footprint - Runs on any Arm device: iOS, Android, Mac, DGX Spark, Raspberry Pi, and moreAll in, a Gemma 4 E2B class model runs at 169 tok/sec on M5 Max, takes 2.7GB disk space with no accuracy degradation from FP16, uses 1.3GB of RAM, and requests help from cloud models when needed.The problem we started with eighteen months ago: inference engines are built for datacenters, but consumer hardware has different physics: you share RAM with the OS, you get thermally throttled, and the same model behaves differently on different hardware.So we wrote a runtime from scratch for resource-...