On-Device AI
Also known as: edge AI, local AI, device AI, on-premise AI model
On-device AI refers to machine learning models that run locally on a device (a smartphone, laptop, smart speaker, or embedded sensor) rather than sending data to a cloud server for processing.
This approach offers significant advantages: lower latency (responses are immediate, not dependent on network round-trips), greater privacy (personal data never leaves the device), offline functionality (works without an internet connection), and lower ongoing infrastructure cost at scale. The trade-off is that models must be small enough to run within the device's memory and compute constraints.
On-Device AI, explained at three levels
At home
On-device AI means the AI runs on your phone or computer, not on a server in a data centre. When you say 'Hey Siri', your phone processes that command locally without sending audio to Apple's servers. When your phone suggests the next word as you type a message, that prediction happens on the phone itself. The benefit is speed (no waiting for a network round-trip) and privacy (your audio or typing never leaves your device).
At work
On-device AI is increasingly important for business applications where data sensitivity, latency, or connectivity requirements make cloud-based AI unsuitable. Medical devices that analyse patient data cannot always rely on a cloud connection. Manufacturing equipment on factory floors may have poor or intermittent connectivity. Edge AI (AI processing at the point of data collection) reduces bandwidth requirements, response latency, and cloud costs for high-volume sensor data.
At home
Running AI models on devices requires models to be much smaller than their cloud counterparts. A frontier cloud LLM is widely believed to have hundreds of billions of parameters or more and requires server-grade GPU infrastructure. On-device language models typically have 1-7 billion parameters and use model compression techniques (quantization, pruning, knowledge distillation) to run within the memory and compute budget of a consumer device. The quality gap between on-device and cloud models is narrowing but still significant for complex reasoning tasks.
At work
Dedicated AI hardware has been built into consumer devices to enable efficient on-device processing. Apple's Neural Engine (in iPhone chips since 2017), Qualcomm's Hexagon NPU (in Android flagship chips), and Intel's NPU (in AI PCs from 2023) are silicon specifically designed to run neural network operations more efficiently and at lower power than general-purpose CPUs or GPUs. These NPUs enable on-device tasks like real-time photo processing, voice recognition, and language model inference without draining the battery.
At home
Privacy is a key driver for on-device AI in consumer products. Apple Intelligence (announced 2024) processes many AI tasks on-device and uses a 'Private Cloud Compute' architecture for tasks requiring more compute, designed so that Apple says it cannot access the data processed on its servers, with server software that outside researchers can inspect. This allows users to benefit from AI features without trusting a cloud service with their personal data. Regulatory requirements in healthcare, legal, and financial sectors may effectively mandate on-device processing for sensitive data.
At work
From a practitioner standpoint, the choice between on-device and cloud AI depends on four factors: latency requirement (does the application need sub-100ms response times?), privacy requirement (is the processed data too sensitive to send to a cloud service?), connectivity requirement (must the application work offline?), and task complexity (is the task within the capability of a small model, or does it require the reasoning power of a large cloud model?). Many production applications use a hybrid approach: simple tasks run on-device, complex tasks escalate to cloud, with the escalation decision made per-request based on context.
Real-world examples
At home
Jerome's smartphone camera automatically recognises faces in his photos and groups them by person: all done locally on the phone without any photos being sent to a server. This on-device AI processing is what lets the feature work even when he has no mobile signal.
At work
A hospital equipment manufacturer embedded on-device AI in a portable ECG monitor used in ambulances. The device uses a model trained to detect arrhythmias from ECG data and alert paramedics in real time: with sub-second latency and no dependency on ambulance network connectivity. Patient data never leaves the device.
How it compares
| Feature | On-Device AI | Cloud AI |
|---|---|---|
| Where processing happens | On the local device (phone, laptop, sensor) | On remote servers in a data centre |
| Latency | Low: no network round-trip | Higher: dependent on network and server queue |
| Privacy | High: data stays on device | Lower: data sent to cloud service provider |
| Offline functionality | Yes: works without internet | No: requires internet connection |
| Model capability | Limited: small models (1-7B parameters) | High: large models (hundreds of billions of parameters) |
| Cost at scale | Lower: no per-call API fee | Higher: ongoing API or infrastructure cost |
On-device AI is better for low-latency, offline, and privacy-sensitive applications. Cloud AI is better for complex reasoning tasks and applications where model capability matters more than latency or privacy. Many applications use both.
For analysts
In BA artefacts, on-device AI requirements surface when designing mobile applications, embedded systems, or any AI feature with privacy, latency, or connectivity constraints. Specify clearly: which tasks must run on-device (for privacy or connectivity reasons), which tasks may use cloud AI (where complexity requires it), and how the application behaves when the device is offline. When on-device AI is required for regulatory reasons (data must not leave the device), this is a hard architectural constraint that should appear as a System-Level Constraint in requirements documentation, not an implementation preference.
History & origin
On-device AI has existed since the late 2010s in the form of device-local ML models for specific tasks: face unlock on smartphones (2017), neural processing unit (NPU) chips in mobile processors from 2017 (Apple A11 Bionic, Huawei Kirin 970), and voice keyword detection ('Hey Siri', 'OK Google') which has always run locally for privacy and latency reasons. The emergence of efficient small language models (SLMs) from 2023 (including Microsoft Phi series, Apple Intelligence models, and Meta Llama) extended on-device AI to general language tasks, enabling LLM-powered features on phones and laptops without cloud API calls.