For most of the last few years, adding intelligence to a mobile app meant one architecture. The phone captures something, sends it to a server, waits, and displays what comes back. Every inference is a network round trip and a line on an invoice.
That architecture still has its place. But a growing share of what apps ask models to do can now run on the phone itself, and the shift changes more than latency. It changes the unit economics, the privacy posture, and the set of features that are viable at all.
The cost curve is the headline
Server-side inference has a property that founders discover late: it scales with usage rather than with revenue.
Every user action that triggers a model call costs money. A feature that delights people is a feature that costs more the more it is used. Teams end up rationing, adding quotas, or quietly degrading the experience for free users, all of which are product decisions made by an infrastructure bill.
On-device inference inverts that. The compute belongs to the user, the marginal cost of an additional call is zero, and a feature can be used a hundred times a day without changing anything on your side. For anything high frequency, that difference is not a rounding error, it is the difference between a viable feature and one that gets cut.
Latency changes what the feature can be
A network round trip is measured in hundreds of milliseconds on a good connection, and in seconds on a bad one. That is fine for a request the user consciously makes and waits for. It is fatal for anything continuous.
Live camera analysis, suggestions that update as someone types, real-time audio processing, anything that has to keep pace with a person rather than respond to them: those only work when the model runs locally. Apple’s silicon has carried dedicated neural hardware for years precisely because of this, and Core ML exists to put it to work.
The practical consequence is that on-device is not simply a cheaper version of the same feature. It enables a different category of feature.
Privacy stops being a promise and becomes an architecture
Every regulated industry asks the same question of an AI feature: where does the data go. If the answer is a third party inference endpoint, the conversation becomes a data processing agreement, a residency review, and a set of retention commitments.
If the answer is that the audio, image or text never leaves the device, most of that conversation disappears. In healthcare, finance, legal and public sector work, that is often the deciding factor rather than a nice property. It also makes offline operation possible, which matters more than people assume once the app leaves the office.
Apple has leaned into this hard, both in its platform messaging and in the frameworks it gives developers. For teams building in regulated Canadian sectors, it is frequently the argument that gets a feature approved.
What the phone cannot do
None of this makes the server obsolete, and pretending otherwise leads to bad architecture.
Local models are smaller. They handle classification, extraction, transcription, summarisation of modest inputs, embeddings and semantic search well. They do not handle very large context, deep multi-step reasoning, or tasks that need the most capable frontier models. Some run only on recent hardware, which means older devices need a fallback path.
App size and memory are real constraints too. A bundled model adds weight to the download, and memory pressure on a mid-range device is unforgiving in a way that a server never is.
The pattern that works is hybrid
The architecture most teams converge on routes by task rather than by principle.
Run locally what is frequent, latency-sensitive, privacy-sensitive or needed offline. Send to the server what is rare, heavy, or genuinely requires a larger model. Design the boundary explicitly, and build a graceful degradation path so that older hardware and network failures produce a reduced experience rather than an error.
Getting that split right is now one of the more consequential decisions in a mobile project, because it determines the cost structure of the product for years. It is also the kind of decision that is expensive to revisit once the data flows are built, which is why experienced iOS app developers tend to raise it during architecture rather than treating it as an optimisation to do later.
Measure before you commit
The way to settle it is not a debate, it is a prototype. Run the candidate model on the oldest device you intend to support and measure three things: how long an inference takes, how much memory it holds, and what it does to battery over a realistic session.
Those numbers decide the design. They are cheap to obtain early and painful to discover late, and they vary far more across a device range than most estimates assume.
The strategic point
The interesting shift is not that phones can run models. It is that the boundary between what belongs on the device and what belongs on a server has moved, and it is still moving with each hardware generation.
Products designed around the assumption that intelligence lives in the cloud are being outcompeted on responsiveness, cost and privacy by products that assume the opposite where it counts. That is a strategy question, not an engineering detail, and it deserves attention before the architecture is set rather than after the infrastructure bill arrives.
