Aria Bolour
Writing

TensorFlow to on-device: converting a model for iOS, and what I'd use today

· 12 min read · On-device MLCore MLiOS

On Hamrah Card, card-number capture runs entirely on-device: point the camera at a card, and a model reads the number off it without a single frame leaving the phone. That constraint wasn't a nice-to-have. On a payments app, sending camera frames of a payment card to a server just to read a number off it is the kind of thing you'd have to justify to a security review, and the honest justification is usually "we didn't want to do the conversion work." The model started life in TensorFlow. Getting it onto an iPhone, running fast enough to feel instant, is a different job than training it, and most of the real problems live in that gap, not in the model itself.

Why convert at all

Three reasons keep showing up, and only one of them is performance. Privacy: card data that never leaves the device doesn't need a data-retention policy, a breach-notification plan, or a line in the privacy policy about what a third party does with it. Latency: a network round trip for every camera frame turns a live viewfinder experience into a spinner. Availability: the scanner has to work on a train, in a basement, wherever a person happens to be holding their card up to a phone, not only where there's a signal.

None of those are TensorFlow's problem to solve. TensorFlow is a training-time tool; nothing about a SavedModel or a Keras .h5 file assumes it needs to run inside an app bundle under a battery budget. The conversion step is where those constraints actually get imposed.

The path: TensorFlow, through ONNX, onto the device

The route that gets used in practice, and the one behind Hamrah Card's scanner, goes through ONNX as an intermediate format rather than jumping straight from TensorFlow to Core ML:

# Export the trained model, then convert the graph
python -m tf2onnx.convert \
  --saved-model card_detector_savedmodel \
  --output card_detector.onnx \
  --opset 13

ONNX (Open Neural Network Exchange) is a format designed to sit between frameworks, not to be a runtime unto itself. Once the graph is in ONNX form, there are two real paths onto an iPhone, and the choice between them is an architecture decision, not a technical detail:

Path one: ONNX Runtime, directly

ONNX Runtime ships an iOS build (as a CocoaPod or Swift package) that executes the .onnx file as-is, no further conversion needed:

import onnxruntime

let session = try ORTSession(
    env: ortEnv,
    modelPath: modelPath,
    sessionOptions: options
)
let outputs = try session.run(
    withInputs: ["input": inputTensor],
    outputNames: ["output"],
    runOptions: nil
)

The appeal is parity: the same .onnx file, or something very close to it, can run on Android through the same runtime, so a cross-platform team ships one model artifact instead of two. The cost is that ONNX Runtime doesn't automatically get the same depth of Neural Engine scheduling that a native Core ML model does; it runs well, but "well" and "as fast as the silicon allows" aren't always the same number.

Path two: on into Core ML

The alternative is one more conversion, from ONNX into Core ML's .mlpackage format, using coremltools:

import coremltools as ct

mlmodel = ct.convert(
    "card_detector.onnx",
    convert_to="mlprogram",
    compute_units=ct.ComputeUnit.ALL,   # CPU + GPU + Neural Engine
    minimum_deployment_target=ct.target.iOS16
)
mlmodel.save("CardDetector.mlpackage")

This gives up cross-platform parity in exchange for the thing Core ML is actually good at: the OS decides, per layer, whether to run on the CPU, the GPU, or the Neural Engine, and that decision is tuned by Apple, not by you. For a model running continuously against a live camera feed, that scheduling is usually worth the platform lock-in.

Where it actually breaks

The conversion command is the easy part. The failures that cost real time are quieter:

  • Preprocessing drift. If training normalized pixels to [-1, 1] and the on-device code normalizes to [0, 1], the model still runs, produces output, and that output is quietly wrong. Nothing crashes. This is the single most common on-device ML bug, and it never shows up in a unit test that only checks the model loads.
  • Unsupported ops. A layer that's ordinary in TensorFlow, a custom activation, a particular resize mode, sometimes has no direct Core ML or ONNX Runtime equivalent, and the converter either substitutes something close enough or refuses. "Close enough" needs to be verified against real output, not assumed.
  • Simulator lies. The Neural Engine doesn't exist in the iOS Simulator; Core ML falls back to the CPU there, silently. A model that "works" in Simulator has only been tested on the one code path it won't primarily use in production. Real-device testing isn't optional for this kind of feature; it's the only place the actual dispatch behavior exists.
  • Size and latency, after conversion, not before. A model that trains fine at full float32 precision is often gratuitously large for a phone. Quantizing to float16, or further to int8 with calibration data, routinely cuts size and latency substantially, with a small, measurable accuracy cost that has to be checked against real inputs, not assumed to be free.

What's changed since

The TensorFlow-through-ONNX path above is still a completely reasonable way to get an existing, already-trained model onto a phone, and it's the honest answer for a team with a TensorFlow training pipeline they're not about to rewrite. But it's no longer the only on-device story worth knowing, and starting a new project today would mean weighing it against a few things that didn't exist, or weren't mature, when that conversion pipeline first went in:

  • Core ML itself has kept moving. Stateful models, on-device adapters for fine-tuning a base model without shipping a new one, and steadily wider op coverage in the converter mean fewer round trips through "the converter doesn't support this layer, rewrite it."
  • Apple's on-device foundation models. For anything language-shaped, summarizing text, extracting structured fields, a general-purpose on-device model exposed through a system framework is now a real option before reaching for a fully custom, trained-from-scratch network. It won't replace a specialized vision model trained for one exact task, like reading a card number off a photo, but it changes the calculus for a wide class of features that used to justify their own model.
  • MLX, for anything trained or fine-tuned on Apple silicon itself. Apple's array framework for on-device and on-Mac machine learning makes it realistic to iterate on a model, including small language models, without ever leaving the Apple toolchain, which is a different workflow than train-in-TensorFlow-then-convert.

The decision I'd make today for the same problem, on-device card-number detection, is still probably a small, purpose-trained vision model, because that's a narrow, well-defined task a general foundation model isn't the efficient tool for. But I'd reach for Core ML directly rather than TensorFlow-through-ONNX if I were starting from nothing, and I'd only keep the ONNX path if the model already existed and rewriting the training pipeline wasn't the actual problem worth solving.

The part that doesn't change

Whichever path gets you there, the reason to do any of this is the same: a payments app shouldn't need a server round trip to read a card number off a camera frame, and a model that never leaves the device is a security property, not just a performance one. The conversion tooling will keep changing. That reason won't.