What I Learned Shipping Computer Vision Models to 10+ Edge Targets

Over four years at VinAI I deployed computer vision models onto NVIDIA Xavier, Qualcomm QCS610 and SA8255P, Ambarella CV25, KingFisher boards, and VinSmart phones — through TensorRT, SNPE, QNN, CVFlow, NCNN, MNN, and TFLite. Different silicon, different toolchains, same recurring lessons. These are the ones I would tell my earlier self.

The bottleneck is almost never the network

The instinct when a model is too slow on device is to make the model smaller. Often that is the wrong lever. Before touching the architecture, it is worth checking:

Are all your operations actually running on the accelerator? A single unsupported op in the middle of a graph can force a fallback to CPU, and the round trip costs more than the op saves. The profile will show a suspiciously expensive layer that looks trivial on paper.
Where is the pre- and post-processing running? Resize, colour conversion, letterboxing, and NMS are easy to leave in Python or in unoptimised CPU code. On a real pipeline these can rival inference time.
Is the pipeline saturated? A model that takes 15 ms but sits idle waiting for frames is not a 15 ms pipeline.

Measuring end to end, on the target, before optimising anything, saves a lot of wasted architecture work.

FLOPs are not latency

A model with fewer multiply-accumulates is not automatically faster on a given SoC. Depthwise separable convolutions are the classic example: excellent FLOP economy, but memory-bandwidth bound, and on some accelerators they run worse than a denser convolution with more nominal work.

This is why designing against the target beats shrinking after the fact. When we built a Neural Architecture Search pipeline to generate object detection architectures for CPU-constrained devices, the useful part was not NAS itself — it was that the search was scored against measured on-device latency rather than a proxy like FLOPs or parameter count. The architectures it found were ones we would not have chosen by intuition.

PTQ first, QAT when you have to

Going from FP32 to INT8 is usually the single largest win available, and post-training quantization gets you most of it for very little effort. My default order of operations:

Post-training quantization with a representative calibration set. “Representative” is doing real work in that sentence — a calibration set drawn from clean, well-lit training images will produce a model that falls apart on the hard inputs it will actually see.
Look at where the accuracy actually went. Quantization error is rarely spread evenly. It concentrates in a few layers — often the first and last, and anything with a wide dynamic activation range.
Selectively keep those layers at higher precision before reaching for anything more elaborate.
Quantization-aware training only when the above is not enough. It works, but it costs a training cycle every time the model changes, which changes how quickly you can iterate.

For face and detection models we compressed substantially with minimal accuracy loss this way. The failure mode to watch for is evaluating the quantized model only on your clean benchmark: INT8 degradation shows up disproportionately on the tail — small faces, low light, motion blur — which is exactly the part of the distribution the product is judged on.

Every runtime has opinions

Supporting many targets means accepting that “export the model” is not a single step. Each runtime has its own supported operator set, its own layout preferences, its own quantization semantics, and its own quiet behavioural differences. Things that helped:

Keep one source model and a per-target export path, not a fork per target. Forks drift, and then a bug fixed in one place lives on in four others.
Test numerical parity at every hop. Compare FP32 PyTorch against exported FP32 against quantized on-device output, on the same inputs. Catching a discrepancy at the export step is minutes of work; catching it after integration is days.
Constrain the architecture to the intersection of what your targets support. An exotic operator that only one runtime handles well will cost more in maintenance than it earns in accuracy.

Accuracy is not the product

A driver monitoring system that is correct 99% of the time but stutters is worse, to the person using it, than one that is slightly less accurate and smooth. Frame-to-frame stability, latency consistency, and graceful behaviour under thermal throttling are all part of whether the thing is any good — and none of them appear in a validation metric.

That gap between the benchmark and the product is, to me, the interesting part of this work.