Dat Viet Thanh Nguyen

What I Learned Shipping Computer Vision Models to 10+ Edge Targets

Fri, 14 Aug 2026 00:00:00 +0000

Over four years at VinAI I deployed computer vision models onto NVIDIA Xavier, Qualcomm QCS610 and SA8255P, Ambarella CV25, KingFisher boards, and VinSmart phones — through TensorRT, SNPE, QNN, CVFlow, NCNN, MNN, and TFLite. Different silicon, different toolchains, same recurring lessons. These are the ones I would tell my earlier self.

The bottleneck is almost never the network

The instinct when a model is too slow on device is to make the model smaller. Often that is the wrong lever. Before touching the architecture, it is worth checking:

Are all your operations actually running on the accelerator? A single unsupported op in the middle of a graph can force a fallback to CPU, and the round trip costs more than the op saves. The profile will show a suspiciously expensive layer that looks trivial on paper.
Where is the pre- and post-processing running? Resize, colour conversion, letterboxing, and NMS are easy to leave in Python or in unoptimised CPU code. On a real pipeline these can rival inference time.
Is the pipeline saturated? A model that takes 15 ms but sits idle waiting for frames is not a 15 ms pipeline.

Measuring end to end, on the target, before optimising anything, saves a lot of wasted architecture work.

FLOPs are not latency

A model with fewer multiply-accumulates is not automatically faster on a given SoC. Depthwise separable convolutions are the classic example: excellent FLOP economy, but memory-bandwidth bound, and on some accelerators they run worse than a denser convolution with more nominal work.

This is why designing against the target beats shrinking after the fact. When we built a Neural Architecture Search pipeline to generate object detection architectures for CPU-constrained devices, the useful part was not NAS itself — it was that the search was scored against measured on-device latency rather than a proxy like FLOPs or parameter count. The architectures it found were ones we would not have chosen by intuition.

PTQ first, QAT when you have to

Going from FP32 to INT8 is usually the single largest win available, and post-training quantization gets you most of it for very little effort. My default order of operations:

Post-training quantization with a representative calibration set. “Representative” is doing real work in that sentence — a calibration set drawn from clean, well-lit training images will produce a model that falls apart on the hard inputs it will actually see.
Look at where the accuracy actually went. Quantization error is rarely spread evenly. It concentrates in a few layers — often the first and last, and anything with a wide dynamic activation range.
Selectively keep those layers at higher precision before reaching for anything more elaborate.
Quantization-aware training only when the above is not enough. It works, but it costs a training cycle every time the model changes, which changes how quickly you can iterate.

For face and detection models we compressed substantially with minimal accuracy loss this way. The failure mode to watch for is evaluating the quantized model only on your clean benchmark: INT8 degradation shows up disproportionately on the tail — small faces, low light, motion blur — which is exactly the part of the distribution the product is judged on.

Every runtime has opinions

Supporting many targets means accepting that “export the model” is not a single step. Each runtime has its own supported operator set, its own layout preferences, its own quantization semantics, and its own quiet behavioural differences. Things that helped:

Keep one source model and a per-target export path, not a fork per target. Forks drift, and then a bug fixed in one place lives on in four others.
Test numerical parity at every hop. Compare FP32 PyTorch against exported FP32 against quantized on-device output, on the same inputs. Catching a discrepancy at the export step is minutes of work; catching it after integration is days.
Constrain the architecture to the intersection of what your targets support. An exotic operator that only one runtime handles well will cost more in maintenance than it earns in accuracy.

Accuracy is not the product

A driver monitoring system that is correct 99% of the time but stutters is worse, to the person using it, than one that is slightly less accurate and smooth. Frame-to-frame stability, latency consistency, and graceful behaviour under thermal throttling are all part of whether the thing is any good — and none of them appear in a validation metric.

That gap between the benchmark and the product is, to me, the interesting part of this work.

SEMACS: Semantic–Acoustic Speech Coding for Ultra-Low Token Rates

Thu, 01 Jan 2026 00:00:00 +0000

This manuscript is currently under review. A link to the preprint, code, and full details will be added here once it is publicly available.

Personality-Driven AI Avatar Bot

Wed, 01 Jan 2025 00:00:00 +0000

Dec 2023 – Jan 2025

Problem

Group chat applications are text-first. Adding an AI participant that feels like a character rather than a chat completion means giving it a face, a voice, and a consistent personality — and generating all of that fast enough that it can take part in a live conversation rather than replying minutes later.

Approach

I built the avatar generation side of the product: turning a character definition into a speaking AI avatar that could be dropped into a group chat.

The pipeline combines several pieces I had been building in parallel:

Identity generation — a consistent visual identity per character, stable across every clip the bot produces.
Audio-driven talking head — driving lip-sync and head motion from a single audio stream while preserving the character’s identity, head pose, and facial expression.
Expression control — routing the character’s intended emotion (joy, sadness, anger) into the generated video, so the avatar’s delivery matches the tone of what it is saying.
Real-time diffusion video — a Stable Diffusion video generation pipeline producing avatar speech video directly from audio input.

The engineering constraint throughout was latency: a conversational avatar that lags is worse than no avatar, so each stage was optimised for real-time or near-real-time throughput rather than best-possible offline quality.

Tech stack

PyTorch · Stable Diffusion · TensorRT · Python · Docker

Outcome

Shipped as the avatar generation component of a group chat application.

Driver Attention Tracking and Analysis

Wed, 10 Apr 2024 00:00:00 +0000

Why this problem

Knowing what a driver is looking at — not just where their head is pointing — is the difference between a monitoring system that can warn about a missed traffic sign and one that cannot. The usual answer is eye-tracking glasses, which are expensive, intrusive, and impractical for mass deployment. They also only tell you what the driver did look at, never what they missed in the scene.

What we did

We built a dashboard-mounted system that needs only two ordinary cameras: one on the driver’s face, one on the road. The core is a convolutional network that reads both images at once and predicts the driver’s point-of-gaze directly in the scene frame.

The hard part is that the geometry between driver and cameras drifts constantly — even within a single drive. Rather than asking the driver to recalibrate, the network includes a camera calibration module that learns an embedding of the driver–camera spatial configuration and is trained end to end with the rest of the model. This module measurably improves gaze accuracy.

We also released a large-scale in-situ dataset: real urban driving sessions with synchronised face, scene, and gaze video, and over one hundred thousand image pairs annotated with points of gaze — addressing a gap no existing gaze dataset filled.

Result

Mean prediction error of 29.69 pixels against a 1280×720 scene camera, outperforming the baseline methods we compared against.

AI Photo Editing App

Mon, 01 Jan 2024 00:00:00 +0000

Sep 2023 – Jan 2024

Problem

The edits people actually want from a photo app — remove that person, drop the background, make the sky dramatic, put him in a suit — are each a different technical problem, but users expect all of them behind a single tap. Each feature needs an accurate mask, a plausible fill, and a result that survives being zoomed into.

Approach

I built four editing features on a shared backbone of segmentation plus generative inpainting:

Object removal — segment the unwanted region, then inpaint it with surroundings-consistent content rather than a blur or clone-stamp patch.
Background removal — high-quality alpha matting around hard cases: hair, glasses, motion blur at the subject boundary.
Sky replacement — sky segmentation combined with colour harmonisation, so the relit foreground matches the new sky instead of looking pasted on.
Suit changer — garment-region segmentation and generative synthesis that keeps the subject’s identity and pose intact.

The common thread is that mask quality dominates perceived output quality: a strong generative model behind a sloppy mask still looks wrong at the edges, so most of the tuning effort went into the segmentation stage and its boundary refinement.

Tech stack

PyTorch · Diffusion Models · Segmentation networks · OpenCV · Python

Outcome

Delivered as a set of consumer-facing editing features.

Experience

Tue, 24 Oct 2023 00:00:00 +0000

QC-StyleGAN: Quality Controllable Image Generation and Manipulation

Mon, 28 Nov 2022 00:00:00 +0000

Why this problem

StyleGAN and its successors are trained on curated, high-quality datasets. That is fine for synthesis, but it breaks the moment you want to edit a real photo: GAN inversion pushes a blurry, noisy, or JPEG-compressed input onto a high-quality manifold, so the model silently “restores” the image while you were only trying to change the hairstyle. You lose the ability to make an edit that preserves the original look.

What we did

We added an explicit quality control code to the generator, so image content and image degradation become separately controllable. The same network can:

synthesise images at any point along a continuous quality axis;
invert and edit a low-quality image without changing its quality;
restore a sharp image by simply moving the quality code — giving blind restoration across noise, blur, compression artifacts, and their mixtures, without training a restoration model;
synthesise, transfer, and interpolate degradations between images.

Published at NeurIPS 2022; the work also received a NeurIPS Scholar Award. Code is available in the official repository.

Real-Time Portrait Segmentation on Mobile

Wed, 01 Dec 2021 00:00:00 +0000

Aug 2021 – Dec 2021

Problem

Portrait effects — background blur, replacement, relighting — need a person mask computed every frame, on a phone, inside the camera’s frame budget, without draining the battery. A model that is accurate but too slow is useless here, and so is a fast model that flickers between frames or cuts through someone’s hair.

Approach

Two halves, and the data half mattered as much as the model:

The network. A deliberately small segmentation architecture designed against the mobile inference budget rather than trimmed down afterwards. Design decisions were driven by what the target runtime could actually execute efficiently, not just by parameter count — an operation that is cheap on paper is not cheap if the on-device runtime falls back to CPU for it.

The data pipeline. I built the full pipeline: collection, annotation, quality control, and augmentation. Boundary quality is where portrait segmentation visibly succeeds or fails, so the annotation standard around hair and edges drove most of the final result.

The model was quantized and deployed through the mobile inference stack used for VinSmart devices, alongside the other lightweight face analysis networks (FaceID, Masked FaceID, camera effects) our team shipped.

Tech stack

PyTorch · TFLite · SNPE · NCNN · INT8 quantization · Python · C++

Outcome

A portrait segmentation network running in real time on target mobile hardware.

Industrial Surface Defect Detection

Thu, 01 Apr 2021 00:00:00 +0000

Jan 2021 – Apr 2021

Problem

Industrial defect inspection is an unbalanced detection problem with an unforgiving cost function: defects are rare, visually subtle, and vary in severity, while a false negative ships a bad part and a false positive stops a line. The output the customer needs is not just defect / no defect but a grade — how severe, and therefore what to do about it.

Approach

I built the detection and grading models, and — the part that shaped the project most — the integration path.

The customer’s existing inspection software was a .NET application, not a Python stack. Rather than standing up a separate inference service and forcing a network hop into a line-rate process, the model was packaged as a .NET-consumable DLL so it could be called directly from the existing application. That removed a whole class of latency and deployment failure modes, at the cost of doing the native packaging and memory-management work properly.

On the modelling side, the emphasis was on the severity grading step: getting a reliable ordering of defect severity out of a detector matters more to the operator than squeezing the last point of detection mAP.

Tech stack

PyTorch · LibTorch · C++ · .NET DLL integration · OpenCV

Outcome

Delivered as an integrated component of the customer’s existing inspection application.

Autopilot Perception on Jetson TX2

Fri, 01 Jan 2021 00:00:00 +0000

Jan 2020 – Jan 2021

Problem

An autonomous vehicle has to see and react inside a control loop that does not wait. On a Jetson TX2, the whole perception stack — not just one model — has to fit in a few watts and a few tens of milliseconds, while staying reliable under changing light, weather, and road surfaces.

Approach

I worked on the perception side of the vehicle: the models that turn camera frames into the lane, sign, and obstacle information the control stack acts on.

The interesting constraint was that accuracy and latency traded against each other continuously. Time spent on a heavier detector is time the controller does not have to react, so the work was as much about inference engineering — optimising the graph, choosing precisions, keeping the pipeline saturated — as about model architecture. The stack was built against TensorRT to get the most out of the TX2’s GPU.

This was developed for the FPT Digital Race, where the system had to run a real course rather than a held-out test set — which surfaces a different class of failure than offline evaluation does.

Tech stack

PyTorch · TensorRT · NVIDIA Jetson TX2 · OpenCV · Python · C++

Outcome

Top 10 — FPT Digital Race 2020 Final.