Real-Time Portrait Segmentation on Mobile

Wed, 01 Dec 2021 00:00:00 +0000

Aug 2021 – Dec 2021

Problem

Portrait effects — background blur, replacement, relighting — need a person mask computed every frame, on a phone, inside the camera’s frame budget, without draining the battery. A model that is accurate but too slow is useless here, and so is a fast model that flickers between frames or cuts through someone’s hair.

Approach

Two halves, and the data half mattered as much as the model:

The network. A deliberately small segmentation architecture designed against the mobile inference budget rather than trimmed down afterwards. Design decisions were driven by what the target runtime could actually execute efficiently, not just by parameter count — an operation that is cheap on paper is not cheap if the on-device runtime falls back to CPU for it.

The data pipeline. I built the full pipeline: collection, annotation, quality control, and augmentation. Boundary quality is where portrait segmentation visibly succeeds or fails, so the annotation standard around hair and edges drove most of the final result.

The model was quantized and deployed through the mobile inference stack used for VinSmart devices, alongside the other lightweight face analysis networks (FaceID, Masked FaceID, camera effects) our team shipped.

Tech stack

PyTorch · TFLite · SNPE · NCNN · INT8 quantization · Python · C++

Outcome

A portrait segmentation network running in real time on target mobile hardware.

Mobile | Dat Viet Thanh Nguyen

Real-Time Portrait Segmentation on Mobile

Problem

Approach

Tech stack

Outcome