Projects | Dat Viet Thanh Nguyen

Personality-Driven AI Avatar Bot

Wed, 01 Jan 2025 00:00:00 +0000

Dec 2023 – Jan 2025

Problem

Group chat applications are text-first. Adding an AI participant that feels like a character rather than a chat completion means giving it a face, a voice, and a consistent personality — and generating all of that fast enough that it can take part in a live conversation rather than replying minutes later.

Approach

I built the avatar generation side of the product: turning a character definition into a speaking AI avatar that could be dropped into a group chat.

The pipeline combines several pieces I had been building in parallel:

Identity generation — a consistent visual identity per character, stable across every clip the bot produces.
Audio-driven talking head — driving lip-sync and head motion from a single audio stream while preserving the character’s identity, head pose, and facial expression.
Expression control — routing the character’s intended emotion (joy, sadness, anger) into the generated video, so the avatar’s delivery matches the tone of what it is saying.
Real-time diffusion video — a Stable Diffusion video generation pipeline producing avatar speech video directly from audio input.

The engineering constraint throughout was latency: a conversational avatar that lags is worse than no avatar, so each stage was optimised for real-time or near-real-time throughput rather than best-possible offline quality.

Tech stack

PyTorch · Stable Diffusion · TensorRT · Python · Docker

Outcome

Shipped as the avatar generation component of a group chat application.

AI Photo Editing App

Mon, 01 Jan 2024 00:00:00 +0000

Sep 2023 – Jan 2024

Problem

The edits people actually want from a photo app — remove that person, drop the background, make the sky dramatic, put him in a suit — are each a different technical problem, but users expect all of them behind a single tap. Each feature needs an accurate mask, a plausible fill, and a result that survives being zoomed into.

Approach

I built four editing features on a shared backbone of segmentation plus generative inpainting:

Object removal — segment the unwanted region, then inpaint it with surroundings-consistent content rather than a blur or clone-stamp patch.
Background removal — high-quality alpha matting around hard cases: hair, glasses, motion blur at the subject boundary.
Sky replacement — sky segmentation combined with colour harmonisation, so the relit foreground matches the new sky instead of looking pasted on.
Suit changer — garment-region segmentation and generative synthesis that keeps the subject’s identity and pose intact.

The common thread is that mask quality dominates perceived output quality: a strong generative model behind a sloppy mask still looks wrong at the edges, so most of the tuning effort went into the segmentation stage and its boundary refinement.

Tech stack

PyTorch · Diffusion Models · Segmentation networks · OpenCV · Python

Outcome

Delivered as a set of consumer-facing editing features.

Real-Time Portrait Segmentation on Mobile

Wed, 01 Dec 2021 00:00:00 +0000

Aug 2021 – Dec 2021

Problem

Portrait effects — background blur, replacement, relighting — need a person mask computed every frame, on a phone, inside the camera’s frame budget, without draining the battery. A model that is accurate but too slow is useless here, and so is a fast model that flickers between frames or cuts through someone’s hair.

Approach

Two halves, and the data half mattered as much as the model:

The network. A deliberately small segmentation architecture designed against the mobile inference budget rather than trimmed down afterwards. Design decisions were driven by what the target runtime could actually execute efficiently, not just by parameter count — an operation that is cheap on paper is not cheap if the on-device runtime falls back to CPU for it.

The data pipeline. I built the full pipeline: collection, annotation, quality control, and augmentation. Boundary quality is where portrait segmentation visibly succeeds or fails, so the annotation standard around hair and edges drove most of the final result.

The model was quantized and deployed through the mobile inference stack used for VinSmart devices, alongside the other lightweight face analysis networks (FaceID, Masked FaceID, camera effects) our team shipped.

Tech stack

PyTorch · TFLite · SNPE · NCNN · INT8 quantization · Python · C++

Outcome

A portrait segmentation network running in real time on target mobile hardware.

Industrial Surface Defect Detection

Thu, 01 Apr 2021 00:00:00 +0000

Jan 2021 – Apr 2021

Problem

Industrial defect inspection is an unbalanced detection problem with an unforgiving cost function: defects are rare, visually subtle, and vary in severity, while a false negative ships a bad part and a false positive stops a line. The output the customer needs is not just defect / no defect but a grade — how severe, and therefore what to do about it.

Approach

I built the detection and grading models, and — the part that shaped the project most — the integration path.

The customer’s existing inspection software was a .NET application, not a Python stack. Rather than standing up a separate inference service and forcing a network hop into a line-rate process, the model was packaged as a .NET-consumable DLL so it could be called directly from the existing application. That removed a whole class of latency and deployment failure modes, at the cost of doing the native packaging and memory-management work properly.

On the modelling side, the emphasis was on the severity grading step: getting a reliable ordering of defect severity out of a detector matters more to the operator than squeezing the last point of detection mAP.

Tech stack

PyTorch · LibTorch · C++ · .NET DLL integration · OpenCV

Outcome

Delivered as an integrated component of the customer’s existing inspection application.

Autopilot Perception on Jetson TX2

Fri, 01 Jan 2021 00:00:00 +0000

Jan 2020 – Jan 2021

Problem

An autonomous vehicle has to see and react inside a control loop that does not wait. On a Jetson TX2, the whole perception stack — not just one model — has to fit in a few watts and a few tens of milliseconds, while staying reliable under changing light, weather, and road surfaces.

Approach

I worked on the perception side of the vehicle: the models that turn camera frames into the lane, sign, and obstacle information the control stack acts on.

The interesting constraint was that accuracy and latency traded against each other continuously. Time spent on a heavier detector is time the controller does not have to react, so the work was as much about inference engineering — optimising the graph, choosing precisions, keeping the pipeline saturated — as about model architecture. The stack was built against TensorRT to get the most out of the TX2’s GPU.

This was developed for the FPT Digital Race, where the system had to run a real course rather than a held-out test set — which surfaces a different class of failure than offline evaluation does.

Tech stack

PyTorch · TensorRT · NVIDIA Jetson TX2 · OpenCV · Python · C++

Outcome

Top 10 — FPT Digital Race 2020 Final.