Personality-Driven AI Avatar Bot

Wed, 01 Jan 2025 00:00:00 +0000

Dec 2023 – Jan 2025

Problem

Group chat applications are text-first. Adding an AI participant that feels like a character rather than a chat completion means giving it a face, a voice, and a consistent personality — and generating all of that fast enough that it can take part in a live conversation rather than replying minutes later.

Approach

I built the avatar generation side of the product: turning a character definition into a speaking AI avatar that could be dropped into a group chat.

The pipeline combines several pieces I had been building in parallel:

Identity generation — a consistent visual identity per character, stable across every clip the bot produces.
Audio-driven talking head — driving lip-sync and head motion from a single audio stream while preserving the character’s identity, head pose, and facial expression.
Expression control — routing the character’s intended emotion (joy, sadness, anger) into the generated video, so the avatar’s delivery matches the tone of what it is saying.
Real-time diffusion video — a Stable Diffusion video generation pipeline producing avatar speech video directly from audio input.

The engineering constraint throughout was latency: a conversational avatar that lags is worse than no avatar, so each stage was optimised for real-time or near-real-time throughput rather than best-possible offline quality.

Tech stack

PyTorch · Stable Diffusion · TensorRT · Python · Docker

Outcome

Shipped as the avatar generation component of a group chat application.

Diffusion Models | Dat Viet Thanh Nguyen

Personality-Driven AI Avatar Bot

Problem

Approach

Tech stack

Outcome