Method comparison
More expressive than lip-sync-first methods
On the same portrait and audio, VAExpress produces clearer motion in the eyes, brows, cheeks, and overall facial expression.
Expressive talking-face generation
Make the face match the feeling.
VAExpress turns one portrait + one audio clip into a natural talking-face video. It goes beyond lip sync by generating facial expressions that follow the emotion of the voice.
- Emotion from audio
- Continuous emotion control
- Identity-aware generation
Goal
Generate a talking face that is not only synchronized, but also emotionally expressive.
Method
Continuous Valence-Arousal guidance with spatial-temporal decoupled training.
Result
Readable expressions, controllable intensity, and smooth emotion transitions.
Generated results
Judge it in motion.
Three independent windows show method comparison, audio-driven emotion, and continuous emotion control.
Audio-driven emotion
The voice can direct the expression
When no manual control is supplied, emotion cues are estimated from speech and used to guide the facial performance.
Continuous emotion control
Adjust intensity and transition smoothly
A continuous emotion sequence can raise expression intensity gradually or move between affective states without abrupt category changes.
Audio-aligned
Expressions follow the affective state of speech.
Fine-grained
Intensity can change continuously over time.
Temporally coherent
Transitions stay smooth without flattening expression.
How it works
Three inputs, one expressive video.
Overall pipeline and two-stage training
Reference identity, speech, and continuous VA emotion signals enter separate conditioning branches before being combined by the diffusion generator. The right side shows the image-first, video-second training strategy.
Continuous guidance
Valence controls feeling. Arousal controls energy.
Instead of choosing only a fixed label such as “happy” or “sad,” VAExpress uses a continuous two-dimensional emotion signal. This makes subtle intensity changes and smooth transitions possible.
Decoupled training
Learn expression first, then learn smooth motion.
Training spatial and temporal modules together can over-smooth facial movement. VAExpress first learns accurate emotion-to-expression mappings on images, then adds temporal training for coherent video.
- Stage 1Static expression learning
- Stage 2Temporal adaptation
Clear expression response, then smooth temporal change.
The top row shows expressions aligned with speech. The bottom row shows that continuous VA signals can adjust intensity and drive an emotion transition. The two-stage training keeps this expressive response before adding temporal coherence.
Temporal modules can suppress expression response.
The orange curves with temporal modules are flatter and react more slowly to changing arousal. The blue curves without them follow the control signal more closely. This motivates learning expressive mappings first, then adding temporal coherence.
The practical effect is visible in the clips above. Controlled evaluation also shows stronger expression accuracy and video quality on MEAD and RAVDESS.
View the full evaluation in the technical report ↗VAExpress