Lehrstuhl Medieninformatik

Computer Graphics

Example based Facial Animation in Images and Video

Based on a dataset of 3D scans of different facial expressions and mouth shapes, we have formed a vector space of 3D expressions. Difference vectors, such as the smile-vector shown above, can be added to new individual faces. Unlike physical models, we observe the appearance of expressions, rather than simulating the muscle forces and tissue properties that cause the surface deformations.

With 3D shape reconstruction, we can apply these expressions to any novel face in an image. Since the expressions are 3D, the method works at any pose and illumination. The modified 3D face model is rendered back into the original image. Unlike many other animation techniques, we can animate faces within their 2D scene contexts.

The method can also be applied to video data: From each original frame (top row), an estimate of pose and mouth shape is calculated (second row). 3D shape and texture were reconstructed from 3 selected frames of the sequence. In the third row, the face with new mouth shapes is rendered into the original image.

Movies:

Mouth Movements in Paintings	Speech Animation in Paintings	Animation of Video
MPEG 7.1MB	MPEG 9.8 MB	MPEG 3.9MB
Quicktime 6.8MB	Quicktime 31MB (recommended)	Quicktime 23MB (recommended)

Lit:
Blanz, V. , Basso, C. , Poggio, T. and Vetter, T. Reanimating Faces in Images and Video, Computer Graphics Forum 22 (3), EUROGRAPHICS 2003, Granada, Spain, p. 641 - 650, 2003. pdf EG2003 Best Paper Award (1st Prize)