Visual Speech Recognition: Lip Segmentation and Mapping

Seeing Speech: The Visual Dimension of Recognition

When we listen to someone speak, we rarely rely on sound alone. The movement of the lips, tongue, and face carries a wealth of information that can clarify ambiguous sounds, aid understanding in noisy environments, and even identify a speaker. Visual Speech Recognition: Lip Segmentation and Mapping brings together research at the intersection of computer vision, speech processing, and biometrics, focusing on how visual cues can be extracted, modeled, and used in automatic recognition systems.

Edited by Alan Wee-Chung Liew and Shilin Wang, this volume collects technical chapters from leading groups across the world. The contributors address the full pipeline from video input to recognition decision, including lip segmentation, feature extraction, modeling, and classification.

What the Book Covers

The chapters are organized into four main areas. The first section surveys audio-visual and visual-only speech and speaker recognition, as well as fundamental techniques for lip feature extraction and segmentation. A second group of chapters explores advanced segmentation methods for complex backgrounds, natural lighting, 3D shape modeling, and the challenging task of evaluating lip analysis systems with manual labelers.

Later chapters turn to recognition itself: visual speech processing, recognition across multiple camera views, viseme modeling with hidden Markov models, motion features, and even the possibility of extracting prosodic focus from lip movement. The final section covers visual speaker recognition and biometric person recognition based on lip motion.

Who This Volume Serves

Graduate students, researchers, and engineers working in speech processing, computer vision, multimodal interaction, or biometric security will appreciate the book’s combination of survey material and practical technical depth. The chapters assume a solid understanding of statistical pattern recognition and image processing, but they provide enough context to help readers new to visual speech recognition understand why the visual channel is valuable and where current research is headed.

A Technical Foundation for Visual Speech Research

Because visual speech recognition is inherently interdisciplinary, the book draws on methods from image analysis, machine learning, speech science, and human-computer interaction. The editors have organized the material so that readers can follow the progression from low-level lip segmentation to high-level speaker recognition. For anyone building or studying systems that combine audio and visual information, this volume provides a solid technical reference.