Research

Our lab studies intelligent systems that can understand, generate, and interact with humans through multiple modalities, including speech, audio, vision, and language. We aim to develop AI models that move beyond single-modality perception and toward human-centered multimodal intelligence. Our research spans multimodal learning, audio-visual processing, multimodal large language models, generative AI, and efficient AI.

Multimodal Human-Centered AI

Multimodal learning focuses on building AI systems that can jointly understand and reason across different types of information, such as visual signals, speech, audio, and text. Our lab explores representation learning, cross-modal alignment, and multimodal reasoning to enable more robust and human-like AI systems.

Speech, Audio, and Multimodal Foundation Models

We study foundation models that can process and generate speech, audio, visual, and textual information. In particular, we are interested in connecting speech and audio representations with large language models and transforming text-based foundation model into speech-based foundation model to enable spoken dialogue and interactive AI agents. Our research includes speech-language model alignment, long-form speech understanding, and multimodal large language models that understand human communication beyond text.

Audio-Visual Processing 

Audio-visual processing investigates how speech, sound, and visual information interact in real-world communication. Human perception naturally combines what we hear with what we see, such as lip movements, facial expressions, gestures, and acoustic cues. Our lab develops models that jointly process audio and visual signals for speech understanding, speech-driven facial animation, and audio-visual dialogue. We aim to build AI systems that can perceive and generate both audio and visual communication signals with accurate synchronization, expressiveness, and realism.

Generative AI for Human Interaction

We develop generative models for realistic and expressive speech, audio, faces, videos, and multimodal responses. Our research focuses on speech tokenizer, emotionally aware multimodal generation, and controllable generation across multiple speakers, identities, and conversational styles. Our research explores speech and audio generation, talking face synthesis, audio-visual dialogue, and conversational agents that can listen and respond in a natural, overlapping manner. 

Efficient Multimodal AI 

As multimodal and generative AI models become increasingly large, efficiency is essential for real-time interaction and practical deployment. We explore efficient model architectures, compact multimodal representations, low-latency inference, and scalable training strategies. We aim to build AI systems that are not only powerful, but also efficient and deployable.