Invited Speech 5: Wednesday 11:30-12:30, December 20th, 2023

Abstract

The Journey of Advancements in Speech Foundation Models

Recent months have seen a surge in discussions about the capabilities of text foundation models, particularly large language models (LLMs). Known for their general processing abilities, LLMs can effectively perform a variety of tasks with appropriate instructions. Unlike text, speech contains rich, hierarchical information, necessitating distinct capabilities for diverse applications. This raises the question: how close are we to developing speech foundation models that can understand and execute task instructions?

This presentation delves into the evolution of foundation models in speech processing, highlighting three significant phases: shared encoders with task-specific heads, universal models with adaptable parameters, and task instruction models. It begins with an introduction to the Speech Processing Universal PERformance Benchmark (SUPERB), which assesses shared encoders across multiple tasks. The discussion then shifts to exploring the use of prompting in speech language models. The presentation concludes with a focus on Dynamic SUPERB, a project aimed at evaluating task instruction models in speech processing.

Biography

Hung-yi Lee (李宏毅) is a professor of the Department of Electrical Engineering of National Taiwan University (NTU), with a joint appointment at the Department of Computer Science & Information Engineering of the university. His recent research focuses on developing technology that can reduce the requirement of annotated data for speech processing and natural language processing. He won Salesforce Research Deep Learning Grant in 2019, AWS ML Research Award in 2020, Outstanding Young Engineer Award from The Chinese Institute of Electrical Engineering in 2018, Young Scholar Innovation Award from Foundation for the Advancement of Outstanding Scholarship in 2019, Ta-You Wu Memorial Award from Ministry of Science and Technology of Taiwan in 2019, and the 59th Ten Outstanding Young Person Award in Science and Technology Research & Development of Taiwan. He owns a YouTube channel teaching deep learning in Mandarin with about 160k Subscribers.