The Science of Evaluation and Alignment for Large Language Models

Lei Li

Carnegie Mellon University

Abstract: Can large language models (LLMs) fairly evaluate and refine model generation performance? Does a large language model reliably know specific knowledge, or does it answer by luck? In this talk, we will present scientific methods for evaluating large language models for knowledge-intense and language-generation tasks. We observe self-bias when using LLMs as evaluators—an LLM favors its own output. We will further discuss post-training methods to refine and align LLMs better with human judgment and valuation.

Bio: Lei Li is an Assistant Professor in Language Technologies Institute at Carnegie Mellon University. His research focuses on machine translation, trustworthy LLMs, and AI drug discovery. He received Ph.D. from CMU School of Computer Science in 2011. He is a recipient of the ACL 2021 Best Paper Award, the CCF Young Elite Award in 2019, the CCF Distinguished Speaker in 2017, the Wu Wen-tsün AI prize in 2017, and the 2012 ACM SIGKDD dissertation award (runner-up), and is recognized as Notable Area Chair of the ICLR 2023. Previously, he was an associate professor (tenured) at UC Santa Barbara. Before that, he was the Founding Director of ByteDance AI Lab, a principal scientist at Baidu, and a postdoc researcher at UC Berkeley. He led and developed ByteDance’s machine translation system VolcTrans and AI writing system Xiaomingbot, and many of his algorithms have been deployed in products (Toutiao, Douyin, Tiktok, Lark), serving over one billion users.

Web: https://www.cs.cmu.edu/~leili/