Title: Evaluating LLM Reliability Across Facts, Evidence, and Cultures

Speaker: Prof. Yuki Arase @ Institute of Science Tokyo

Time : 16:00 - 17:00, Sep 14th, 2026

Location:

Online: https://hli.skku.edu/InvitedTalk260914

In-person: 85602, Natural Sciences Campus (Seoul), SKKU

Language: English speech & English slides

Abstract:

How should we evaluate an LLM response that is partly correct, contains unsupported phrases, or behaves differently across cultural contexts? Many existing evaluation methods rely on coarse labels or aggregate scores and therefore overlook important variations in factuality, evidential grounding, and cultural sensitivity. This talk presents three of our recent studies toward more fine-grained and inclusive evaluation of large language models. First, I introduce an agentic framework for graded factuality verification that acquires external evidence and assigns scalar factuality scores. Second, I present a method for detecting hallucinated spans while aligning faithful output tokens with supporting evidence in the input. Finally, I briefly introduce a multilingual benchmark for evaluating entity-centric cultural biases across Asian languages and cultures. Together, these studies highlight the need to evaluate not only whether LLM outputs are correct, but also how correct they are, what evidence supports them, and how reliably they perform across cultural contexts.

Bio:

Yuki Arase is a professor at the School of Computing, Institute of Science Tokyo, formerly known as Tokyo Institute of Technology, Japan. She received her PhD in Information Science from Osaka University in 2010 and subsequently worked at Microsoft Research Asia, where she began her research career in natural language processing. Her research interests include paraphrasing and NLP technologies for language education and healthcare. She also serves as a Member-at-Large of the Executive Committee of the Association for Computational Linguistics (ACL), a member of its Peer Review Standing Committee, and a Director of the Association for Natural Language Processing in Japan.