April 19, 2023

Kelvin Guu is virtually visiting UW and giving a talk on Friday, April 28th. Kelvin is a staff research scientist & manager at Google. He has done amazing work on factuality and attribution in LLMs. Everyone interested in this topic is welcome to join the talk!
Date & Time: Friday, April 28th, 12–1:30pm
Location: https://washington.zoom.us/j/96847566049

Title: “Fact-checking” Large Language Models and Tracing Their Knowledge Back To Training Examples

Abstract:

LLMs perform ever more impressive feats every day. But we often still cannot trust what they say (due to hallucinations), or determine where they learned a particular belief, behavior, or skill. To tackle these challenges, I will share two new research ideas. In the first approach (RARR), I show how we can fix many LLM hallucinations using a sequence of simple “fact-checking” steps that LLMs already perform well: knowledge retrieval, entailment, and text editing to fix any errors. In the second half (Simfluence), I turn to the causal question of where particular LLM errors or successes come from — tracing model behavior back to specific training examples. Simfluence tracks how much “smarter” your model gets after consuming each example, and uses this to simulate how your model would have performed if certain examples were removed, reweighted, etc.

Bio:

Kelvin Guu is a staff research scientist and manager at Google Brain. His recent research includes some of the early work on retrieval-augmented language models (REALM) and training LLMs to follow instructions (FLAN). At Google, he led some of the first efforts to leverage pre-trained LMs and neural retrievers, with >30 launches across multiple products.