Workshop on Insights from Negative Results in NLP
Budapest, Hungary, October 29, 2026
(co-located with EMNLP)
Invited Speakers
Anya Belz
Quantified Reproducibility Assessment for NLP/ML
Bio. Anya Belz is Professor of Computer Science at Dublin City University and PI in the Rinn AI Research Centre. Her research spans natural language generation, evaluation of NLP systems, and reproducibility, with a recent focus on formal methods for assessing and quantifying reproducibility in NLP, including the ReproHum project on reproducibility in human evaluation, and the ReproNLP shared task series. More broadly, Belz has worked on human evaluation methodology, standardised evaluation reporting, multilingual text generation, data-to-text and controlled text generation, and efficiency, modularity and reasoning augmentation for large language models.
Ryan L. Boyd
What Is a Result a Result About?
Abstract. Empirical findings are rarely about the single thing that we take them to be about. Any given result is the product of several intersecting factors: a hypothesis, a measure, an operationalization, and a considerable number of assumptions that are unexamined. When the bundle performs well, we credit the hypothesis; when it performs poorly, we fault the hypothesis as well. Neither inference is especially well-founded. In this talk, I draw on case material and several long-standing arguments from the philosophy of science to consider what our results are actually about, and what it would take to find out.
Bio. Ryan L. Boyd is an Assistant Professor of Connected Computing at Vanderbilt University. His research applies natural language processing and machine learning to the study of verbal behavior, developing computational systems for measuring and understanding psychological constructs — including mental health, self-perception, extremism, and social influence — in large-scale naturalistic data. A recurring concern in his research is the gap between what language-based models are assumed to measure and what they actually capture, and the methodological consequences of that gap for using text as psychological measurement. He has published over 100 scholarly works across CS venues (ACL, NAACL, ICWSM) and psychology journals (American Psychologist, Perspectives on Psychological Science, Nature Human Behaviour), and holds a PhD in Social and Personality Psychology from the University of Texas at Austin.
Dirk Hovy
When Success Is the Negative Result: What NLP Evaluations Fail to Establish
Abstract. NLP has made undeniable progress. However, we often point towards high scores, fluent output, or statistically significant results as evidence of an underlying model capability. In this talk, I will argue that the true progress often lies precisely in what those things can not measure. Drawing on recent studies of socioeconomic variation in human–LLM interaction, moral plurality, LLM annotation, and automated peer review, we look at how aggregate metrics conceal population differences, how fluent behavior fails to establish social understanding, and how seemingly robust conclusions depend on arbitrary choices of prompts, models, or evaluation procedures. Rather than viewing such outcomes as failed research, I argue that such negative results serve as useful boundary markers for our claims, and as guidelines for what to explore next.