Program

The goal of this workshop is to provide not only a publication venue for negative results, but also a forum for meaningful discussion and learning from them. This year the event will feature invited talks + live QA with Rada Mihalcea and Byron C. Wallace, and five thematic sessions, with live group QA with the authors working in the same subfield. These sessions wil span NLI, QA, dialogue, and representation learning, plus a general session for “lessons learned the hard way”. Eight more insightful negative results papers on a wide range of topics will be presented in an interactive section on gather.town, to be immediately followed by a panel on leaderboardism, reproducibility, and frustrations of trying to getting published systems to work (with Kawin Ethayarajh, Jesse Dodge and Rachael Thomas). And of course there will be plenty of opportunities to chat in gather.town during three social breaks.

Small changes may occur, please check the program again before the event. All times are in PST.

---

7:00–7:15 Opening remarks

7:15–8:15 Invited talk + live QA: Rada Mihalcea (University of Michigan) [VIDEO]
The ups and downs of word embeddings

Abstract: Word embeddings have largely been a “success story” in our field. They have enabled progress in numerous language processing applications, and have facilitated the application of large-scale language analyses in other domains, such as social sciences and humanities. While less talked about, word embeddings also have many shortcomings – instability, lack of transparency, biases, and more. In this talk, I will review the “ups” and “downs” of word embeddings, discuss tradeoffs, and chart potential future research directions to address some of the downsides of these word representations.

8:15–8:45 Thematic session + group QA: representation learning

• Embedding Structured Dictionary Entries
  Steven Wilson, Walid Magdy, Barbara McGillivray and Gareth Tyson
• Can Multimodal Embeddings Tell Us What Fact-checked Claims Are About?
  Valentina Beretta, Sébastien Harispe, Katarina Boland, Luke Lo Seen, Konstantin Todorov and Andon Tchechmedjiev
• Layout-Aware Text Representations Harm Clustering Documents by Type
  Catherine Finegan-Dollak and Ashish Verma

8:45–9:15 Thematic session + group QA: dialogue

• On Task-Level Dialogue Composition of Generative Transformer Model
  Prasanna Parthasarathi, Sharan Narang and Arvind Neelakantan
• HINT3: Raising the bar for Intent Detection in the Wild
  Gaurav Arora, Chirag Jain, Manas Chaturvedi and Krupal Modi
• Effects of Naturalistic Variation in Goal-Oriented Dialog
  Jatin Ganhotra, Robert Moore, Sachindra Joshi and Kahini Wadhawan

9:15–10:00 Social break / meal time

10:00–11:00 Invited talk + live QA: Byron C. Wallace (Northeastern University) [VIDEO]
Negative results yield interesting questions, or: a bunch of stuff that didn’t work

Abstract: I will discuss recent projects in which ideas did not pan out as expected, but where these initial negative results led to (arguably) more interesting questions. My hope is that these case studies of negative results — which ultimately led to work we viewed as compelling enough to warrant write-up — will foster discussion about when “negative” results are nonetheless interesting, and about the kinds of questions we ask in empirical NLP research.

The first two cases that I will discuss involve methods for learning from limited amounts of supervision, namely active learning and data augmentation. Concerning the former, we set out to investigate whether training data selectively sampled using one model was also comparatively useful when used to train a second, distinct model. But what we found was that active learning in general afforded modest and inconsistent benefits across datasets, even when considering only the first model. These findings reframed the work: Where we had been asking a somewhat narrow question about dataset transfer, our results called into question whether active learning was worth using in the first place, given inconsistent benefits and inherent drawbacks. With respect to data augmentation, we developed (what we thought was) a clever augmentation strategy appropriate for sequence tagging that seemed to afford performance benefits when coupled with a recently proposed unsupervised data augmentation method. But then we realized that random augmentation strategies worked just as well, and actually random augmentation (as opposed to, e.g., performing backtranslation) works for text classification, too. These findings raised the question: When augmentation “works” for NLP, what is behind this? Time permitting, I will also discuss ongoing work in which we have (so far) failed to: (i) Recover sensitive patient information from transformers pretrained over clinical notes, and (ii) Improve the factuality of abstractive summaries by explicitly penalizing inaccurate tokens, and how we are thinking about moving forward with these initial negative results.

11:00–11:30 Thematic session + group QA: question answering

• Do Transformers Dream of Inference, or Can Pretrained Generative Models Learn Implicit Inferential Rules?
  Zhengzhong Liang and Mihai Surdeanu
• What do we expect from Multiple-choice QA Systems?
  Krunal Shah, Nitish Gupta and Dan Roth
• Q. Can Knowledge Graphs be used to Answer Boolean Questions? A. It’s complicated!
  Daria Dzendzik, Carl Vogel and Jennifer Foster

11:30–12:00 Thematic session + group QA: natural language inference

• Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data
  William Huang, Haokun Liu and Samuel R. Bowman
• The Extraordinary Failure of Complement Coercion Crowdsourcing
  Yanai Elazar, Victoria Basmov, Shauli Ravfogel, Yoav Goldberg and Reut Tsarfat
• Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder
  Alvin Chan, Yi Tay, Yew-Soon Ong and Aston Zhang

12:00–12:30 Thematic session + group QA: lessons learned the hard way

• Evaluating the Effectiveness of Efficient Neural Architecture Search for Sentence-Pair Tasks
  Ansel MacLaughlin, Jwala Dhamala, Anoop Kumar, Sriram Venkatapathy, Ragav Venkatesan and Rahul Gupta
• NMF Ensembles? Not for Text Summarization!
  Alka Khurana and Vasudha Bhatnagar
• If You Build Your Own NER Scorer, Non-replicable Results Will Come
  Constantine Lignos and Marjan Kamyab

12:30–13:00 Social break / meal time

13:00–14:00 Interactive orals

• Domain adaptation challenges of BERT in tokenization and sub-word representations of Out-of-Vocabulary words
  Anmol Nayak, hariprasad timmapathini, karthikeyan Ponnalagu and Vijendran Gopalan Venkoparao
• How Far Can We Go with Data Selection? A Case Study on Semantic Sequence Tagging Tasks
  Samuel Louvan and Bernardo Magnini
• Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models
  Silvia Terragni, Debora Nozza, Elisabetta Fersini and Messina Enza
• How Effectively Can Machines Defend Against Machine-Generated Fake News? An Empirical Study
  Meghana Moorthy Bhat and Srinivasan Parthasarathy
• Label Propagation-Based Semi-Supervised Learning for Hate Speech Classification
  Ashwin Geet D’Sa, Irina Illina, Dominique Fohr and Dietrich Klakow
• An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios
  Andrew Zupon, Faiz Rafique and Mihai Surdeanu
• How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?
  Shayne Longpre, Yu Wang and Christopher DuBois
• WER we are and WER we think we are
  Piotr Szymański, Piotr Żelasko, Mikolaj Morzy, Adrian Szymczak, Marzena Żyła-Hoppe, Joanna Banaszczak, Lukasz Augustyniak, Jan Mizgajski and Yishay Carmiel

14:00–14:45 The frustrations of leaderboardism. Panel discussion with Kawin Ethayarajh, Jesse Dodge and Rachael Tatman [VIDEO]

Leaderboards do not only drive progress in NLP: the bias towards publication of positive, and particularly state-of-the-art results implicitly encourages the development of highly specialized and brittle systems. If the reported success cannot be reproduced, or does not generalize well, the main result is much frustration by the developers who pick up academic papers in search of something that would actually work.

Relevant work by the panelists:

• Utility is in the Eye of the User: A Critique of NLP Leaderboards. EMNLP 2020.
  K. Ethayarajh, D. Jurafsky
• Show Your Work: Improved Reporting of Experimental Results. EMNLP 2019.
  J. Dodge, S. Gururangan, D. Card, R. Schwartz, N. Smith
• A Practical Taxonomy of Reproducibility for Machine Learning Research. 2nd Reproducibility in Machine Learning Workshop at ICML 2018.
  R. Tatman, J. VanderPlas, S. Dane

14:45–15:00 Breakout

15:00–15:15 Closing remarks

15:15–16:00 Virtual happy hour