Logo

Workshop on Insights from Negative Results in NLP


June, 2024
(co-located with NAACL)

About

Call for papers

Accepted papers

Program

Sponsors

Organization

Invited Speakers

Program Committee

Insights 2023

Insights 2022

Insights 2021

Insights 2020

Call for papers

(June 2024, co-located with NAACL 2024)

Overview

Publication of negative results is difficult in most fields, but in NLP the problem is exacerbated by the near-universal focus on improvements in benchmarks. This situation implicitly discourages hypothesis-driven research, and it turns creation and fine-tuning of NLP models into art rather than science. Furthermore, it increases the time, effort, and carbon emissions spent on developing and tuning models, as the researchers have no opportunity to learn what has already been tried and failed.

This workshop invites both practical and theoretical unexpected or negative results that have important implications for future research, highlight methodological issues with existing approaches, and/or point out pervasive misunderstandings or bad practices. In particular, the most successful NLP models currently rely on Transformer-based large language models (LLMs). To complement all the success stories, it would be insightful to see where and possibly why they fail. Any NLP tasks are welcome: sequence labeling, question answering, inference, dialogue, machine translation - you name it.

A successful negative results paper would contribute one of the following:

In 2024, we will invite the authors of accepted negative results papers to nominate the specific work reporting the original positive results. The goal is to organize joint discussion sessions, so that the community can learn the most from the specific insightful failure.

Important Dates

All deadlines are 11.59 pm UTC -12h (“anywhere on Earth”).

Submissions

Submission is electronic, using the Softconf START conference management system. Submission link: https://softconf.com/naacl2024/insights2024

The workshop will accept short papers (up to 4 pages, excluding references), as well as 1-2 page non-archival abstract submissions for papers published elsewhere (e.g. in one of the main conferences or in non-NLP venues). The goal of this event is to stimulate a meaningful community-wide discussion of the deep issues in NLP methodology, and the authors of both types of submissions will be welcome to take part in our get-togethers.

The workshop will run its own review process, and papers can be submitted directly to the workshop by March 10, 2024. It is also possible to submit a paper accompanied with reviews from the ACL Rolling Review system by April 7, 2024. The submission deadline for ARR papers follows the ACL RR calendar. Both research papers and abstracts must follow the ACL two-column format. Official style sheets: Overleaf template

Please do not modify these style files, nor should you use templates designed for other conferences. Submissions that do not conform to the required styles, including paper size, margin width, and font size restrictions, will be rejected without review. Please follow the formatting guidelines outlined here: formatting guidelines

Authorship

The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. No changes to the order or composition of authorship may be made after the paper submission deadline.

Citation and Comparison

You are expected to cite all refereed publications relevant to your submission, but you may be excused for not knowing about all unpublished work (especially work that has been recently posted and/or is not widely cited).

In cases where a preprint has been superseded by a refereed publication, the refereed publication should be cited instead of the preprint version.

Papers (whether refereed or not) appearing less than 3 months before the submission deadline are considered contemporaneous to your submission, and you are therefore not obliged to make detailed comparisons that require additional experimentation and/or in-depth analysis.

For more information, see the ACL Policies for Submission, Review, and Citation.

Multiple Submission Policy

The workshop cannot accept work for publication or presentation that will be (or has been) published elsewhere and that have been or will be submitted to other meetings or publications whose review periods overlap with that of Insights. Any questions regarding submissions can be sent to insights-workshop-organizers@googlegroups.com.

If the paper has been rejected from another venue, the authors will have the option to provide the original reviews and the author response. The new reviewers will not have access to this information, but the organizers will be able to take into account the fact that the paper has already been revised and improved.

Anonymity Period

The workshop will follow the new ACL policy

Presentation

All accepted papers must be presented at the workshop to appear in the proceedings. Authors of accepted papers must notify the program chairs by the camera-ready deadline if they wish to withdraw the paper. At least one author of each accepted paper must register for the workshop.

Previous presentations of the work (e.g. preprints on arXiv.org) should be noted in a footnote in the camera-ready version (but not in the anonymized version of the paper).

The workshop will take place during NAACL 2024 (June 16-21 2024). It will be hybrid, allowing for both in-person and virtual presentations.

Organization Committee

Contact info

Any questions regarding the workshop can be sent to insights-workshop-organizers@googlegroups.com.