Pith. sign in

REVIEW

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2101.00133 v2 pith:ZF4KWATK submitted 2021-01-01 cs.CL cs.AI

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Sewon Min , Jordan Boyd-Graber , Chris Alberti , Danqi Chen , Eunsol Choi , Michael Collins , Kelvin Guu , Hannaneh Hajishirzi
show 45 more authors
Kenton Lee Jennimaria Palomaki Colin Raffel Adam Roberts Tom Kwiatkowski Patrick Lewis Yuxiang Wu Heinrich K\"uttler Linqing Liu Pasquale Minervini Pontus Stenetorp Sebastian Riedel Sohee Yang Minjoon Seo Gautier Izacard Fabio Petroni Lucas Hosseini Nicola De Cao Edouard Grave Ikuya Yamada Sonse Shimaoka Masatoshi Suzuki Shumpei Miyawaki Shun Sato Ryo Takahashi Jun Suzuki Martin Fajcik Martin Docekal Karel Ondrej Pavel Smrz Hao Cheng Yelong Shen Xiaodong Liu Pengcheng He Weizhu Chen Jianfeng Gao Barlas Oguz Xilun Chen Vladimir Karpukhin Stan Peshterliev Dmytro Okhonko Michael Schlichtkrull Sonal Gupta Yashar Mehdad Wen-tau Yih
This is my paper
classification cs.CL cs.AI
keywords competitionsystemsanswersbudgetsefficientqalanguagelearnedmemory
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.