Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

article · 2022

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Michael Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

Cited by 619Open access

Abstract

Large language models (LMs) are able to in-context learn—perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required—randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of endtask performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.

Cite this paper

APA 7

Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? https://doi.org/10.18653/v1/2022.emnlp-main.759

MLA 9

Min, Sewon, et al. “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?” 2022, pp. 11048–64, https://doi.org/10.18653/v1/2022.emnlp-main.759.

Chicago

Min, Sewon, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Michael Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?” https://doi.org/10.18653/v1/2022.emnlp-main.759.

Harvard

Min, S. et al. (2022) ‘Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?’. Available at: https://doi.org/10.18653/v1/2022.emnlp-main.759.

BibTeX
@article{min2022rethinking,
  title = {{Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?}},
  author = {Min, Sewon and Lyu, Xinxi and Holtzman, Ari and Artetxe, Mikel and Lewis, Michael and Hajishirzi, Hannaneh and Zettlemoyer, Luke},
  pages = {11048--11064},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.759}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.

Related papers