preprint · arXiv (Cornell University) · 2024

Parallel Structures in Pre-training Data Yield In-Context Learning

Yanda Chen, Chen Zhao, Yu Zhou, Kathleen McKeown, He He

Cited by 0Open access

Abstract

Pre-trained language models (LMs) are capable of in-context learning (ICL): they can adapt to a task with only a few examples given in the prompt without any parameter update. However, it is unclear where this capability comes from as there is a stark distribution shift between pre-training text and ICL prompts. In this work, we study what patterns of the pre-training data contribute to ICL. We find that LMs' ICL ability depends on $\textit{parallel structures}$ in the pre-training data -- pairs of phrases following similar templates in the same context window. Specifically, we detect parallel structures by checking whether training on one phrase improves prediction of the other, and conduct ablation experiments to study their effect on ICL. We show that removing parallel structures in the pre-training data reduces LMs' ICL accuracy by 51% (vs 2% from random ablation). This drop persists even when excluding common patterns such as n-gram repetitions and long-range dependency, showing the diversity and generality of parallel structures. A closer look at the detected parallel structures indicates that they cover diverse linguistic tasks and span long distances in the data.

Cite this paper

APA 7

Chen, Y., Zhao, C., Zhou, Y., McKeown, K., & He, H. (2024). Parallel Structures in Pre-training Data Yield In-Context Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2402.12530

MLA 9

Chen, Yanda, et al. “Parallel Structures in Pre-training Data Yield In-Context Learning.” arXiv (Cornell University), 2024, https://doi.org/10.48550/arxiv.2402.12530.

Chicago

Chen, Yanda, Chen Zhao, Yu Zhou, Kathleen McKeown, and He He. 2024. “Parallel Structures in Pre-training Data Yield In-Context Learning.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2402.12530.

Harvard

Chen, Y. et al. (2024) ‘Parallel Structures in Pre-training Data Yield In-Context Learning’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2402.12530.

BibTeX

@misc{chen2024parallel,
  title = {{Parallel Structures in Pre-training Data Yield In-Context Learning}},
  author = {Chen, Yanda and Zhao, Chen and Zhou, Yu and McKeown, Kathleen and He, He},
  journal = {arXiv (Cornell University)},
  year = {2024},
  doi = {10.48550/arxiv.2402.12530}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.