Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

preprint · arXiv (Cornell University) · 2024

Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Rozière, Jonas Gehring, Gabriel Synnaeve, Hugh Leather

Cited by 1Open access

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of software engineering and coding tasks. However, their application in the domain of code and compiler optimization remains underexplored. Training LLMs is resource-intensive, requiring substantial GPU hours and extensive data collection, which can be prohibitive. To address this gap, we introduce Meta Large Language Model Compiler (LLM Compiler), a suite of robust, openly available, pre-trained models specifically designed for code optimization tasks. Built on the foundation of Code Llama, LLM Compiler enhances the understanding of compiler intermediate representations (IRs), assembly language, and optimization techniques. The model has been trained on a vast corpus of 546 billion tokens of LLVM-IR and assembly code and has undergone instruction fine-tuning to interpret compiler behavior. LLM Compiler is released under a bespoke commercial license to allow wide reuse and is available in two sizes: 7 billion and 13 billion parameters. We also present fine-tuned versions of the model, demonstrating its enhanced capabilities in optimizing code size and disassembling from x86_64 and ARM assembly back into LLVM-IR. These achieve 77% of the optimising potential of an autotuning search, and 45% disassembly round trip (14% exact match). This release aims to provide a scalable, cost-effective foundation for further research and development in compiler optimization by both academic researchers and industry practitioners.

Cite this paper

APA 7

Cummins, C., Seeker, V., Grubisic, D., Rozière, B., Gehring, J., Synnaeve, G., & Leather, H. (2024). Meta Large Language Model Compiler: Foundation Models of Compiler Optimization. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2407.02524

MLA 9

Cummins, Chris, et al. “Meta Large Language Model Compiler: Foundation Models of Compiler Optimization.” arXiv (Cornell University), 2024, https://doi.org/10.48550/arxiv.2407.02524.

Chicago

Cummins, Chris, Volker Seeker, Dejan Grubisic, Baptiste Rozière, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. 2024. “Meta Large Language Model Compiler: Foundation Models of Compiler Optimization.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2407.02524.

Harvard

Cummins, C. et al. (2024) ‘Meta Large Language Model Compiler: Foundation Models of Compiler Optimization’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2407.02524.

BibTeX
@misc{cummins2024meta,
  title = {{Meta Large Language Model Compiler: Foundation Models of Compiler Optimization}},
  author = {Cummins, Chris and Seeker, Volker and Grubisic, Dejan and Rozière, Baptiste and Gehring, Jonas and Synnaeve, Gabriel and Leather, Hugh},
  journal = {arXiv (Cornell University)},
  year = {2024},
  doi = {10.48550/arxiv.2407.02524}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.

Related papers