article · IEEE Transactions on Evolutionary Computation · 2024

A Comparison of Large Language Models and Genetic Programming for Program Synthesis

Dominik Sobania, Justyna Petke, Martin Briesch, Franz Rothlauf

Cited by 6Abstract and details only

Abstract

Large language models have recently become known for their ability to generate computer programs, especially through tools, such as GitHub Copilot, a domain where genetic programming (GP) has been very successful so far. Although they require different inputs (free-text versus input/output examples) their goal is the same—program synthesis. Therefore, in this work, we compare how well GitHub Copilot and GP perform on common program synthesis benchmark problems. We study the structure and diversity of the generated programs by using well-known software metrics. We find that GitHub Copilot and GP solve a similar number of benchmark problems (85.2% versus 77.8%, respectively). We find that GitHub Copilot generated smaller and less complex programs as GP, while GP is able to find new and unique problem solving strategies. This increase in diversity of solutions comes at a cost. When analyzing the success rates for 100 runs per problem, GitHub Copilot outperforms GP on over 50% of the problems.

Cite this paper

APA 7

Sobania, D., Petke, J., Briesch, M., & Rothlauf, F. (2024). A Comparison of Large Language Models and Genetic Programming for Program Synthesis. IEEE Transactions on Evolutionary Computation, 29(4), 1434–1448. https://doi.org/10.1109/tevc.2024.3410873

MLA 9

Sobania, Dominik, et al. “A Comparison of Large Language Models and Genetic Programming for Program Synthesis.” IEEE Transactions on Evolutionary Computation, vol. 29, no. 4, 2024, pp. 1434–48, https://doi.org/10.1109/tevc.2024.3410873.

Chicago

Sobania, Dominik, Justyna Petke, Martin Briesch, and Franz Rothlauf. 2024. “A Comparison of Large Language Models and Genetic Programming for Program Synthesis.” IEEE Transactions on Evolutionary Computation 29 (4): 1434–48. https://doi.org/10.1109/tevc.2024.3410873.

Harvard

Sobania, D. et al. (2024) ‘A Comparison of Large Language Models and Genetic Programming for Program Synthesis’, IEEE Transactions on Evolutionary Computation, 29(4), pp. 1434–1448. Available at: https://doi.org/10.1109/tevc.2024.3410873.

BibTeX

@article{sobania2024comparison,
  title = {{A Comparison of Large Language Models and Genetic Programming for Program Synthesis}},
  author = {Sobania, Dominik and Petke, Justyna and Briesch, Martin and Rothlauf, Franz},
  journal = {IEEE Transactions on Evolutionary Computation},
  volume = {29},
  number = {4},
  pages = {1434--1448},
  year = {2024},
  doi = {10.1109/tevc.2024.3410873}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.