article · IEEE Transactions on Software Engineering · 2023
INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
Cited by 7Abstract and details only
Abstract
Pre-trained models of source code have recently been successfully applied to a wide variety of Software Engineering tasks; they have also seen some practical adoption in practice, e.g. for code completion. Yet, we still know very little about <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">what</i> these pre-trained models learn about source code. In this article, we use <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">probing</i> —simple diagnostic tasks that do not further train the models—to discover to what extent pre-trained models learn about specific aspects of source code. We use an extensible framework to define 15 probing tasks that exercise surface, syntactic, structural and semantic characteristics of source code. We probe 8 pre-trained source code models, as well as a natural language model ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BERT</monospace> ) as our baseline. We find that models that incorporate some structural information (such as <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GraphCodeBERT</monospace> ) have a better representation of source code characteristics. Surprisingly, we find that for some probing tasks, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BERT</monospace> is competitive with the source code models, indicating that there are ample opportunities to improve source-code specific pre-training on the respective code characteristics. We encourage other researchers to evaluate their models with our probing task suite, so that they may peer into the hidden layers of the models and identify what intrinsic code characteristics are encoded.
Cite this paper
Karmakar, A., & Robbes, R. (2023). INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers. IEEE Transactions on Software Engineering, 50(2), 220–238. https://doi.org/10.1109/tse.2023.3341624
Karmakar, Anjan, and Romain Robbes. “INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers.” IEEE Transactions on Software Engineering, vol. 50, no. 2, 2023, pp. 220–38, https://doi.org/10.1109/tse.2023.3341624.
Karmakar, Anjan, and Romain Robbes. 2023. “INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers.” IEEE Transactions on Software Engineering 50 (2): 220–38. https://doi.org/10.1109/tse.2023.3341624.
Karmakar, A. and Robbes, R. (2023) ‘INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers’, IEEE Transactions on Software Engineering, 50(2), pp. 220–238. Available at: https://doi.org/10.1109/tse.2023.3341624.
@article{karmakar2023inspect,
title = {{INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers}},
author = {Karmakar, Anjan and Robbes, Romain},
journal = {IEEE Transactions on Software Engineering},
volume = {50},
number = {2},
pages = {220--238},
year = {2023},
doi = {10.1109/tse.2023.3341624}
}Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.