Qingtao Jiang, Xin Peng, Hai Wang et al. · Journal of Software Evolution and Process · 2017
Abstract The life cycle of a large‐scale software system can undergo many releases. Each release often involves hundreds or thousands of revisions committed by many developers over time. Many code changes are made in a systematic and collaborative way. However, such systematic and collaborative code changes are often undocumented and hidden in the evolution history of a software system. It is desirable to recover commonalities and associations among dispersed code changes in the evolutionary trajectory of a software system. In this paper, we present Summarizing Evolutionary Trajectory by Grouping and Aggregation (SETGA), an approach to summarizing historical commit records as trajectory patterns by grouping and aggregating relevant code changes committed over time. The SETGA extracts change operations from a series of commit records from version control systems. It then groups extracted change operations by their common properties from different dimensions such as change operation types, developers, and change locations. After that, SETGA aggregates relevant change operation groups by mining various associations among them. We implement SETGA and conduct an empirical study with 3 open‐source systems. We investigate underlying evolution rules and problems that can be revealed by the identified patterns and analyze the evolution of trajectory patterns in different periods. The results show that SETGA can identify various types of trajectory patterns that are useful for software evolution management and quality assurance.
Cited by 8
Fabiano Cutigi Ferrari, Vinícius H. S. Durelli, Sten F. Andler et al. · Software Testing Verification and Reliability · 2023
Summary Model‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT: (i) some studies did not fully describe how tools transform abstract tests into concrete tests, (ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments.
Cited by 4Open access
Anonymous Author(s) · Zenodo (CERN European Organization for Nuclear Research) · 2026
Digital Twins (DTs) have emerged as a core technology across various domains, such as automotive and manufacturing, enabling the digital representation, monitoring, and control of physical systems. However, the development of DTs is often complex and time-consuming, as it requires integrating domain-specific knowledge with software development expertise. Low-code development approaches aim to reduce technical complexity by providing higher levels of abstraction. While these approaches are gaining attention, it remains unclear how such abstraction mechanisms are currently applied in the development of DTs.To address this challenge, we conduct a systematic mapping study of low-code development solutions (LCDSs) for DTs. From an initial set of 374 retrieved publications, 39 primary studies were selected for detailed analysis based on predefined selection criteria.Our results provide a structured overview of the research landscape on LCDSs for DTs and highlight current limitations as well as directions for future research.
Cited by 0Open access
Anonymous Author(s) · Zenodo (CERN European Organization for Nuclear Research) · 2026
Digital Twins (DTs) have emerged as a core technology across various domains, such as automotive and manufacturing, enabling the digital representation, monitoring, and control of physical systems. However, the development of DTs is often complex and time-consuming, as it requires integrating domain-specific knowledge with software development expertise. Low-code development approaches aim to reduce technical complexity by providing higher levels of abstraction. While these approaches are gaining attention, it remains unclear how such abstraction mechanisms are currently applied in the development of DTs.To address this challenge, we conduct a systematic mapping study of low-code development solutions (LCDSs) for DTs. From an initial set of 374 retrieved publications, 39 primary studies were selected for detailed analysis based on predefined selection criteria.Our results provide a structured overview of the research landscape on LCDSs for DTs and highlight current limitations as well as directions for future research.
Cited by 0Open access
Haolin Jin, Huaming Chen · arXiv (Cornell University) · 2026
Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to verify if code implementation satisfy task requirements, thereby ensuring code robustness and accuracy. However, it remains unclear whether LLMs can reliably determine code against the given task descriptions, which is usually in a form of natural language specifications. In this paper, we uncover a systematic failure of LLMs in matching code to natural language requirements. Specifically, with widely adopted benchmarks and unified prompts design, we demonstrate that LLMs frequently misclassify correct code implementation as non-compliant or defective. Surprisingly, we find that more detailed prompt design, particularly with those requiring explanations and proposed corrections, leads to higher misjudgment rates, highlighting critical reliability issues for LLM-based code assistants. We further analyze the mechanisms driving these failures and evaluate the reliability of rationale-required judgments. Building on these findings, we propose a Fix-guided Verification Filter that treats the model proposed fix as executable counterfactual evidence, and validates the original and revised implementations using benchmark tests and spec-constrained augmented tests. Our results expose previously under-explored limitations in LLM-based code review capabilities, and provide practical guidance for integrating LLM-based reviewers with safeguards in automated review and development pipelines.
Cited by 0Open access
Sofia Ali, Giovanni Sileno, Tom van Engers · Frontiers in artificial intelligence and applications · 2025
This paper presents a framework for evaluating approaches to formalizing legal sources. The framework integrates adoption-related criteria, extending the scope of assessment beyond purely technical dimensions. We build upon three contemporary comparison frameworks for normative specification languages, combining the Technology-Organization-Environment framework with grounded theory and adding criteria from other IT fields. As an illustration, we applied the resulting framework to RegelSpraak (RuleSpeak) from the Dutch Tax Administration.
Cited by 0Open access
Kicheol Kim, Misoo Kim, Eunseok Lee · 2022
Automated program repair(APR) is in the spotlight in academia and the field to reduce the time and cost of maintenance for developers. Recently, APR has continued to study based on deep-learning models to understand and learn how to fix software bugs. Text-to-Text Transfer Transformer(T5), which scored state-of-the-art in natural language processing benchmarks, also showed promising results on program repair in recent studies. In deep-learning-based program repair studies, studies commonly propose code abstraction techniques to avoid vocabulary problems and learn fine code transformation to generate bug-fixing patches. However, there is not enough systematic analysis of code abstraction according to each bug type in deep-learning-based program repair. Therefore, We leverage TFix, T5-based program repair, to evaluate how code abstraction techniques affect neural program repair. Our experimental results showed that defect-specific code abstraction achives a higher average BLEU score than the existing code abstraction technique in both T5 and multilingual-T5(mT5) model-based TFix results. Also, mT5 model-based TFix, which is applied defect-specific code abstraction, gets a higher BLEU score in 37 error types of 52 ESLint error types than TFix.
Cited by 2
Shahin Honarvar, Marek Rei, Alastair F. Donaldson · IEEE Transactions on Software Engineering · 2025
We present the concept of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">question neighbourhood</i> for systematically evaluating instruction-tuned large language models (LLMs) for code generation via a new benchmark, Turbulence. Turbulence consists of a large set of natural language <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">question templates</i>, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">test oracle</i> that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">neighbourhood</i> of very similar programming questions, and assess the correctness of the result returned for each question. This allows gaps in an LLM’s code generation abilities to be identified, including <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">anomalies</i> where the LLM correctly solves <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">many</i> questions in a neighbourhood but fails for particular parameter instantiations. We present experiments against 22 state-of-the-art proprietary and open-source LLMs, each at two temperature configurations. Our evaluation is based on three complementary scores: accuracy score, correctness-potential score, and consistent-correctness score. Our findings show that, across the board, Turbulence is able to reveal cases where LLMs do not behave in a correct and consistent manner, highlighting gaps in their reasoning ability. This goes beyond merely highlighting that LLMs sometimes produce wrong code (which is no surprise): by systematically identifying cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood, our method provides detailed insight into the behavioural characteristics of current code-generating LLMs. We present data and examples that shed light on the kinds of mistakes that LLMs make when they return incorrect code results.
Cited by 1
Andrew J. Zerngast · Holmes Museum Of Anthropology (Wichita State University) · 2010
Thesis (M.S.)--Wichita State University, College of Engineering, Dept. of Electrical Engineering and Computer Science.
Cited by 0Open access
Sabiya Banu Masthan Ali, Oussema Kirmani, Syed Muhammad Danish et al. · arXiv (Cornell University) · 2026
Large Language Models (LLMs) are widely used in software engineering to generate, complete, translate, and fix code, improving developer productivity. While most research focuses on the energy consumption and carbon emissions of model training and inference, far less attention has been given to the sustainability of the code these models produce. The efficiency of generated code affects the long-term environmental impact of software systems. Inefficient code can increase CPU usage, memory consumption, execution time, and overall energy use during deployment and operation. As LLM-generated code becomes more common in real-world projects, even small inefficiencies can lead to high environmental costs over time. This paper examines existing research on the sustainability of code generated by LLMs. We conduct a systematic literature review to analyze selected primary studies and investigate the extent to which LLMs are capable of producing sustainable code. In addition, we examine how sustainability is defined and measured in this context, including the metrics and evaluation strategies used to assess energy efficiency and resource usage. We also explore whether techniques such as fine-tuning and prompt engineering influence the sustainability of generated code. Through a structured analysis of the selected studies, we categorize research efforts based on their methodological approaches, evaluation practices, and experimental settings. The findings indicate that research in this area remains relatively limited and fragmented, with no widely accepted framework for measuring or benchmarking the sustainability of LLM-generated code. These observations highlight the need for clearer definitions, standardized evaluation methods, and systematic research to support environmentally friendly AI-assisted software engineering.
Cited by 0Open access
Imran Ali, Sajjad Hussain Rizvi, Syed Hasan Adil et al. · The Asian Bulletin of Big Data Management · 2024
Today we are experiencing rapid enhancements in software systems and their development. The software industry's demand for tools and techniques for software development, especially automatic and less time-consuming, is increasing daily. Software refactoring and code smell detection are now expanded from code-level changes to the architecture, model, and requirements restructuring. We are moving from an object-oriented paradigm towards cloud computing, web development and mobile application development and so much more. Therefore, code smell and refactoring techniques are talk of the town in various research communities in their objectives. In this paper we will study the existing tools and techniques, research progress by doing a systematic literature review in the field of code smell detection and software refactoring’s. We will also classify the existing research techniques, identify the trends in code smell detection and refactoring and try to highlight the gaps in the area for researchers.
Cited by 1Open access
Fernando Payón-Aguilar, Ángel J. Sanchéz-García, Juan Carlos Pérez-Arriaga et al. · 2025
Cited by 0
Manpreet Kaur, Dhavleesh Rattan, Madan Lal · Computer Science Review · 2025
Cited by 0
Li Li, Daoyuan Li, Tegawendé F. Bissyandé et al. · IEEE Transactions on Information Forensics and Security · 2017
The Android packaging model offers ample opportunities for malware writers to piggyback malicious code in popular apps, which can then be easily spread to a large user base. Although recent research has produced approaches and tools to identify piggybacked apps, the literature lacks a comprehensive investigation into such phenomenon. We fill this gap by: 1) systematically building a large set of piggybacked and benign apps pairs, which we release to the community; 2) empirically studying the characteristics of malicious piggybacked apps in comparison with their benign counterparts; and 3) providing insights on piggybacking processes. Among several findings providing insights analysis techniques should build upon to improve the overall detection and classification accuracy of piggybacked apps, we show that piggybacking operations not only concern app code, but also extensively manipulates app resource files, largely contradicting common beliefs. We also find that piggybacking is done with little sophistication, in many cases automatically, and often via library code.
Cited by 161Open access
Sana Ali, Giovanni Sileno, Tom van Engers · SSRN Electronic Journal · 2025
Cited by 0Open access
Mandana Vaziri, Joshua A. Tauber, Michael J. Tsai et al. · DSpace@MIT (Massachusetts Institute of Technology) · 2004
The Input/Output (I/O) automaton model developed by Lynch and Tuttle models components in asynchronous concurrentsystems as labeled transition systems. IOA is a precise language for describing I/O automata and for stating their properties. A toolset is beingdeveloped for IOA to support distributed software design and implementation. One of the tools consists of a userassisted code generator fromIOA into an imperative programming language such as C or Java. One aspect that distinguishes IOA programs from programs written inimperative languages is the presence of nondeterminism which comesin the form of explicit nondeterministic statements and implicit scheduling choices made during execution. Code generation therefore consistspartially of systematically removing all forms of nondeterminism. In this paper, we describe our approach and design for code generation.We focus on the issue of removing implicit nondeterminism and specify a transformation on IOA programs that makes all nondeterminismexplicit. The programmer can then replace all explicit nondeterminismwith deterministic statements prior to code generation. We also describethis transformation at a semantic level i.e., at the level of the I/O automaton mathematical model. We show that the transformation definedat the IOA level conforms to the one at the semantic level.
Cited by 2Open access
Haolin Jin, Huaming Chen · arXiv (Cornell University) · 2026
Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to verify if code implementation satisfy task requirements, thereby ensuring code robustness and accuracy. However, it remains unclear whether LLMs can reliably determine code against the given task descriptions, which is usually in a form of natural language specifications. In this paper, we uncover a systematic failure of LLMs in matching code to natural language requirements. Specifically, with widely adopted benchmarks and unified prompts design, we demonstrate that LLMs frequently misclassify correct code implementation as non-compliant or defective. Surprisingly, we find that more detailed prompt design, particularly with those requiring explanations and proposed corrections, leads to higher misjudgment rates, highlighting critical reliability issues for LLM-based code assistants. We further analyze the mechanisms driving these failures and evaluate the reliability of rationale-required judgments. Building on these findings, we propose a Fix-guided Verification Filter that treats the model proposed fix as executable counterfactual evidence, and validates the original and revised implementations using benchmark tests and spec-constrained augmented tests. Our results expose previously under-explored limitations in LLM-based code review capabilities, and provide practical guidance for integrating LLM-based reviewers with safeguards in automated review and development pipelines.
Cited by 0Open access
Myoungkyu Song, Eli Tilevich · IET Software · 2017
In modern web‐based applications, an increasing amount of source code is generated dynamically at runtime. Web applications commonly execute dynamically generated code (DGC) emitted by third‐party, black‐box generators, run at remote sites. Web developers often need to adapt DGC before it can be executed: embedded HTML can be vulnerable to cross‐site scripting attacks; an API may be incompatible with some browsers; and the program's state created by DGC may not be persisting. Lacking any systematic approaches for adapting DGC, web developers resort to ad‐hoc techniques that are unsafe and error‐prone. This study presents an approach for adapting DGC systematically that follows the program‐transformation‐by‐example paradigm. The proposed approach provides predefined, domain‐specific before/after examples that capture the variability of commonly used adaptations. By approving or rejecting these examples, web developers determine the required adaptation transformations, which are encoded in an adaptation script operating on the generated code's abstract syntax tree. The proposed approach is a suite of practical JavaScript program adaptations and their corresponding before/after examples. The authors have successfully applied the approach to real web applications to adapt third‐party generated JavaScript code for security, browser compatibility, and persistence.
Cited by 0Open access
Rare & Special e-Zone (The Hong Kong University of Science and Technology) · 2003
Cited by 0
Rania Ben Chekaya, Kamel Garrouch, Mohamed Nazih Omri · Research Square · 2026
Cited by 0Open access