article · 2008

Measuring the Effectiveness of Retrieval Techniques in Software Engineering

Jane Huffman Hayes, Alex Dekhtyar, Senthil Karthikeyan Sundaram

Cited by 0Abstract and details only

Abstract

Abstract. Mining textual artifacts is important for a large array of software engineering tasks: software reuse, software maintenance, software quality assurance, to name a few. Much of the work on mining software repositories has tended to “exclude ” such non-structured artifacts. At the same time, we find these items to be rich in semantic information and feel that mining techniques should treat text as software and address their efficient mining. We investigate the application of information retrieval (IR) techniques to the tracing of textual elements in the software repository. Some textual mining activities are very critical (e.g., tracing artifacts to assure satisfaction of safety requirements) and require analyst participation. We describe our approach to eliciting and processing analyst feedback for the tracing of textual elements of a repository. We then present a study that shows that standard IR methods combined with analyst feedback outperform IR methods alone in terms of coverage (recall- did we find all the relevant links?) and signal-to-noise ratio (precision- were the links we found relevant?). With the analyst “in the loop, ” it is necessary to ensure that the tracing software possesses quality from the perspective of the analyst. We examined standard measures for evaluating IR methods and found that they do not always suffice for examining a tool from the analyst’s perspective. To address this, we developed a set of secondary measures for evaluating the tracing software. We show, by counter-examples from two projects, that standard measures alone do not provide the detail necessary for adequately evaluating mining tools from the analyst’s perspective. Keywords:software repositories, data mining, software artifacts, requirements tracing, sec-ondary measures, analyst feedback, technique evaluation, Information Retrieval. 1

Cite this paper

APA 7

Hayes, J. H., Dekhtyar, A., & Sundaram, S. K. (2008). Measuring the Effectiveness of Retrieval Techniques in Software Engineering.

MLA 9

Hayes, Jane Huffman, et al. “Measuring the Effectiveness of Retrieval Techniques in Software Engineering.” 2008.

Chicago

Hayes, Jane Huffman, Alex Dekhtyar, and Senthil Karthikeyan Sundaram. 2008. “Measuring the Effectiveness of Retrieval Techniques in Software Engineering.”

Harvard

Hayes, J.H., Dekhtyar, A. and Sundaram, S.K. (2008) ‘Measuring the Effectiveness of Retrieval Techniques in Software Engineering’.

BibTeX

@article{hayes2008measuring,
  title = {{Measuring the Effectiveness of Retrieval Techniques in Software Engineering}},
  author = {Hayes, Jane Huffman and Dekhtyar, Alex and Sundaram, Senthil Karthikeyan},
  year = {2008}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.