Towards a Benchmark for Evaluating Design Pattern Miner Tools

Date:102/1/17Publisher:IEEE Software Maintenance and Reengineering, 2008. CSMR 2008. 12th European Conference onAuthor:Lajos Jen˝o F¨ul¨op¤, Rudolf Ferenc and Tibor Gyim´othy組員 :余世淇 ,游家瑋

Introduction

• Recovering design pattern usage in source code is a very difficult task. Several tools are described in the literature for this purpose, but there is little work invested in evaluating them. The main reason for this is the lack of an approved benchmark for these tools.

Architecture

• We use the well-known issue and bug tracking system called Trac [34] (version 0.9.6) as the basis of the benchmark. Trac is written in Python and it is an easily extendible and customizable plug-in oriented system.

Architecture

Architecture

Architecture

A. InitializationB. Pattern Matching processing on GPUC. Data Output

• True Positives (TP): true instances found by the tool (correctly).• False Positives (FP): false instances found by the tool

(incorrectly).• False Negatives (FN): true instances not found by the tool

(incorrectly).• The precision value is defined as TP/TP+FP , which means the

ratio of correctly identified instances with respect to all found instances. The recall value is defined as TP/TP+FN , which means the ratio of correctly identified instances with respect to all existing real instances.

Adding a New Tool• Upload. Uploading the results of a tool requires the name of

the tool, name of the mined software, the programming language of the software, source location, and information about the found design pattern instances in comma separated value (CSV) file format

Benchmark Contents

Conclusion and FutureWork• This paper presents work in progress, the benchmark is

operational, but will certainly need further development based on user feedback. A possible extension of the benchmark would be, for instance, a flexible solution for selecting siblings among the pattern instances. Currently, users can add comments to an instance to argue about the judgements. In the future, we want to extend this with categorizing the votes of a user to control the overrating of his own tool.

• http://www.inf.u-szeged.hu/designpatterns/