A Survey of Bug Tracking Tools:

Presentation, Analysis and Trends

Trajkov Marko, Smiljkovic Aleksandar

markostrajkov@gmail.com aleksandarsmiljkovic@gmail.com

Department of Computer Science, University of Belgrade

Abstract

Nowadays, when project are so extensive, the one sure thing that will happen is

bag. Because of that, it becomes very important to have appropriate bug tracking

system. Poorly designed bug tracking systems can are sometimes blamed for bad

information exchanging between developers. The purpose of this paper is to

present some most common used such systems, to address his problems and give

some direction for enhancement.

1. Introduction

In the introductory section we specify the following three issues of importance: (a) Definition of the

general field of interest for this research, (b) Definition of the special terminology of interest for this

research, and (c) Definition of the viewpoint of interest for this research (definition of the axiomatic

requirements of this research).

Today, the using bug tracking system for tracking bugs and other issues is well spread. Bug tracking

systems are using for organizing and monitoring bugs. Without this systems, monitoring a large amount

of bugs will be impossible or very hard. Because of that, today you can find a lot of this bug tracking

systems. They very differ by quality, security, costs, and functionality they offer to the user.

We can observe bug tracking system as a part of much wider system for project tracking or as an

separate system. Both cases have the advantages and disadvantages but form this point of view we will

observe them as an separate systems. The reason for this is to reduce the scope of research and

therefore easier and more precise treat the problem. From the other side, the project tracking systems

are more tighter related to the specific problem.

Previous research has shown that items such as stack traces, steps to reproduce, observed and expected

behavior, test cases, and screenshots are often omitted by reporters [4, 6] and that items are preferred

information for developers. The effect of this delay is that bugs take longer to be fixed and more and

more unresolved bugs accumulate in the project’s bug tracking system.

The audience for this paper we can split into two groups. The first of them are individuals or companies

who work on projects which require some bug or issue tracking system. This paper should help them to

better understand the ground and chose appropriate tool for their system. The second group is

developers of the same. They should know the competition. Also this paper should help them to build

the system which can be better than others in the set of features and functionality, and the system

which can satisfy the larger population.

In the next of this document we present issues of importance for the understanding this paper (Section

2). Further, we present existing solutions of the problem and their criticism (Section 3). The paper

closes with trends and the optimal solutions for future bug tracking systems (Section 4) followed by

conclusions (Section 5).

2. Problem statement

In this section we focus on the following issues of importance for the understanding of the basic

orientation of this research: (a) Problem definition, (b) Elaboration of problem importance (why is the

problem important?), and (c) An assessment of the problem development trends (why will the

importance of this problem grow over time?).

Bug Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their

product effectively. Having complete information in the initial bug report helps developers to quickly

resolve the bug. So choosing a good bug tracking system for your product helps you to reduce

downtime, increase productivity, raise customer satisfaction, and improve communication between

developers.

Nowadays, modern projects become much more complex, robust and harder to keep it track with them.

Parallel with that, it is becoming much more harder to keep in tracks a huge amount of bugs. On the

other side, the projects passes through many phases such as alpha, beta, release candidate and through

many versions since first version. Because of that it is important to keep in tracks of all malfunctions

which are noticed in projects life circle.

The complexity of project will grow over time. The versions of existing projects will continue to increase.

The same bugs will continue to appear in every generation of developers. So, it is recommended to keep

in record all of this malfunctions to improve future software development.

3. Existing solutions of the problem and their criticism

In this section, we give a short overview of existing solutions of the problem specified in the Problem

Definition section, and we criticize the existing solutions from the point of their usability. In general, the

overviewed solutions give excellent results under the specific problem of interest, but they do not

address the general problem of interest for this research.

This section starts with an overview for each of selected solutions, gives the following main points: (a)

The basic information of each solution, (b) Specific details for each selected solution, (c) Further

development trends of the approach, and (d) A criticism of the solution, and finally (e) Possible

improvements that could overcome the noticed drawbacks.

It concludes with a classification of each elaborated solution. The classification criteria were chosen to

reflect the essence of the basic viewpoint of this research. It will summarize all important parts of

elaborated solutions.

3.1. Presentation of existing solutions and their drawbacks

This section is divided in several subsections, one per each solution. For each solution, several sentences

per paragraph are given.

3.2.1. Bugzilla

Bugzilla very popular, actively maintained and free bug tracking system, used and developed together

with Mozilla, giving it considerable credibility. Bugzilla is based on Perl and once it is set up, it seems to

make its users pretty happy. It's not highly customizable, but in a odd way, that may be one of its

features: Bugzilla installations tend to look pretty much the same wherever they are found, which

means many developers are already accustomed to its interface and will feel they are in familiar

territory. Bugzilla has a system that will send you, another user, or a group that you specify the results of

a particular search on a schedule that you specify. Bugzilla has a very advanced reporting systems and

you can create different types of charts including line graph, bar graph or pie chart.

3.2.2. Mantis

Mantis is a free web-based bug tracking system. It is written in the PHP scripting language and works

with MySQL, MS SQL, and PostgreSQL databases and a webserver. Mantis can be installed on Windows,

Linux, Mac OS and OS/2. Almost any web browser should be able to function as a client. It is released

under the terms of the GNU General Public License (GPL). The main complaint is its interface which

doesn’t meet modern standards. On the other hand, is easy to navigate, even for inexperienced users.

There not exists some advanced features such as charts and reports. In short, the whole

system is sloppily done, there are plenty of bugs and very little functionality.

3.2.3. BugTracker.NET BugTracker.NET is a free, open-source, web-based bug tracker or customer support issue tracker written

using ASP.NET, C#, and Microsoft SQL Server Express. BugTracker.NET is easy to install and learn how to

use. When you first install it, it is very simple to setup and you can start using it right away. Later, you

can change its configuration to handle your needs. It has a very intuitive interface for generating lists of

bugs. It has two very useful features. First of them is a screen capture utility that enables you to capture

the screen, add annotations and post it as bug in just a few clicks. The second feature is the fact that it

can integrate with your Subversion repository so that you can associate file revision check ins with bugs.

3.2.4. Flyspray Flyspray is a web-based bug tracking system written in PHP. Flyspray is free software, released under the

General Public License. This essentially means that you can get Flyspray and use it free of charge. The

source code is available, and everyone are welcome to modify it to suit their needs. Its web pages

describe it as "uncomplicated", and the list of features includes: multiple database support (currently

MySQL and PGSQL), multiple projects, 'watching' tasks, with notification of changes (via email or

Jabber), comprehensive task history, CSS theming, file attachments, advanced search features,

RSS/Atom feeds, wiki and plaintext input, voting, dependency graphs.

3.2.5. Redmine Redmine is a flexible web-based project management web application. Written using Ruby on Rails

framework, it is cross-platform and cross-database. Redmine is open source and released under the

terms of the GNU General Public License. Redmine is flexible issue tracking system. You can define your

own statuses and issue types. He support multiple projects and subprojects. Each user can have a

different role on each project. Interface is very simple, intuitive and easy to navigate. Shortly, this is very

good product and our recommendation.

3.2.6. Bug-Track Bug-Track is web-based defect and bug tracking software allows you to document, manage and assign

all of your bugs and tasks and empowers you to organize your bugs, defects or issues into distinct

projects. It can run on virtually any web-server like Microsoft, Linux, Unix, etc... Since it is an commercial

application it is expected that it is better than other free products. But it isn’t true. He has nothing new

and better than other free bug tracking systems. One better thing is fact that he have more intuitive

interface then others and that is his only benefit.

3.2.7. Bugzero Bugzero is a web-based bug, defect, issue and incident tracking software. Its single code base supports

both Windows and Unix (based on Java™) and supports database systems including Access, MySQL, SQL

Server, Oracle, and etc. Bugzero can be customized for software bug tracking, hardware defect tracking,

and help desk customer support issue and incident tracking. Bugzero have intuitive interface but he

lacks form features. The main drawback is the fact that Bugzero is an commercial product and you can

find much better product for free.

3.2.8. Conclusion about existing solutions

From all above presented, we conclude that, among the existing solutions, the none of them can be

treated as the best one, for the general solution of this research. Every of them have some advantages

and disadvantages. Some of them have some feature more than others but in the general, the set of

features are the same.

3.2. Classification criteria and the classification tree

For classification criteria for this research are selected the most common features that mosses most of

the selected bug tracking tools. This features are useful for an average user and they give an easy to use

bug tracking system. Decision making using this criteria is not good and some more advanced criteria

will be given in the next section (Section 4).

The classification criteria of interest for this research, are given in Table 1. All selected classification

criteria are explained in the caption of Table 1, and elaborated in the paragraph to follow. Such that

criteria are widely used in various systems that are in use today.

Search Email

notifications

Reports Charts Time

Tracking

RSS/Atom

Feed

Configurable Free

Bugzilla yes yes yes yes yes no no yes

Mantis yes no no no yes yes no yes

BugTracker.NET yes yes yes yes yes no yes yes

Flyspray yes yes yes no no yes yes yes

Redmine yes yes yes yes yes yes no yes

Bug-Track yes yes yes yes no yes no no

Bugzero yes yes yes no no no no no

Table 1: Classification criteria. Legend: search, email notifications, reports, charts, time tracking,

RSS/Atom Feed, Configurable, Free. Explanation: This table summarize criteria that are used in decision

making process of choosing suitable bug tracking system.

Table 1. summarize criteria that are used in decision making process of choosing suitable bug tracking

system. That criteria are: search, email notifications, reports, charts, time tracking, RSS/Atom Feed,

configurable (can system be configure its settings to meet client standards). The brief description of

each criteria are given next.

The classification tree, derived from the above introduced classification criteria, is presented in Figure 1.

Each leaf of the classification tree is given a name, as described in the caption of Figure 1. For each leaf,

the list of existing solutions is given.

Figure 1: Classification tree. Legend: Charts, Configurable, Free. Explanation: This tree summarize basic

criteria that are used in decision making process of choosing suitable bug tracking system.

Now, we can summarize all criteria given above in Table 1 and Figure 1. Search is very useful criteria and

it is present in all selected products. Email notifications gives user opportunity to be noticed about

happenings in the current bug tracking system. The user do not need to check frequently bug tracking

system for new changes. All he need is an email account. Reports give user a brief and concise overview

about past happenings in our system. Charts give clear graphical view of selected criteria which is very

intuitive to the human being. Time tracking is an feature that give information about happenings of

some specific bug trough time. Like an email RSS/Atom feed gives user opportunity to be noticed about

happenings in the current bug tracking system. Configurable system is capable to be configured to meet

certain user needs, so the system should be configurable as much as possible, because that will help to

satisfy the larger population of customers. At the end, the much important issue for choosing the right

bug tracking system is the fact is it free or not and how much he cost.

4. Trend and the optimal solution for future

When a user submits a bug report she is asked many questions: What is the name of the product? In

which plugin/component? What is the Build ID? What is the bug about? What are the steps to

reproduce the bug? Any additional information? However, the initial information provided in a bug

report is often incomplete and developers often have follow-up questions: Do you have flash installed?

Is there any screenshot? Getting replies by users takes time (often weeks) and slows down the progress

on open bugs.

We therefore propose to build expert systems that automatically ask relevant questions and gather all

required information once an initial failure description has been provided. The selection and order of

questions should not be static (as in current bug tracking systems), but rather depend on previous

responses. We believe that such an expert system can provide better bug descriptions.

The classification criteria of interest for this research, are given in Table 2. All selected classification

criteria are explained in the caption of Table 1, and elaborated in the paragraph to follow.

Stack

Trace

Steps to

Reproduce

Observed

behavior

Expected

Behavior

Test

Cases

Screenshots Dependencies

Bugzilla no no no no no no no

Mantis no yes no no no no no

BugTracker.NET no no no no no yes yes

Flyspray no no no no no no no

Redmine no no no no no no no

Bug-Track no no no no no no no

Bugzero no no no no yes no yes

Table 2: The better selection criteria for proposed approach. Legend: stack traces, steps to reproduce,

observed behavior, expected behavior, test cases, screenshots and dependencies. Explanations: This

table showing that all of today bug tracking tools lacks from some important properties.

The essence of this proposed system is presented in better bug description. As you can see form the

Table 2, the most of the selected criteria are not supported in current bug tracking systems. The current

bug tracking systems rely on some different selection criteria, which sometimes cannot give satisfying

results. So, we believe that including selection criteria like stack traces, steps to reproduce, observed

behavior, expected behavior, test cases, screenshots and dependencies will improve current bug

tracking systems.

The advantages of the improved set of criteria will give an user better information about current bug.

This will lead to faster locating of the current defect and to the faster respond to it.

5. Conclusion

Current bug tracking systems do not effectively collect all of the information needed by developers.

Without this information developers cannot resolve bugs in a timely fashion and so we believe that

improvements to the way issue tracking systems collect information are needed.

We summarized criteria that are used in modern bug tracking systems. Such criteria often doesn’t give

appropriate results in describing bug. So, we proposed an improved set of criteria that will give much

more satisfying solution for the current system.

This work can be important to the designers of the future bug and defect tracking systems. They should

know importance of selected criteria for describing bug because a well described bug will be easier to be

find and solved. Designing such an system will give us an answer to our assumption. That will confirm

our work and apply new ideas to the current bug tracking systems.

At the end we can notice that current bug tracking system are web oriented, and such that trend will be

continued. It is very important to design system that selection criteria for bug describing will meet such

trend.

6. References (annotated bibliography) [1] J. Anvik, L. Hiew, and G. C. Murphy. “Who should fix this bug?,” Proceedings of the ICSE-2006, 28

th

International Conference on Software engineering, 2006, pp. 361–370.

[2] J. Aranda and G. Venolia. “The secret life of bugs: Going past the errors and omissions in software

repositories,” Proceedings of the ICSE -2009, Proceedings of the 31st International Conference on

Software engineering (to appear), 2009.

[3] S. Artzi, S. Kim, and M. D. Ernst. “Recrash: Making software failures reproducible by preserving object

states,” In Proceedings of the 22nd European Object-Oriented Programming Conference, 2008, pp. 542–

565.

[4] N. Bettenburg, S. Just, A. Schroter, C. Weiss, R. Premraj, and T. Zimmermann. “What makes a good

bug report?,” Proceedings of the FSE-2008, 16th International Symposium on Foundations of Software

Engineering, November 2008, pp. 308–318.

[5] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. “Duplicate bug reports considered harmful ...

really?,” Proceedings of the ICSM-2008: Proceedings of the 24th IEEE International Conference on

Software Maintenance, 2008, pp. 337–345.

[6] S. Breu, J. Sillito, R. Premraj, and T. Zimmermann. “What developers ask about your bug?,” Technical

report, University of Calgary, December 2008.

[7] P. Fritzson, T. Gyimothy, M. Kamkar, and N. Shahmehri. “Generalized algorithmic debugging and

testing,” Proceedings of the PLDI-1991, ACM SIGPLAN Conference on Programming Language Design and

Implementation, 1991, pp. 317–326.

[8] P. Hooimeijer and W. Weimer. “Modeling bug report quality,” In Proceedings of the 22nd

International Conference on Automated software engineering, 2007, pp. 34–43.

[9] S. Just, R. Premraj, and T. Zimmermann. “Towards the next generation of bug tracking systems,” In

Proceedings of the 2008 IEEE Symposium on Visual Languages and Human-Centric Computing,

September 2008.

[10] A. J. Ko and B. A. Myers. Debugging reinvented: asking and answering why and why not questions

about program behavior. Proceedings of the ICSE-2008, International Conference on Software

Engineering, 2008, pp. 301–310.