LIFT: A Legacy LIFT: A Legacy InFormation retrieval ToolInFormation retrieval Tool

Student: Kellyton dos Santos BritoStudent: Kellyton dos Santos Brito Advisor: Silvio Romero de Lemos MeiraAdvisor: Silvio Romero de Lemos Meira

Informatics Center Informatics Center -- Federal University of PernambucoFederal University of Pernambuco

C.E.S.A.R. C.E.S.A.R. -- Recife Center for Advanced Studies and SystemsRecife Center for Advanced Studies and Systems {ksb, srlm}@cin.ufpe.br{ksb, srlm}@cin.ufpe.br

2/50

OutlineOutline

Introduction

Key Developments in the field of Reengineering

Reverse Engineering Tools

LIFT Tool

Requirements

Architecture and Implementation

Usage Scenario

Case Study

Conclusions

IntroductionIntroduction

4/50

Software is key point of business

Business dynamics needs Software dynamics

Objectives

Low Costs

High Productivity

High Quality

Boehm’s analysis (Boehm 1999)

working-faster savings:

8%

working-smarter savings:

17%

work-avoidance savings: 47%

5/50

Software Reuse Initial ideas from McIlroy (1968)

Software reuse is the process of creating software systems from existing software rather than building them from

scratch (Krueger 1992)

Reusable Assets Products, Processes, Knowledge …

Reuse Aspects Processes, methods, environments, tools and non-technical

aspects

Based on these aspects…

6/50

RiSE ProjectRiSE Project

Understand Fundamental steps to

introduce reuse in companies

Technical and non-technical

aspects

7/50

One (of many) points is...One (of many) points is...

Legacy Systems

Well Tested, stable, low bugs and defects

A lot of embedded knowledge

Problems

Obsolete technologies, languages, tools and processes

Non useful documentation

Degradation due to maintenance operations

Few specialized people

Directions

Reverse engineer applications

Knowledge Reuse

Knowledge reuse from legacy systemsKnowledge reuse from legacy systems

8/50

ProposalProposal

This work defines the requirements, designs and This work defines the requirements, designs and implements aimplements a

tool for reverse engineeringtool for reverse engineering,,

aiming to aid system engineers aiming to aid system engineers

to retrieval knowledge from legacyto retrieval knowledge from legacy systemssystems,,

as well as toas well as to

increase their productivityincrease their productivity

in reverse engineering and system in reverse engineering and system understanding tasksunderstanding tasks

9/50

ContextContext

Key Developments in the Key Developments in the field of Software field of Software ReengineeringReengineering

11/50

The GoalThe Goal

To Understand

Concepts

Evolution

Approaches

Strong and Weak points

New trends

12/50

Reengineering Approaches Reengineering Approaches (Garcia 2005)(Garcia 2005)

13/50

Unresolved Issues Recover all system, as interface, design and database

Trace entire requirements from interface to database

Deal with large systems

Discover HOW programs work, not WHAT programs do

New Trends Aspect Orientation

Data Mining

Lack of effective tools to support Reverse Engineering

Reverse Engineering Tools: Reverse Engineering Tools: The StateThe State--ofof--thethe--art and art and PracticePractice

15/50

MotivationMotivation

Automate tasks

Existence of some of tools

They are useful in practice?They are useful in practice? Not in 2000 year

Muller et al. Reverse Engineering: A Roadmap

Proceedings of ICSE. Future of Software Engineering Track

Not NOWNOW Canfora & Penta

New Frontiers of reverse Engineering

Proceedings of Future of Software Engineering (FOSE’07)

“Despite the maturity of reverse engineering research, and the fact

that many pieces of reverse engineering work seem to timely solve crucial problems and to answer relevant industry needs, its adoption in industry is still

limited”(Canfora 2007)

16/50

Reverse Engineering ToolsReverse Engineering Tools

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Santanu Paul

SCRUPLE Tool

Müller et al.

Rigi Project

Software Understanding

Favre

GSEE: a Generic

Software Exploration

EnvironmentStorey et al.

SHriMP Views

Exploration Tool

Singer et al.

TKSEE Software

Exploration Tool

Zayour et al.

DynaSee Reverse

Engineering Tool

Schäfer et al.

SEXTANT

Software

Exploration Tool

Lanza

CodeCrawler

17/50

Reverse Engineering ToolsReverse Engineering Tools

Almost all of them shows a call graph

Each one implements its proper requirements set Some with explorationexploration capabilities Some with visualizationvisualization capabilities Some with cognitivecognitive capabilities

All of them highly user dependent: lack of automatic or semi-automatic code analysis

Lack of recover and traceability of entire system, from interface to database

Problems dealing with big systemsbig systems

18/50

Problems dealing with big systemsProblems dealing with big systems RiGI Call Graph (928 nodes, 4203 dependencies)

LIFT: A Legacy InFormation LIFT: A Legacy InFormation Retrieval ToolRetrieval Tool

20/50

LIFT RequirementsLIFT Requirements Based on Surveys, RiSE and Industry Experiences.

LIFT Functional Requirements:LIFT Functional Requirements: (FR1) Visualization of entities and relations

(FR2) Abstraction mechanisms

(FR3) High user interactivity

(FR4) Search capabilities

(FR5) User activities trace capabilities

(FR6) Metrics visualization support

(FR7) Recovery of the entire system (interface, design and database)

(FR8) Trace of requirements from interface to database access

(FR9) Possibility of semi-automatic suggestions

Existent Requirements

New Requirements

21/50

LIFT Non Functional RequirementsLIFT Non Functional Requirements

(NFR1) Cross Artifacts support

(NFR2) Extensibility

(NFR3) Integration with other tools

(NFR4) Scalability

(NFR5) Maintainability and Reusability Existent Requirements

New Requirements

22/50

LIFT ArchitectureLIFT Architecture

Pre Processing

Persistence Layer

Legacy

Code

Parser Analyzer Visualizer

Parser Cluster

Visualization

Patterns

Detection

Normal

Visualization

Understanding Environment

Paths

Visualization

Patterns

Visualization

Paths

CalculationCluster

Analysis

23/50

Implementation: Parser ComponentImplementation: Parser Component

Parser Module Parses NATURAL/ADABAS source code

First version developed by Pitang team

Uses C# technology

Integrated as a component

Pre-Processing Module Works with parser output

Store useful information in the database SQL ANSI

Performs the system slice

Deduction of database layer

24/50

Implementation: Analyzer ComponentImplementation: Analyzer Component

Call Graph Generation Paths Calculations

Full paths Minimal paths

Using Dijkstra shortest path algorithm Running time O(n.log n)

Cluster analysis Hierarchical Clustering Mark Newman's “edge betweenness clustering algorithm”

Running time O(k.m.n)

Patterns detection (second interaction) Text pattern detection Graph pattern detection Clone detection

25/50

Implementation: Visualizer ComponentImplementation: Visualizer Component

Based on JUNG: Java Universal Network/Graph Framework

Visualizations of call graph and Analyzer modules

Normal visualization

Cluster visualization

Paths visualization

Patterns visualization

Uses Polimetric-Views concept

26/50

Understanding EnvironmentUnderstanding Environment

Graphical interface

Integrate the other components

Areas for paths, graphs and details

Shows source code

Works with views concept

Isolate subgraphs

Allow comments

Views comments

Modules comments

Source code comments

27/50

LIFT Usage: Initial StepsLIFT Usage: Initial Steps

Parser and Organizer

Called by simple menu commands

28/50

LIFT Usage: Initial GraphLIFT Usage: Initial Graph

29/50

LIFT Usage: Isolating GraphsLIFT Usage: Isolating Graphs

30/50

LIFT Usage: Detecting ClustersLIFT Usage: Detecting Clusters

31/50

Case StudyCase Study

33/50

The ContextThe Context

Pitang Software Factory

Infra-structure

Experienced staff

Real demands for reverse engineering

NATURAL/ADABAS systems of a financial institution

Previous experience with reverse engineering: Almost 2 million LOC

Company time and budget constraints

Customer dependent

34/50

The DefinitionThe Definition

Goal:

To analyze the reverse engineering tool for the purpose of evaluating it with respect to its efficiency of the tool from the point of view of researchers and software engineers in the context of software reverse engineering projects.

Questions

Does the tool provides effort reduction in reverse engineering projects?

Is the tool scalable enough to be used in large projects?

Do the subjects have difficulties to use the tool?

35/50

The PlanningThe Planning

Method of comparison Comparison with two sibling projects

Same technologies: NATURAL/ADABAS

Same domain: Financial

Same customer

Same understanding process

Same number of participants

Similar engineers experience: more than 10 years

Different tools

Performed nearly same time (February-June 2007)

36/50

The PlanningThe Planning

Null Hipothesis

H0: µproductivity by LOC with previous tools > µproductivity by LOC using LIFT

H0: µproductivity by program modules with previous tools > µproductivity by program modules using LIFT

H0: µproductivity by recovered requirement with previous tools > µproductivity by recovered requirement using LIFT

The Projects

LIFT Project: 210 KLOC NATURAL/ADABAS

Sister projects: 65 KLOC and 131KLOC NATURAL/ADABAS systems of financial domain

37/50

The OperationThe Operation

Training

28 hours. 3 meetings with all staff, 3 lectures to the user and 2 days using the tool with previous projects data.

Subjects

One system engineer in each project

Costs

Planning: C.E.S.A.R

Operation: Pitang

Period

February-June, 2007

38/50

The Quantitative AnalysisThe Quantitative Analysis

Lines/Hour Productivity

66% higher than Project 1 and 41% higher than Project 2

Null Hipothesis:

H0: µproductivity by LOC with previous tools =>

µproductivity by LOC using LIFT

39/50

The Quantitative AnalysisThe Quantitative Analysis

Modules/Hour Productivity

12% higher than Project 1 and 127% higher than Project 2

Null Hipothesis:

H0: µproductivity by program modules with previous tools >

µproductivity by program modules using LIFT

40/50

The Quantitative AnalysisThe Quantitative Analysis

High Level Requirements/Hour Productivity

the same of Project 1 and 167% higher than Project 2

Null Hipothesis:

H0: µproductivity by recovered requirement with previous tools

> µproductivity by recovered requirement using LIFT

41/50

Increase Rate

0

0,5

1

1,5

2

Size (KLOC) Parse Time

(s)

Pre-

Processing

Time (s)

Minimal

Paths Time

(s)

Full Analysis

and Graph

Creation (s)

Rate Increase Rate

The Quantitative AnalysisThe Quantitative Analysis Scalability

LIFT project and “Project 2” evaluation

Pentium IV / 512MB Database Server x Dual Core 2 / 2GB Client

42/50

The Qualitative AnalysisThe Qualitative Analysis

Based on a questionnaire

Tool effectivity

Effort reduction of about 20%

Easy to locate system features and to generate system documentation

Weak Points

Delay to load the application (Full Analysis and Graph Creation)

Efficient Training Program

43/50

Case Study SummaryCase Study Summary

Questions

Does the tool provides effort reduction in reverse engineering projects?

YesYes

Is the tool scalable enough to be used in large projects?

YesYes

Do the subjects have difficulties to use the tool?

NoNo

44/50

Case Study Case Study –– Lessons LearnedLessons Learned

Training

Questionnaire

Management Commitment

Subjects Motivation

Parser problems

ConclusionsConclusions

46/50

ContributionsContributions

The survey about Reverse Engineering Tools

The Requirements of an Effective Reverse Engineering tool

The Tool for Reverse Engineering

Cluster approach

Minimal-paths approach

Scalability using database approach

The Case Study

47/50

ContributionsContributions

Brito, K. S.; Garcia, V. C.; Lucrédio, D.; Almeida, E. S.; Meira, S. R. L. LIFT: Reusing Knowledge from Legacy Systems, In the Brazilian Symposium on Software Components, Architectures and Reuse (SBCARS), Campinas, São Paulo, Brazil. August, 2007.

Brito, K. S.; Garcia, V. C.; Almeida, E. S.; Meira, S. R. L. A Tool for Knowledge Extraction from Source Code, 21st Brazilian Symposium on Software Engineering (SBES), Tools Session, João Pessoa, Paraíba, Brazil. October, 2007 (to appear).

Invited Paper

Journal of Universal Computer Science (JUCS), Special Issue: Software Components, Architectures and Reuse. April, 2008

48/50

Other Contributions during the CourseOther Contributions during the Course

Conferences Brito, K. S.; Alvaro, A.; Lucrédio, D.; Almeida, E. S.; Meira, S. R. L.,

Software Reuse: A Brief Overview of the Brazilian Industry's Case, In the 5th ACM-IEEE International Symposium on Empirical Software Engineering (ISESE), Short Paper, Rio de Janeiro, Brazil, 2006

Invited Talk Software Reuse: Brazilian Industry Case. In II Workshop para

Introdução do Reuso em Empresas de Desenvolvimento de Software (WIRE). Porto de Galinhas, Brazil. June, 2007.

JournalJournal Daniel Lucredio, Kellyton dos Santos Brito, Alexandre Alvaro, Vinicius

Cardoso Garcia, Eduardo Santana de Almeida, Renata Pontin de Mattos Fortes, and Silvio Romero de Lemos Meira. Software Reuse: The Brazilian Industry Scenario (to appear). Journal of Systems and Software, 2008

49/50

Future WorksFuture Works

Development of Patterns Detection and Visualization modules

Plug-ins for other input languages

Automatic documents generation

Metrics Extraction and Reports Generation

More Case Studies

To be a part of a complete framework for reengineering: Processes, Methods and Tools

50/50

ReferencesReferences

Boehm, B. (1999). "Managing software productivity and reuse." IEEE Computer Vol.(32), No. 9, p. 111-113.

McIlroy, M. D. (1969). "Mass Produced Software Components". NATO Software Engineering Conference Report, Garmisch, Germany, p. 79-85.

Krueger, C. W. (1992). "Software Reuse." ACM Computing Surveys Vol.(24), No. 2, p. 131-183.

Garcia, V. C. (2005), "Phoenix: An Aspect Oriented Approach for Software Reengineer(in portuguese). M.Sc Thesis." Federal University of São Carlos, São Carlos, Brazil, March/2005.

Müller, H. A., Jahnke, J. H., Smith, D. B., Storey, M.-A., Tilley, S. R. and Wong, K. (2000). "Reverse Engineering: A Roadmap". Proceedings of the 22nd International Conference on Software Engineering (ICSE'2000). Future of Software Engineering Track, Limerick Ireland, p. 47--60.

Canfora, G. and Penta, M. D. (2007). "New Frontiers of reverse Engineering". Future of Software Engineering (FOSE), IEEE Computer Society, p. 326--341.

LIFT: A Legacy LIFT: A Legacy InFormation Retrieval InFormation Retrieval

ToolTool

Student: Kellyton dos Santos BritoStudent: Kellyton dos Santos Brito Advisor: Silvio Romero de Lemos MeiraAdvisor: Silvio Romero de Lemos Meira

Informatics Center Informatics Center -- Federal University of PernambucoFederal University of Pernambuco

C.E.S.A.R. C.E.S.A.R. -- Recife Center for Advanced Studies and SystemsRecife Center for Advanced Studies and Systems {ksb, srlm}@cin.ufpe.br{ksb, srlm}@cin.ufpe.br