View
20.939
Download
5
Category
Tags:
Preview:
DESCRIPTION
ALM describes automated project management. It includes * Scrum based task management with issue tracking system. * Contiguous Build * Regression Test * V-Model based Test process * Defect management process
Citation preview
Practical Application Life cycle Management Framework
- DRAFT Version
Principle ConsultantByungwook Cho C.
(byungwook.cho@oracle.com)
AGENDA
• Realistic of software development• Application Lifecycle Management• Practical Application Lifecycle Management Framework• Delivery Practical ALM Framework• Strategy for delivery practical ALM Framework• Case study – TBD
<Insert Picture Here>
Realistic of Software development
Software development
What we want
justwell defined process, well structured managementIn-depth software”
Software development
But what’s real?
Realistic of developer •Additional Requirement•Changing Requirement •Meeting,reporting,education•Communication•Overtime job•Not tracked work
Realistic of manager •Always 99.999..% complete•Always overtime, but what is he doing? •When can I release?•What are risk items?•Wants know status of issue.•Task Priority
Realistic of software • A house of straw• Defects are discovered after freezing code• No testing for changing• Big bag integration
The three little pigs‘A house of straw’
<Insert Picture Here>
Many Dr. and genius developed many methodologies RUP,CBD,CMMI
Theories
So…
There are only few genius Theory is theory We need more realistic and practical methodology
But…
• Practical development methodology• Erich Gamma, Joel Spolsky, Kent beck, Andrew Hunt• Iterative & Incremental• Based on agile
• Difference between traditional methodology.• Traditional methodology assumes that there are no change
and no defect in each previous step. (eg. Waterfall model)• It assumes there are many potential defects, and focus on
finding the defects by frequent testing. • It assumes there may be requirement changes, and try to
accept the changes.• Focus on collaboration and communication
What is alternative solution?Practical Development
<Insert Picture Here>
ALM Application Lifecycle Management
ALM definition
Application lifecycle management (ALM) is the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.[1]
• Definition from Wikipedia
• Scope of ALM• Covers whole SDLC• Complex and getting theatrical
Current ALM market
• All of vendor uses ALM as a marketing message• McCave just has testing solution, but they said “We are ALM
company”• HP also has testing oriented solution, but they said “We
have ALM solution”
• There are few pure ALM players but..• Serena, Polarion etc already has matured ALM product.• But it is complex and hard to apply. (needs genius)
• We need more practical ALM framework.• Depending on teams’ maturity level.• Depending on customer’s mega process (methodology)• Depending on budget and time
Four pillars of ALM
• Process– Cover software life cycle from requirement analysis,implementation to release.
• Quality Control– Provide capabilities to assure software quality– For example : testing, short release
• Tracking– Provide visibilities and traceability of project activity.– Provide traceability between deliverable. For example traceability between
requirement to actual code• Collaboration
– Especially collaborate with Biz dept,.– Requirement management– Task management etc
ProcessQualityControl
TrackingCollaboration
<Insert Picture Here>
Practical ALM Framework
Practical ALM
• Lightweight, more easier and practical• Composed with open source tools and cheap popular
commercial tools– Open source tools Reduce learning curves– Less investment Free or cheap commercial tools
• Based on practical and agile methodology
Practical ALM use case scenarioDeveloper Perspective
Scenario• Developer goes to office in morning.• Run eclipse IDE.• In IDE, there are task list that are assigned to him.• He checks tasks with priority and pick up one.• Change a status of the task to In Progress• Check out code from source code repository.• After implementation, he implement unit test with testing framework.• Execute test with standard build script. It automatically generate test result, code coverage rate and
run code inspection. Code inspection shows illegal naming convention or potential bugs.• He fix the code and re-run the script.• He checks code coverage rate in eclipse IDE. After confirming the code coverage rate is meet target
rate. He commit the code with the task ID.• CI (Contiguous Integration) system get a notify that main source repository has been changed.• It check out the code and run build script.• Compile and deploy the code to staging machine.• Run a test case. If test is broken. It automatically sends a notification to the developer and project
manager. And then roll back the code in source repository and staging machine.• If the test is passed. it invokes code inspection and coverage analysis. If there is a problem (cannot
match up target test code coverage rate or some static bug.) it will also report the problem.• Developer notify that all of implementation has been done successfully.• He write a his work history in task management system thru eclipse IDE. and change a status
to Resolved• Project Manage get a notification the task is resolved. He check up a note of the task and task result
etc. And then change a status of the task toClosed• Developer come back home.
Practical ALM use case scenarioProject Leader/Manager perspective
Scenario• Project manager attend on requirement meeting with customer.• He summarize meeting minutes and extract tasks from that with customer's
agreement (or confirm)• He creates new tasks with priority.• He plans the task followed by project milestone.• He check up developer's load by using task management system to find who is
under lower load.And assign it to appropriate person.• Project manager estimate task implementation time with assigned developers.• Project manager adjust plan with schedule and estimated time.• During the project, project manager tracking the task status and everyday adjust
the tasks.• Project manager or senior developer checks project health status with unit
testing result and code coverage rate.• In addition senior developer concentrate testing in high code complexity area. It
is founded by code complexity analysis report from CI tool's daily build.• Project manager also manage risk by using project status report from task
management tool and reports from CI tools. And he check up high priorityOpened task every day to control risk.
Practical ALM use case scenarioProject Leader/Manager perspective
Scenario• Customer side PM come to office in the morning.• Execute web browser and go to ALM dash board page.• Each project has one page dash board. It shows how many tasks are
opened and closed for each milestone.How many pending tasks are there. How many critical tasks are opened.
• He just notify potential risk and project healthy status by using the dash board.
• Make a meeting with Project Manager or Project Leader to control the risks.
• If he want to track the critical issue (task). He subscribe the task. All of comment and change are automatically reported. So he doesn't need to make a meeting or make a phone call to check up progress of the critical task.
Modules
Extract Requirement
Task Management
Dashboard
Source code management
Contiguous Build
Standard IDE
Standard Build Script
Testing Process
Testing Framework
Static Testing
Defect Management
Wiki based doc mgmt
Forum based bi-directionalcommunication
Code review
•Project management •Provide development environment
•Assure quality •Leverage collaboration and communication
Module 1. Task Management
• OverviewCovers project management like- Project schedule management- Project resource (human) management- Requirement gathering- Provide KPI dashboard for management group
• Scope
TraditionalDevelopment process
ALM / Task Management Process(Scrum based customized for Enterprise)
•Manage major milestone•Define mega process
• Manage daily task• Delicate resource and schedule management
Module 1. Task ManagementExtract Requirement
• Process1. Extract requirement from
meeting with client2. Gather all of conversation
to meeting minutes 3. Summarize meeting
minutes and post it into internal wiki
4. Extract action items and get a agreement from customer.
5. And put that action items into task list.
• Useful tools– Google site : gather all
meeting minutes into one place with jump start
– Atalssian confluence wiki : cheap and powerful.
CustomerCustomer
PLPL
Meeting Minutes
Extract Action ItemEx)AI 1. Implement Logging with JMS (JIRA-101)AI 2. Evaluate Logging architecture in Test (JIRA-102)
Extract Action ItemEx)AI 1. Implement Logging with JMS (JIRA-101)AI 2. Evaluate Logging architecture in Test (JIRA-102)
Task
1.Log whole of meeting1.Log whole of meeting
2. Extract action item in end of meeting.Action item should be agreed between customer and PL
2. Extract action item in end of meeting.Action item should be agreed between customer and PL
WIKIWIKI
MeetingMeeting
Task List or Task Management System
Task List or Task Management System
Register action item to TaskRegister action item to Task
ProcessProcess
Module 1. Task Management Extract Requirement
• Sample
Action items
Task number in task listIt provide traceability from requirement to task
Module 1. Task ManagementTask Management
• Overview• Managing schedule and task of project team• Agile scrum methodology based.• Subset of project management methodology
– Mega process is up to traditional methodologies or customer’s methodologies.
– Cover delicate daily task management only
Module 1. Task ManagementTask Management
• Scrum methodology
• Original scrum methodology is not appropriate Enterprise project management• Hard to manage risk.• Customer usually has their own methodology.
• Need to customize for Enterprise project
Module 1. Task ManagementTask Management
• Task Management Process1. Prepare product backlog It means requirement list
2. Release Planning – Define major milestone and release
3. Sprint Planning – Planning task for each iteration
4. Track Sprint – Daily tracking progress. Eg. Burn down chart
5. End Sprint – Simply provide demo to customer and get feed back
6. Update Sprint backlog – Update and re priories requirement
7. Retrospective – Review process and mature it
Module 1. Task ManagementTask Management
• Task Management ProcessProcess
Duration 4~6 weeksDuration 4~6 weeks
Requirement Use case
(Sub) RequirementTask
Task
Task
AnalysisDesign
AnalysisDesign
ImplementationImplementation
TestingTestingWorking Item
Documentation
Meeting
TestingInternal ReleaseQA has to test every internal release based on test plan
Internal ReleaseQA has to test every internal release based on test plan
General Available(Release)
General Available(Release)
High RiskMandatory FunctionsNon-Functional RequirementMajor UI
High RiskMandatory FunctionsNon-Functional RequirementMajor UI
Low RiskOptional FunctionsEnhancement
Low RiskOptional FunctionsEnhancement
Scheduling(Before implementation phase)
Scheduling(Before implementation phase)
WikiWiki
Task Management SystemTask Management System
Tracking in WBS (Link to MS Project)Tracking in WBS (Link to MS Project)
Use numbering in title to track of RequirementUse numbering in title to track of Requirement
Task
Task
TaskTask
LINKLINK
ReleaseRelease
Sprint Sprint Sprint
Requirement
Requirement
Requirement
Requirement
Requirement
Requirement
Meeting minutes
OptionalDepends on requirement granularity
OptionalDepends on requirement granularity
Module 1. Task ManagementTask Management
• Task Type– Task is real working item to implement requirement– Task has to have traceability to requirement– Task type definition is very important.– Task can have hierarchy.
System Task Sub Task Description
Requirement N/A Detailed functional requirement.Requirement cannot contain any child requirement. It is leaf node of requirement tree. Name of requirement has to have number to provide traceability.Ex) 2.1.1 Logging inbound transaction
WorkingItem Working items for developer. Eg. Coding,debugging etc.
Documentation Write deliverables as document on wiki or document
Test Develop and execute test of “Requirement”. It includes implementing Unit Test,design, execute etc.
Meeting Meeting about specific issue
Bug N/A Bug reported by somebody(not a author) after every internal release phase.
[!] Requirement can be separated into more detail sub requirement
Module 1. Task ManagementTask Management
• Workflow– Workflow defines task handling process.– It is very important for task management– Keep it simple and easy to understand– It needs to define task type and hierarchy
Module 1. Task ManagementTask Management
• Workflow- Requirement management process
Open Scheduled In Progress Resolved Closed
Customer decide to close the child requirement based following decision factor. 1. Code coverage rate2. Code inspection result3. Deliverables4. Unit Test Result
Customer decide to close the child requirement based following decision factor. 1. Code coverage rate2. Code inspection result3. Deliverables4. Unit Test Result
PLPL
PLPL
PLPL
PLPL
CustomerCustomer
Based on technical use case.Extract Child requirement and
register it into JIRA
Based on technical use case.Extract Child requirement and
register it into JIRA
Scheduling child requirement based on iterative development. It is linked to WBS in this phase
Scheduling child requirement based on iterative development. It is linked to WBS in this phase
In Every iteration, when the requirement related sub-task is started, change status to In Progress
In Every iteration, when the requirement related sub-task is started, change status to In Progress
After finishing all of sub tasks for the child requirement. Change the status to “Resolved”
After finishing all of sub tasks for the child requirement. Change the status to “Resolved”
Module 1. Task ManagementTask Management
• Workflow- Task management process
Open Assigned In Progress Resolved Closed
PLPL
Need more
Information
Postponed
PL creates detail task for developer.Task is usually finished 1~4 day. (1~2 days are recommended)
Discuss with developer and estimate time.Assign the time to task
PL creates detail task for developer.Task is usually finished 1~4 day. (1~2 days are recommended)
Discuss with developer and estimate time.Assign the time to task
DeveloperDeveloper
DeveloperDeveloper
DeveloperDeveloper
DeveloperDeveloper
PLPL
PLPL
Developer discuss with PLPL lets developer to postpone the sub task.The postponed task should be finished in same iteration phase. Or closed
Developer discuss with PLPL lets developer to postpone the sub task.The postponed task should be finished in same iteration phase. Or closed
Module 1. Task ManagementTask Management
• Mega Process Summary
Module 1. Task ManagementTask Management
• Implementation– Jump start with Microsoft Excel
- Product back log
- Sprint back log
Module 1. Task ManagementTask Management
• Implementation– Build task management system with Atlassian product
JIRA : for task management (2400 USD for professional version)
Confluence wiki : for requirement management (2200 USD for 50 user)
zAgile portal : Integrate distributed view MS project : Manage schedule by PMO perspective
Reference Architecture
http://www.atlassian.comhttp://www.zagile.com
Module 1. Task ManagementTask Management
• Implementation– Build task management system with VersionOne product
VersionOne provides hosted service. (No install is required, just use it)
It is based on agile methodology. (Atlassain JIRA is started from bug tracking system)
Sometimes hard to integrated with testing framework, CI tools, SCM etc.
http://www.versionone.com
Module 1. Task ManagementDashboard
• Overview– Provide personalized view for each stakeholder– Show KPI for management group (Very important to convince
manager)
Progress Risk items Velocity of team Workload for resources Escalated issues
– Has to be integrated with other ALM modules Test Result Build Status Test coverage rate
– Just use only meaningful KPI“Patterns in our team velocity and temperature of Seoul are
same..?! “
Module 1. Task ManagementDashboard
• View For project manager or customer
– Burn down chart– Task or Bug open/close progress– Risk item list– Escalated task list
For project leader or manager in small project– Build status– Test result and test coverage rate– Workload of team members
For developer– Assigned task– Project status (Burn down chart, build status) / To
understand whole of project status as a project team member
Module 1. Task ManagementDashboard
• Implementation
Task Management System
Contiguous IntegrationSystem
Testing Framework
DashboardOPEN APIFILE
PORTALOR
WIKIOR??
•Personalization•Integrated view
Module 1. Task ManagementDashboard
• ExampleDashboard with Atlassian JIRA
Number of task by priority
Number of task by status
Progress of each iteration
TASK CREATE/CLOSE TRACKING
Workload of team members
Module 1. Task ManagementDashboard
• ExampleDashboard with Inflectra/Spira Team
Module 2. Build EnvironmentSource Code Management
• Source Code Management (SCM)• Share source code• Track change of source• Manage branch
Module 2. Build EnvironmentSource Code Management
• Branch management strategy
Tagging
Branch
Tagging every build
Every release
Trunk(main branch)
•Merge heuristically from Release Branch.•Merge systemically from patch branch.
Module 2. Build EnvironmentSource Code Management
• Branch management strategy• Release branch
• This branch is created when software has been released. (Even if it is internal release,manage branch)
• Bug fix and enhancements of this version are applied to this branch.
• Bug fix and enhancement of this version are merged to trunk branch by heuristically. (If released versions are different, logic or architecture can be changed, so not to use system merge)
• Working branch• When some co-work is required and it will not be commited during
a day, make a working branch.
• Enhancement or patch branch• To make temporary patch or evaluate enhancement, make patch
branch before applying fix to trunk branch.• It can be applied by using simple system merging.
• Making a new branch provides additional managing effort.
Module 2. Build EnvironmentStandard IDE
• Standard IDE• Enforce all of developer to develop on same
environment- Reduce error from environment
• Save time to set up the environment• Process
1)Developer download standard development environment from wiki
2)Just unzip the file to personal computer.3)Set up personal preference (server ip, directory
structure etc )4)And followed by instruction in wiki, quick start to
develop
Module 2. Build EnvironmentStandard IDE
• Standard IDE Packaging
IDE
JVM
Test Framework Std build script
JEE Server
Property file
Install Guide
Development Framework
Zipped achieves
Module 2. Build EnvironmentContiguous Build• CI Process
Check out source
Build
Packaging
Deploy
Testing& coverage anaysis
Inspect code
Roll back code
Tagging
Reporting
Send Notification
Build Failed
Test failed
Tagging current version andRoll back SCM to previous tagging version
SCM CommitEvent
Scheduled time
EventTrigger
Send SMS/Email/IMS
Module 2. Build EnvironmentContiguous Build
• System configuration for CI
ContiguousIntegration Build Script
Testing Framework
SCM
Notifier
Check out codeTagging orRollbackwhen build fail
Invoke
Send notification message when build or test failed
Use
Module 2. Build Environment Contiguous Build
• Jobs for CI
Build Task Description
Compile Only compile
Packaging Packaging archives
Deploy Deploy achieves to server
Testing Run Testing
Testing and coverage analysis
Run Testing with code coverage analysis
Code Inspection Run Code Inspection
Full build Compile Packaging Deploy Testing with coverage analysis Code Inspection
Job Description
Full build Compile Packaging Deploy Testing with coverage analysis Code Inspection
Module 3. Testing AutomationOverview• Overview
• One of most important objective in ALM is guarantee quality of software
• Test Automation Module is designed for cover whole test cycle of V-Model.
• Unit/Integration tests are covered by development team and System/UAT are covered by QA team. Followed by your role , you can customize this process and framework
• Theoretical background• Mega process is based on ISTQB testing principals.• Unit testing and automated regression testing is
based on Practical methodology
Module 3. Testing Automation
• Testing Model– V-Model : Enhancement of waterfall model– Verification & Validation
Verification Validation
To verify the artifacts that has been produced in each development cycle.
Valid & estimate the system
Review & Inspectionwith artifacts from each development step
Testing with system
Static testing Dynamic testing
< Verification & Validation >< V-Model >
Module 3. Testing Automation
• Testing LevelLevel Description Testing Type Responsibility
Unit Test Verify software component White box test Lead by Dev Team Use Code coverage
Integration Test Verify integration between component. Verify software flow ,interface & interaction
White box test Lead by Dev Team Use Mock Object (Test Stub ,Driver)
System Test Test system over production environment to verify system itself and production environmentInclude Functional & Non Functional Test (Availibility, Stablity, Extendability, Performance etc)
Black box test Lead by QA Team(Specialized for system test)
Acceptance Test Verify customer’s requirement (End User)Verify customer’s Legal issue (Legal)Verify customer’s maintanance issue (SM)
Black box test Lead by QA Team or Customer
Alpha TestBeta Test
※ Integration strategiesTop down Bottom up Big-bang Backbone
Desc Integrate from top(use test stub)
Integrate from bottom(use test driver)
Integrate all in time Most important & high risk component first
Adv easily find architectural defect
Can test implementation logic
Short time
Ex. BPM SOA svc Small project
Module 3. Testing Automation Overview
• Test Cycle– Each test is consists of test cycle– Test cycle
Pre test - check up testability Main test Conformation test - check up the defect found in
previous “Main test” Regression test - check up the impact from change
Pre Test Main Test
ConformationTest
RegressionTest
TEST CYCLE
Module 3. Testing Automation Testing Process
• Mega process of Testing– Based on ISTQB testing model
Define test scope
objective
Risk anal
Set up strategy
Estimateresource & time
Planning
PRE TEST
TEST
Monitoring
Report
Module 3. Testing Automation Testing Process
• Step 1. Risk Analysis– Risk = Likelihood * impact– Risk identification
– Recommend < 36 identification.– Likelihood : Complexity, Implementation technical Level,
Size,Developer skill– Impact : Biz impact
Likelihood Impact
factor factor factor factor
Risk item
Risk item
Likelihood
Impact
STA
(Severe Test Area)
SSTA
(Strong Test Area)
FTA(Fundamental
Test Area)
ITA
(Intensive Test
Area)
Erik van neneendaal, Risk Based Testing, STAREAST,2006
Module 3. Testing Automation Testing Process
• Step 2. Define test strategy– Risk based test strategy
– Other strategies• Analytical approaches• Model based approaches• Methodical approaches• Dynamic and heuristic approaches
Likelihood
Impact
STA
(Severe Test Area)
SSTA
(Strong Test Area)
FTA(Fundamental
Test Area)
ITA
(Intensive Test
Area)
Likelihood
Impact
STA
(Severe Test Area)
SSTA
(Strong Test Area)
FTA(Fundamental
Test Area)
ITA
(Intensive Test
Area)
( Low Level Test) ( High Level Test)
- Unit Test
- Integration Test
- System Test
- Acceptance Test
Module 3. Testing Automation Testing Process
• Step 3. Estimate resource and time
Module 3. Testing Automation Testing Process
• Step 4.Test planning & control– Define activity of each testing level– Define milestone ,resource ,schedule.– Make a plan based on analyzed risk factor– Define test strategy based on risk– Define approach & techniques for testing
(testing techniques, coverage, test item, test ware)– Must include time & resource for preparing testware– Define completion condition
Module 3. Testing Automation Testing Process
• Step 5. Test analysis and design– Test analysis
• Review - test basis, testability• Identify test requirement & test data• Identify test infra and tools
– Test design• Identify test condition• Test case specification• Define test procedure
Module 3. Testing Automation Testing Process
• Step 5. Test analysis and design (Technique)
Design technique•Specification based
•Equivalence partitioning, Boundary value analysis
•Pairwise, Decision table testing
•State transition testing, Usecase testing
•Structure based
•Control Flow Test
•Basic Path Test – Based on Cyclomatic complexity (Edge – Node + 2P) P = number of connected component
•Elementary comparision test – Based on MC/DC Coverage
* Coverage : Statement, Condition, Decision, Condition Decision Coverage, Modified Condition/Descision Coverage(MC/DC), Multiple Condition Coverage
•Experience based
•Exploratory Testing – Adhoc testing, based on test charter, define timezone
•Classification Tree method
•CheckList
Module 3. Testing Automation Testing Process
• Step 6. Execute Test– Execution
• Define serverity & priority• Develop TC,Test data,Test stub & driver,Test
script,Test suite
– Reporting• Logging expected result, real result• Current status• Time & resource usage
– Defect tracking• Use Bug(Issue) tracking system
( Mantis,Bugzilla,JIRA,Trac) can be integrated with Task Management module in ALM.
• Define tracking workflow
TESTDRIVER
TESTSTUB
TARGET SYSTEM
TEST CASE
TESTINGTOOL
TEST DATA MODULE
MODULE MODULE MODULE
TEST SCRIPT
TEST SUITE1
TEST SUITE2
TEST
SIMULATOR
Test Execution
Test Report Defects
Issue Tracking System
Progress Report
Test logs
Module 3. Testing Automation Testing Process
• Step 7. Evaluate exit criteria and reporting• Test exit Criteria
• Coverage rate, Duration, # of unresolved defect
• Report Test Result• Progress report• Release advice• Final report per testing level
• Matrix• Defects per KLOC, Days test effort per requirement,
Unresolved defect impact analysis, Defects / Hour ,Cumulative defects / Day ,Test coverage / Day , Risk vs number of TC
Module 3. Testing Automation Testing Process
• Step 8. Testing closure activities• Evaluating
• Evaluate test target• Evaluate test process for maturing next text
• Final reporting• Store test-ware
Module 3. Testing Automation Testing Framework
• Testing framework– Tools and environment to support executing testing
Unit Test FrameworkUnit Test Framework
Load TestingTools
Load TestingTools
UI basedAutomated
Testing
UI basedAutomated
Testing
MonitoringMonitoring
Test caseManagement
Test caseManagement
Defect Management ToolDefect Management Tool
Contiguous Integration ToolContiguous Integration Tool
Unit Test Integration Test System Test UAT
RegressionTest
RegressionTest
RegressionTest
Report Defects
TestcaseManagement
Execute Test
Module 3. Testing Automation Testing Framework
• Unit Test framework
User Interface
BusinessLogic
DAO
Interface(RMI,REST,WS*)
xUnitxUnit
for DBTestingxUnit
for Protocol or message Type
xUnitfor UI
In Container Testing Tool
Code Coverage Analysis Tool
Test Test Test Test
Ap
plicati
on
Un
it T
est
fram
ew
ork
Support J2ee incontainerTesting
Validate test itself
Module 3. Testing Automation Testing Framework
• Load Testing• Support non-functional testing.• Generate load.
• Monitoring• Application Performance Monitoring Tools• Used for finding bottleneck in performance. • Eg. Jennifer(Korea),Wily Introscope. (different from Profiler)
• UI Based Automated Tool• Let tester to free from factory job.• Record user activity and replay it.• Validate expected action.• It is used for user acceptance test. (Especially functional testing)※ Test case can be reused in Stress testing.
• Test case management
Module 3. Testing Automation Static Testing
• Static testing techniques• Not running code, review code statically.• Code Review - Manual Review is done by human• Static Analysis - Static review is done by code inspection
tools
Static TestingStatic Testing
By humanBy human
By Automated toolBy Automated tool
Code ReviewCode Review
Static AnalysisStatic Analysis
Automated inspection toolsAutomated inspection tools
• Code Review• Introduced by Fagan as Code Inspection.• Developed to more vary Code Review.• Very effective and low cost action to keep code quality. • Kind of Code Review
• Code Inspection• Intend to find a defect. Executed with formal process and team. (For more information. See Fagan’s Code inspection)• Team has four roles. (Mediator – project manager, Reader – understand system to be inspected, Designer/Coder – Inspect
and generate recommendation, Tester – Test inspected defect and validate) And Six steps in process.• To execute code inspection, you need to have formal inspection team in your organization or bring a consulting who has
highly trained. • Performance of inspection is poor. But effect is very nice. Code Inspection is done after implementation. Recommend to
use code inspection in system testing phase.
• Team Review• Less formal than code inspection. But it has process also.• Mediator manage Team review session, and code author explains about his code.• Find a defects or gather idea for enhancement and write it to code review report. (Recommend to
use wiki)• The ideas and defects are handled as Task in ALM/Task Management • Recommend one time per week. Or after testing, after internal releasing, after sprint.
• Walkthrough• Informal review. It can be used to share idea.• Only author has a rule that explains his code. Nobody has a responsibility.• All things are up to the author
• PeerReview• Review code on a desk. • Usually 2~3 people can attend on review. Review code over a monitor and share idea.• It is nice to educate junior developer or who doesn’t have experience about product or skills.
Module 3. Testing Automation Static Testing
Module 3. Testing Automation Static Testing
• Static Analysis• Test code or diagram with “static analysis tool”• Analysis
• Can find syntax error or well known bugs• Validate naming rules• Can find dead code or missing parameter• Can analysis code complexity & dependency
• Quality?• Efficient but not so smart• It need a rule and need a maturing
• Easy to appeal your manager• Tools
• Open source – PMD, Find Bugs,Jdepend etc.• Commercial – Klockworks, Jtest etc.
Module 3. Testing Automation Defect management
• Defect management Process
DefectManagement System
Call coordinator
Validator Project manager
Developer
QA Engineer User
1.Create case
Filter
•Summary•Description•Log and screen shout•Reproducible case
2.Yank& validate
2-1 Request more information
3.Assign 5.Assign
4. Priories and schedule
6.Fix (Workaround or patch)
7. Provide Solution
8. Confirm
Source codeManagement system
9. Commit to main branch with Defect #
Defect Reporter
Implementation Team
Resolved defects list is combined to release note
Module 3. Testing Automation Defect management
• Defect management system• Defect management system is integrated with SCM to track change
of source code to fix the defect.• Web based SCMbrowser shows changes for defect in a web from
Defect management system link• Task Management system can be used as Defect management
system as well.• In Big project or if there are so many defects are reported then
separate defect management system and integrate it into task management system. It needs to separate organization to implementation and defect handling team
DefectManagement System
Source codeManagement system
Web basedSCM browser
Track changes
LinkShows change of code
TaskManagement System
Defect is task
<Insert Picture Here>
DeliveryPractical ALM Framework
Aapproach to delivery ALM solution
• There is 3 perspective to delivery ALM.
Process
Analysis Design Implementation Testing
UnitTesting
Integration
Testing
SystemTest
UAT
Task Management Process (Traditional Methodology)
Task Management Process (ALM/Scrum based)
Task Management Process (ALM/Scrum basder)
Build Process
Test Process
SCM Process
Test Process
Defect Mgmt Process
Task Management Process (ALM/Scrum based))
Developer
Management(PM/PMO/PL)
QA
DevelopmentLifecycle
Testing Model(Based on V-Model)
Agile/Scrum Base
Practical methodology base
V-Model Based
Reference ArchitectureALM architecture overview
• Reference architecture describes what we need to build up ALM system with tools and software.
• Layout of ALM architecture
Task Mgmt
ContiguousIntegration
Testing Framework
Build Script
Source CodeManagement
IDE
Build Script
Testing Framework
Defect Management
Testing Framework
(System& UAT)
Dashboard
Build Status Test Result
Task Status
Defect Status
Check out
Check in
Management(PM/PMO/PL)
Developer
QAAssign Task
Wiki Provide guide & processManage document.
Reference ArchitectureTask Management.
• Atlassian product is very popular, so easy to get a support and get a resource who can use it.
• Atlassian has CI tools, SCM connector, Coverage analysis tools,,Wiki etc. So it provides strong integrity.
• But, origin of JIRA is for Bug Tracking System not a Agile based task management system. It needs a customization in process and tool.
Atlassian Confluence Wiki
Atlassian JIRA
zAgilePortal
MS Project
Requirement Management Task Management Major Milestone managementfor mgmt org. (eg. PMO)
Personalization, Integrated view
Scrum based task management process
Scrum in JiRA : http://www.slideshare.net/paulrene/adapting-jira-for-scrum-presentation
Reference ArchitectureTask Management – Alternative solution “Spira Team”
• SpriaTeam has great agile approach. It already contains concept of Release,Iteration,Requirement,Task,Test..
• Most excellent thing is it provides traceability from Requirement-Task-Test. It enables track whole things in manner of ALM.
• It also provides some connector to integrate with issue tracking system (JIRA,Bugzilla)
• But it doesn’t support integration with CI tools or coverage analysis tools . It should be implemented with other product.
• Company looks new and small, so they seems like have few reference yet. But cheap..
Atlassian Confluence Wiki
InfrectraSpira Team
Requirement ManagementRequirement Management, Task Management, Schedule mgmt& Test case management
Scrum based task management process
LINK
Reference ArchitectureTask Management – Another alternative
• VersionOne (http://www.versionone.com)- Agile Oriented- They provide hosted task management tool and sell it as package.- Well support scrum methodology. - Free license for one small team for evaluation. Try it…
• Rally Enterprise (http://www.rallydev.com) - Agile Oriendted- They provide hosted (Saas) Task management tool.- It is based on Agile methodology. And they have great agile knowledge.- But it is not inconvenient for my practical ALM approach. (It doesn’t mean their product is
poor)- If you want to get a deep consulting about Task management process, I want to
recommend this one.
• Serena (http://www.serena.com)- It is traditional pure ALM player.- It is very popular and covers big area of ALM.- It needs high level team maturity. - I just want to recommend this one to Enterprise who really willing to invest ALM and
Governance.
• Polarion (http://www.polarion.com)- It is also very powerful ALM vendor. - Same as serena, it covers whole of ALM scope and IT governance. It supports CMMI level
as well and has very rich capabilities.- There is light version- Polarion Track & Wiki (Issue tracking + Wiki + light build mgmt +
Eclipse Plug in)
• Codebeamer
※This evaluation of ALM tools is up to my personal opinion.
Reference ArchitectureTest Automation
JUnit
EasyMock Cactus
Selenium DBUnit XMLUnit
Japex
Basic testing framework
Mockup test J2EE In container test
UI Test DB Test XML Test
Junit based stress test
Cobertura
Test Code coverage analysis
SourceForgeGrinder
Stress Test
SOAPUI
Stress Tes t forWebservice,REST
Find Bugs
Code inspection
AtlassianJIRA
Defect tracking
Functional Testing Framework(Unit testing/Integration Testing)
Non functional testing(System Testing)
Static Testing Defect mgmt
V model
Alternative•HP Load Runner (Commercial)•Oracle ATS (Commercial)•Apache Jmeter (OpenSource)
Alternative•PMD (Open Source)
Alternative•Atlassian Clover (Commercial)
Alternative•Mantis (Free)•Bugzilla (OpenSource)
Reference ArchitectureTest Automation
• Design principal of Test framework is to support automated testing and regression test.
• Test framework covers functional testing includes unit test and integration test. (Integration test can be viewed as coarse grained unit test)
• Test framework combines – Functional testing capabilities followed by characteristics of component like DB Testing,UI Testing,XML based messaging and Unit based stress test, coverage analysis and mock up test environment etc.
• This test framework covers broad range of testing. It means it is not easy. (concentrated on Unit testing and Integration testing. Partially covers System test and UAT. I recommend to use more matured tools for System test and UAT)
• ROI of code inspection is very high. How to managing rule set is most important.
• Optionally, I recommend to set up test case management tools from QA team. (ex. Testopia)
Reference ArchitectureBuild Environment
• Apache ant will not be developed anymore, Maven will replace ant. Maven also support unified library management.
• Hudson is now most popular free CI tools.- It supports a lot of plug ins for JIRA,Junit,Find Bugs etc.- It is very easy to learn.
Functional TestingFramework
Test framework
Maven
Standard Build Script
Subversion
Source code Management
Eclipse
Standard IDE
Hudson
Contiguous Integration
Practical methodology (Joel Spolsky,Kent Beck, Erich Gamma)
Alternative•Atlassian Bamboo
Reference ArchitectureCollaboration
• Just provide minimum collaboration environment.• Atlassian confluence provides a lot of plug in. Like MS-
Office connector etc. It is very useful to sharing your team knowledge.
• If you already have collaboration infrastructure in your company, do not extend product.
AtlassianConfluence Wiki
Document ManagementKnowledge Transfer
JForum
Bi-directional knowledge exchange
Reference ArchitectureDevelopment environment
• In staging environment, highly recommend to use hardware virtualization (CPU,Memory). It is very useful to validate system capacity. (Scalability test result is be basis of capacity planning)
DevelopmentEnvironment
Staging Production
Responsibility Development Team QA Team SM Team
GA version(General Available)
InternalRelease
GA
Unit TestIntegration Test
System Test(Software perspective)
System Test(Include validating hardware infrastructure)UAT
Virtualization is useful
OrganizationRole & Responsibility
PMO(Project Mgmt Office)
PMO(Project Mgmt Office) Implementation Team
(PL)
Implementation Team(PL) Application Architect
Application ArchitectQuality Architect
Quality Architect
ALM MasterALM Master
Extract Requirement
Task Management
Dashboard
Source code management
Contiguous Build
Standard IDE
Standard Build Script
Testing Process
Testing Frameworkfor Unit,Integration Testing
Static Testing
Defect ManagementWiki based doc mgmt
Forum based bi-directionalcommunication
Code review
Testing Frameworkfor System,UAT
• Who has a responsibility to build up process and system?
Mentoring (Very important)
Mentoring (Very important)
OrganizationRole & Responsibility
Implementation Team(PL)
Implementation Team(PL)
Extract Requirement
Task Management
Dashboard
Source code management
Contiguous Build
Standard IDE
Standard Build Script
Testing Frameworkfor Unit,Integration Testing
Static Testing
Defect ManagementWiki based doc mgmt
Forum based bi-directionalcommunication
Code review
• Implementation team uses a lot of things in ALM It needs a strong mentoring to use it. Mentoring is not a
option but a mandatory..!!
ALM MasterALM Master
Mentoring (Very important)
Mentoring (Very important)
OrganizationRole & Responsibility
• ALM Master Required capabilities : Understand whole ALM architecture &
tools and process - May be genius? No. think with customer, act with customer. (Just right understanding of their ALM and help them not to miss a way..)
Responsibility Help customer to build up their ALM process and system. Help customer to adapt ALM. Inspect ALM usage in the team. (Inspect task management
system note, Review Wiki usage and structure. Check up Build script and CI status. ) and mentor them to right way.
Convince customer or project manager that no more free lunch.!! (to do something, let them know it needs a time and resource)
Each stakeholder wants to customize ALM for their perspective. In ALM, all of stakeholder has to share their perspective. (Even if they can see different views for their role) – It is liquid flow. So ALM master lead the stakeholder to have same perspective..
Help customer to change their culture slowly,smooth without revolt, awareness.
<Insert Picture Here>
Strategy for deliveringPractical ALM Framework
Key Strategy
• LiquidEach process of ALM are integrated as LIQUID
• SeamlessTools and solution that are used to realize ALM are integrated as seamless. Should be looks as one platform.
• Process OrientedTool and Systems are just tool to realize ALM concept. Do not depend on Tool. Process is key.
• Step by step depends on team levelEven if ALM is lightweight. It is hard to apply at once.Apply ALM modules step by step followed by team maturity level.
• Be SimpleStart of ALM is to remove complexity of traditional methodology.Complexity reduce efficiency and requires a lot of learning cost.
• And need a coach. (ALM Master)• Do not enforcing process or tool, but change culture.
Most important point of delivering ALM is changing culture.
• Delivery ALM secretly. Developer should not aware change. It means changes are applied smoothly.
Delivery Step
Overall road map of applying ALM
Build Environment Task Management Test Automation Collaboration
Level 1 SCM,CI, (AA)Standard BuildScript
Level 2 Standard IDE (PL) Excel based Task Management(PL)
Code Review (PL) Wiki (PL)
Level 3 Branch mgmt strategy (AA)
System based Task Mgmt (PM/PMO)Dashboard (PMO)
UnitTest (AA or PL) Regression Test (AA or PL)
Level 4 Extract Requirement (AA) Static Analysis (AA)
Level 5 Testing process (QA)
Level 6 Defect mgmt system (QA)
Recommended