Upload
drewz-lin
View
305
Download
3
Embed Size (px)
Citation preview
Automation Test Framework
Continue Improvement
Welcome Presentation
Presented by Christina Geng
AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation
Title:7 Times Evolution of Test Automation Framework under JS
Product for Efficiency Improvement
Target:
❶ Build a robust test automation framework
❷ Manage test cases efficiently
❸ Reduce test case running time and maintenance cost
❹ Shared framework under scrum development
CASE INTRODUCTION
WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing • analytics • data mining • process mining • event processing • business performance management • benchmarking • text mining • predictive analytics
WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing • analytics • data mining • process mining • event processing • business performance management • benchmarking • text mining • predictive analytics
Programming language: Javascript/Java
Provide service:
The Visualization Service is a software module which provides various type of visualization objects that can be used in different applications on different platforms (Web, Mobile and Desktop).
Visualization types:
• A basic visual component like chart (Bar, Column Pie, Line, etc ), Geo Map, Gauge, Table or Grid. • An advanced visual component like (Tree map, Candlestick, Tagcloud)
• Above visual component with interactions and animations • Combination of multiple visual components, it could be composed by same type of visual components or different type
of component.
AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation
We h
ave it!
Seleniu
m
Data D
riven
Jenkin
s
Test Link
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-1
Test page
SilkTest
TC: test page Scheduler: ASTEC Test tool: SilkTest TC Management: Excel
VERSION-1
Test page
SilkTest
+ -
HTML Test
page
1. First automation framework setup 2. Overview functional test case
coverage 3. Reduce testing overlap for shared
component
writing test case
maintenance
the relation between TC and record
resource
We h
ave it!
Seleniu
m
Data D
riven
Jenkin
s
Test Link
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-2
DIV
DIV DIV
DIV Test page
Selenium
Scheduler: ASTEC TC Management: Excel/XML (test manager) TC: test page Test tool: Selenium Test Driver
Other tools • Auto generate test plan from Excel to xml • Test Manager
DIV
DIV DIV
DIV Test page
VERSION-2
+ - 1. Reduce test pages 2. Reduce resource running cost 3. Test steps turns to be keyword driven in xml 4. Write test case in .xls, auto generate .xml
Selenium
DIV
DIV DIV
DIV Test page
test driver
test case sync
case unit
test data redundancy
We h
ave it!
Seleniu
m
Data D
riven
Jenkin
s
Test Link
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-3
Wrapper API
Unified Test Driver
Abstract Test Data
Eliminate Test Page
And How?
VERSION-3
Wrapper API
backgroud color
visible
legend position
verticalOffset horizontalOffset
visible labelFont
labelColor formatString
layout padding
tooltip enable
customization text
mainValue subValue
title main sub
Selectability mode
deSelectale defaultSelectedItems
xAxisMember yAxisMember
bar barMarkerSize
measureMember
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
VERSION-3
Wrapper API
backgroud color
visible
legend position
verticalOffset horizontalOffset
visible labelFont
labelColor formatString
layout padding
tooltip enable
customization text
mainValue subValue
title main sub
Selectability mode
deSelectale defaultSelectedItems
xAxisMember yAxisMember
bar barMarkerSize
measureMember
updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
VERSION-3
Wrapper API
backgroud color
visible
legend position
verticalOffset horizontalOffset
visible labelFont
labelColor formatString
layout padding
tooltip enable
customization text
mainValue subValue
title main sub
Selectability mode
deSelectale defaultSelectedItems
xAxisMember yAxisMember
bar barMarkerSize
measureMember
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
testUtil.generateChart(viz/line; TestData.a1a1m1m1)
VERSION-3
Wrapper API
backgroud color
visible
legend position
verticalOffset horizontalOffset
visible labelFont
labelColor formatString
layout padding
tooltip enable
customization text
mainValue subValue
title main sub
Selectability mode
deSelectale defaultSelectedItems
xAxisMember yAxisMember
bar barMarkerSize
measureMember
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]); testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
VERSION-3
Abstract Test Data
public
private
testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
mySQL
testUtil.generateChart(viz/line; TestData.a1a1m1m1; legendVisibleTrue)
VERSION-3
Eliminate Test Page
VERSION-3
Unified Test Driver
selenium1
selenium2 {StartBrowser}
IE, Chrome, Firefox (selenium2)
RemoteWebDriver webDriver = null; webDriver = new FirefoxDriver();
Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2
{StartBrowser}
Safari (selenium1)
SeleniumServer seleniumServer = new SeleniumServer(); seleniumServer.boot(); selenium = new DefaultSelenium("localhost", 4444, "*safariproxy "+ safariLocation,urlRoot); CommandExecutor executor = new SeleneseCommandExecutor(selenium); DesiredCapabilities dc = new DesiredCapabilities(); webDriver = new RemoteWebDriver(executor, dc);
Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2 {TakeSnapshot}
Point p = this.getElementOffset(targetElement); Dimension d = this.getElementSize(targetElement);
BufferedImage bi = new Robot().createScreenCapture(p.x, p.y, d.width, d.height);
Robot()
ImageComparer
VERSION-3
Unified Test Driver
selenium1
selenium2
ImageComparer
{FileCompare} cmd.exe ImageComparer.exe –file filepath1 filepath2 – threshhold 100
Robot()
We h
ave it!
Seleniu
m
Data D
riven
Head
less
Test Link
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-4
Tooltip Selection
Gesture
CSS Button
Link
Dropdown
Checkbox
Table Text
Scroll
Hover
Animation
Theme
Upload
Download
Tooltip Selection
Gesture
VERSION-4
Selection
VERSION-4
Gesture
VERSION-4
var eventObj = {
type : originalEvent.type,
timeStamp : originalEvent.timeStamp,
pageX : originalEvent.pageX,
pageY : originalEvent.pageY
};
eventRecord.push(eventObj);
var handler = function(event){
var clickEvt = document.createEvent('MouseEvents');
clickEvt.initMouseEvent(event.type, event.pageX, event.pageY…);
d3.select('.main .plot')[0][0].dispatchEvent(clickEvt)
};
testAPI_lasso_heatmap = [
{"type":"mouseover","timeStamp":1348739963751,"pageX":126,"pageY":94},
{"type":"mouseover","timeStamp":1348739963751,"pageX":485,"pageY":94},
{"type":"mousemove","timeStamp":1348739963766,"pageX":485,"pageY":208},
{"type":"mousemove","timeStamp":1348739963773,"pageX":126,"pageY":208}
]
We h
ave it!
Seleniu
m
Data D
riven
Head
less
Test Link, Jen
kins
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
VERSION-5
How to arrange test case?
VERSION-5
Test
Link
TC (.xls/.xml)
Executable TP
import
export
Result log save
VERSION-5
Continuous Integration
1. Trigger
CI Server Test Executor
Build Machine Code repository
Manual Auto
TC mngmt
Test Case Pool
Test Driver
Selenium2 Java/C++
Web Server
2.TP/Platform/build info
resources
4. get assets
libs
4. get test plan 3. Build info 5. result write back
5. result log
.xml
.xls
VERSION-5
Where we are.
1. Trigger
CI Server Test Executor
Build Machine Code repository
Manual Auto
TC mngmt
Test Case Pool
Test Driver
Selenium2 Java/C++
Web Server
2.TP/Platform/build info
resources
4. get assets
libs
4. get test plan 3. Build info 5. result write back
5. result log
.xml
.xls
VERSION-5
Test Link
Jenkins Java Tool
Perforce
Where we are.
We h
ave it!
Seleniu
m
Data D
riven
Head
less
Test Link, Jen
kins
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
Engineering
VERSION-6
Paint
Layout Display
http://phantomjs.org PhantomJS (Headless Webkit)
VERSION-6
Test driver inner logic
CasperJS IE/Safari/FF/Chrome
config Browser?
Open browser
RunJS (test step)
….
Take Snapshot
Result validation
Save result log
Read JS list TP
PhantomJS: 1. open webkit 2. runJS 3. Take snapshot 4. close
not complete not complete
Reporting
And that’s OK
We h
ave it!
Seleniu
m
Data D
riven
Head
less
Test Link, Jen
kins
Full A
uto
matio
n
Develo
per en
viron
men
t
Wh
at’s mo
re…
2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12
trigger
CI Server Test Executor
Build Machine
P4 server
TC Management
Test Case Pool
Test Driver
Selenium2 Java/C++
get resources
sync up
.java
per check-in
local
Test Executor
result log
VERSION-7
Casper/PhantomJS
trigger
CI Server Test Executor
Build Machine
P4 server
TC Management
Test Case Pool
Test Driver
Selenium2 Java/C++
get resources
sync up
.java
per check-in
local
Test Executor
result log
VERSION-7
Casper/PhantomJS
Eclipse Plug-in
Perforce
Jenkins
AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation
1. Set up environment
2. Test case execution
3. Test result reporting
4. Maintenance
CASE ROI ANALYSIS
1) Test machine 2) Assets (util, BM…) 3) TP 4) Libs
1) Browser compatibility 2) Run in headless
1) Email sending 2) Sonar integrated with CI
1) BM update 2) TC management
Functional Test: BAT (nightly), FA
1000+ cases
4.5h
1. Set up environment
2. Test case execution
3. Test result reporting
4. Maintenance
CASE ROI ANALYSIS
1) Test machine 2) Assets (util, BM…) 3) TP 4) Libs
1) Browser compatibility 2) Run in headless
1) Email sending 2) Sonar integrated with CI
1) BM update 2) TC management
Functional Test: BAT (nightly), FA
1000+ cases
47min
AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation
SUCCESS FACTOR ❶ ❷❸❹
Focus
SUCCESS FACTOR ❶ ❷❸❹
Team work
SUCCESS FACTOR ❶ ❷❸❹
Continuous Improvement
SUCCESS FACTOR ❶ ❷❸❹
Learn 360°
AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation
OUR REVELATION
不被传统的测试方法束缚;从产品用到的技术找到突破;持续改进的思想;不断尝试的理念
CONTRIBUTOR Xiaolu, Ye
Christina, Geng Helen, Jiang Chen, Zhou Peipei, Jiang
Coco, Dai Grace, Zhao Minghao, Lu Kent, Wang Izzie, Wang
Phil, Xu
SUPPORTED FROM Daoyu, Wei
Li, Zhao Jimmy, Yang Alex, Feng Alex, Su Jiwei, Li ... …
THE END NO
Email: [email protected] Sina Weibo: 耿晓儿de测试