Embed Size (px)
Citation preview
This presentation shares experience gotten from testing such products as
Java FXJava FXDes ig nerDes ig ner
No logo yet
Agenda
Test automation effectiveness“To automate or not to automate”
UI specificWhat's UI testing about
How to write testsDifferent approaches to incapsulatelogic into test library
ToolsRequirementsJemmy
The three things about automated tests
Inexpensive to execute.That's good, but
Expensive to createAnd, guess what ...
Require maintenance'Cause ... they ... FAIL
Hmm ... need to optimize the overall value, then ...
Automation effectiveness constituents
TM – time needed to run the tests manually
this is also a multiplier for other time values.T
D – time needed for automated test development
which includes developing of tools, harnesses, etc.T
S – time needed for automated test support
tests are needed to be updated if the product gets updated.
NR – number of releases (test cycles)
vary a lot for different group of testsN
C – number of tested configurations
OSs, external servers, etc.Putting it all together ... (next slide, please) ...
The formula
EA – automation effectivenessTo be used for every particular product.
NR and N
C are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and T
S - higher the E
A.
Some examples ...
TD + *T
S NR
TM * N
RN
C*E
A =N
C*
Charts
1 2 3 4 5 6 7 80
5
10
15
20
25
0
1
2
3
4
Nc = 1 Ratio
Auto Manual Effect
Time
1 2 3 4 5 6 7 80
5
10
15
20
25
0
1
2
3
4Nc = 3 Ratio
Auto Manual Effect
Time
Assumptions: TM = 1 engineer*week T
S = 0.1 * T
M T
D = 5 * T
M N
R = 8
Td or T
s – what to minimize
TS - if (N
C * N
R) is big
Multi-platformCompatibility with external products (servers, browsers, ...)Long-living
Other words, when you want the testing to be repeated
TD - if (N
C * N
R) is small
Proof of conceptPreview
Just to run them several times for one release
Coding vs. recordingScripting/codingTests are created manually. T
D could be significant.
Record/PlaybackUser actions are recorded by some tool and stored for future use.T
D is close to T
M, but T
S is usually big (for all tools known to authors).
Semi-automaticMix of the two above.
Is this all about ROI? Not at all!
Commit
BuildExecuted
automagicallyafter commit
Success
TestingIs it working?
Passed
Analysis. Rollback!!!
Code changes
Testchanges
Continuos build system:
How's that possible with manual tests???
No
No
Test furtherBuild is good
Code line is healthyGo on ...
Yes
Yes. = Compilation
successful
Earlier is better
With continuous build, you find bugs at coding stage.
Performance && load testsAre executed many times.
EAP
=
NP – Number of times test is executed during
performance/load test cycle
EA N
P*
What about UI
What's special about UI testing
UI test ...
Find the damn control!Such as the “OK” buttonDo something with it!Push the damn thingCheck for the desired effectSome other control is displayedSome file changedSomething happened ...Find the next controlKeep going ...
So, UI test automation is aboutControl lookup
find that thingy over thereand
Human input simulationscroll it or something
and
Result verificationcheck if scrolled
and
WaitingFor something to happen
Component lookup criteriaBy IDEasiest – may not be possible
By typeMost common
By indexUnavoidable
By toString()sucks
By text/tooltip/associated labelBest, if possible
Are these all the search capabilities
you need?
Oh, com'on!
Oh, com'on! continues
And
aga
in ..
.
Component lookup revisitedBy .. whatever you want ...
class MySearchCriteria implements SomeSearchCriteriaInterface {
public boolean theControVerificationMethod (ControlType control) {
return control.getSomeControlProperty(). equals(someThing); }}
InputMouseclick(Point point, int button, int modifiers, int times)press(int button, int modifiers)release()dragNDrop(Point from, Point to, int button, ...)Keyboardpush(int keyCode, int modifiers)press(int keyCode, int modifiers)release()
Is this all you need?
Component specificText fieldtype(String text)clear()Check boxselect(boolean)Scroll barscroll(int value)Tableselect(int row, int column)etc.
Verifying resultUI feedbackDialog appearedText displayedObject property“white box”External resourcefile on diskrecord in database
Timing ...
Test is executed in one threadApplication in ... manyMain application threadEvent queue thread(s)Background threadsModal dialogsLoading somethingetc
ANY test operation must be done with waiting!
Two options
Sleep No good, 'causeTests are slowThe timeouts are not big enoughWaiting
For a control to appearbe ready for inputhave some property
Event queue to be empty
Some file on the disk to be updatedetc.
UI test coding models
Remember the formula?Application exampleDifferent approachesSummary
Remember the formula?
TD + *T
S NR
TM * N
RN
C*E
A =N
C*
EA – automation effectivenessTo be used for every particular product.
NR and N
C are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and T
S - higher the E
A.
Coefficient depend on the way you write your tests
Ts depends on the way tests are written
TS mainly consists of time for test modification.
When product changes, tests need to be changed accordingly.
Many tests! Hundreds.
“Less changes of test code per a change in the product UI.”Ideally ... no more than one.
But ... how? You are the coders – you know:Move code to test library.
Application domain modelCa
r rec
ord
Car r
ecor
d
Color
Model Make
Year License plate
VIN
...
That's ... car catalog of some sort
Application UI
Product UI
Coordinatesclick(134,32) //selects some recordclick(215,122) //hits “Properties”sleep(5) //sleeps to let dialog be paintedclick(64,182) //expands color comboclick(235,182) //selects Grayclick(235,212) //hit OK
TTdd ~= 1.1 * T ~= 1.1 * Tmm, , TTss ~= 1 * T ~= 1 * T
mmNever tried, but ...
WidgetsFind “Car records” frameFind tableSelect “1abc234” cellPush “Properties” buttonWait for “1abc234” dialogSelect “Gray” color in combo boxPush “OK”
Prod
uct U
IDo
main
mode
l
Car record
ColorModel Make Year License plateVIN
Test
Widgets or coordinates
TTDD ~= 3 * T ~= 3 * TMM, , TTSS ~= .5 * T ~= .5 * T
MM
UI PrimitivesFind car list frameCarListFrame list = new CarListFrame()
Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);
Set color to graypropDialog.setColor(Color.GRAY);
Apply changespropDialog.ok();
LibraryPr
oduc
t UI
Doma
inmo
del Car record
Test library
ColorModel Make Year License plateVIN
CarListFrame CarDialog
TestTTDD ~= 5 * T ~= 5 * T
MM, , TTSS ~= .2 * T ~= .2 * TMM
Domain modelSet color to gray for a car “1abc234”
new CarRecord(“1abc234”). setColor(Color.GRAY);
Underneath the cover, CarRecord class does all described earlier
Domain library
Prod
uct
UIDo
main
mode
l Car record
Domain test library
ColorModel Make Year License plateVIN
UI test libraryCarList CarDialog
CarRecord
TestTTDD ~= 7.5 * T ~= 7.5 * T
MM, , TTSS ~= .05 * T ~= .05 * TMM
Td and T
s together
Coordinates Widgets UI Library Domain library0
0.51
1.52
2.53
3.54
4.55
5.56
6.57
7.5
1.1
3
5
7.5
1
0.50.1 0.05
Td/TmTs/Tm
TD and T
S for N
C=3, N
R=8, T
M=1
Coordinates Widgets UI Library Domain library0
2.5
5
7.5
10
12.5
15
17.5
20
22.5
25
27.5
1.1
3
5
7.5
24
12
2.41.2
25.1
15
7.48.7
TdTsTd+(Ts*Nc*Nr)
EA for N
C=3, N
R=8
Coordinates Widgets UI Library Domain library0
0.250.5
0.75
11.251.5
1.752
2.252.5
2.753
3.25
0.9561752988
1.6
3.24324324324
2.75862068966
Ea
Tools
RequirementsJemmy
Tool requirementsStabilityIf tests fail randomly, Ts is increasing for nothingPowerYou must be able to automate all the tests you needFlexibilityYou must be able to reuse tool API efficiently to build on top of it
Jemmy history
19992000Started to be used for NB modules20002001Accepted as NetBeans UI test tool
~2001Made it's way to Java SQE20022004Java Studio Creator (Project Rave)20082009
Started as a tool for testing TeamWare UI
Jemmy v2
Jellytools
Hosted on dev.java.netJemmy v3
Jemmy v2 factsCovers Swing/AWTTests written in JavaUsed extensively within SUNOpen sourceVery stable code
Operators in Jemmy v2Wrapper around java.awt.Componentpublic Component <?>Operator.getSource()Mimic Swing/AWT hierarchyJButtonOperator extends AbstractButtonOperator extends JComponentOperator extends ComponentOperator
Lookup constructorsComponentOperator(ContainerOperator, ComponentChooser)ComponentOperator(ContainerOperator, ComponentChooser, int)JTextOperator(ContainerOperator, String)JTextOperator(ContainerOperator, String, int)JTableOperator(ContainerOperator, String, int, int)JTableOperator(ContainerOperator, String, int, int, int)JListOperator(ContainerOperator, String, int)JListOperator(ContainerOperator, String, int, int)etc
Convenient methods ...ComponentOperatorclickMouse(...)pushKey(...)JTextOperatortype(String)AbstractButtonOperatorpush()JTreeOperatorselectPath(String)...
Jemmy 3JemmyCoreUI library independent moduleJemmyAWTSmall wrapper around Jemmy2Reference implementationJemmySGScenegraphJemmyFXFranca (limited)Marina (coverage for javafx.scene.control)
LookupInterface LookupCriteria<CONTROL>Boolean check(CONTROL)ByTextLookupTo look by textCoordinateLookupBy position
Parent and Lookup interfacesParentLookup<CONTROL> lookup(LookupCriteria<CONTROL>)Search by criteriaLookup<ST extends CONTROL> lookup(Class<ST>, LookupCriteria<ST>)Search by type and criteria
Lookup<CONTROL> extends Parent<CONTROL>void wait(int)CONTROL get(int)Control<CONTROL> control(int)void dump(OutputStream)int size()
Control<CONTROL> classWrapperCONTROL getControl()
boolean is(Class<INTERFACE>)Checks if the control could be treated as an interface
INTERFACE as(Class<INTERFACE>)Treats the control as interface
LookupParent<MyControl> parent = ...;LookupCriteria<MyControl> lookup1 = ...;//MySubControl extends MyControlLookupCriteria<MySubControl> lookup2 = ...;parent.lookup(lookup1).size(); //how manyparent.lookup(lookup1).get(0); //firstparent.lookup(lookup1).control(0); //wrap//narrow down the search further.parent.lookup(lookup1).lookup(MySubControl.class, lookup2). control(0);
InterfacesParent – seen alreadyMouseclick(...), press(...), release(...)Keyboardpush(...), press(...), release(...)Dragdnd(...)ScrollFor scroll bars, sliders, spinnersSelectableFor check boxes, radio buttonsWindowFor anything resize-able and movable.
Interfaces cont.Control<MyControl> control = ...;//basic input is built incontrol.mouse().click(); // same as control.as(Mouse.class).click();control.drag().dnd(...);
//easy to use other functionalitycontrol.as(Scroll.class).scroller().scrollTo(50);
//here it gets interesting ...control.as(Parent.class, MyControl.class).lookup(lookup1).control(0). mouse().click();
Lookup revisitedParent<Scene> allScenes = ...;allStages.lookup(new ByTitleSceneLookup(“My window”)). control(0). as(Parent.class, Node.class). lookup(CheckBox.class, new ByTextLookup(“check box”)). control(0). as(Selectable.class, Boolean.class). selector().select(true);
Demo
Are you still alive?
