Practical Sikuli: using screenshots for GUI automation and testing

  • View
    78.007

  • Download
    1

Embed Size (px)

DESCRIPTION

This's a Sikuli tutorial I gave to the Megason Lab at Harvard Medical School.

Text of Practical Sikuli: using screenshots for GUI automation and testing

  • 1. PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1
  • 2. WHAT IS SIKULI? 2
  • 3. WHAT IS SIKULI? 2
  • 4. 2 3 13
  • 5. Low-level API?Macs system design? 4
  • 6. EMPTY THE TRASH CAN 5
  • 7. OUTLINE What Sikuli is How Sikuli works Sikuli for GUI automation Sikuli for GUI testing Conclusions 6
  • 8. How Sikuli Works 7
  • 9. TEMPLATE MATCHINGclick( ) 8
  • 10. TEMPLATE MATCHINGclick( ) 8
  • 11. SYSTEM DESIGN 9
  • 12. DEMO 1:EMPTY TRASH CAN 2 3 1 10
  • 13. REGION AND MATCH Regionmatch = region.nd( or text pattern) 11
  • 14. REGION basic attributes: x, y, width, height Match is a Region Screen is also a Region chained methods: Screen(0).nd(W).click(X) 12
  • 15. KEY METHODS OF REGION nd, ndAll, exists, wait, waitVanish Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp Keyboard Actions type(text), paste(text) keyDown, keyUp 13
  • 16. EX2: FACEBOOK STALKER 14
  • 17. EX2: FACEBOOK STALKER 14
  • 18. RESTRICT SEARCH REGION 15
  • 19. But the Region constant is bad 16
  • 20. APP CLASS refox = App.open(Firefox) refox.focus(), refox.close() refox.window([n]) returns the Region of Firefoxs n-th window 17
  • 21. FIND IN CERTAIN APP 18
  • 22. EX3: MUTE SPEAKERS 19
  • 23. EX3: MUTE SPEAKERS 19
  • 24. SPATIAL OPERATORS 20
  • 25. SPATIAL OPERATORS 20
  • 26. SPATIAL OPERATORS (2) 21
  • 27. TEXT MATCHING AND RECOGNITION click(OK), nd(Cancel) region.text() - read the text in a region 22
  • 28. VISUAL-EVENT-DRIVEN PROGRAMMING observes visual events in any Region Region.onAppear(Pattern, handler) Region.onVanish(Pattern, handler) Region.onChange(Pattern, handler) 23
  • 29. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  • 30. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  • 31. VISUAL DICTIONARY 25
  • 32. GUITESTING 26
  • 33. VISUAL ASSERTION make sure the appearance of visual feedback 27
  • 34. VISUAL ASSERTION make sure the appearance of visual feedback 27
  • 35. JUNIT INTEGRATION 28
  • 36. SIKULI IN COMMAND LINEusage:Sikuli-IDE [--args ] [-h] [-r ] [-s] [-t ] --args specify the arguments passed to Jythons sys.argv -h,--help print this help message -r,--run run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test run .sikuli as a unit test case with junits text UI runner 29
  • 37. CONCLUSIONS 30
  • 38. PLATFORM INDEPENDENCE Works on ANY GUI that can be displayed on Windows/Linux/Mac Virtual machines Remote desktop Mobile simulators: Android, iPhone Web: Flash, HTML+Javascript 31
  • 39. PROGRAM AGAINST UISikuli programs are written against the user interfaceinstead of an API UI: visible, familiar, always exists API: faster, probably more stable 32
  • 40. FUTURE WORK fast and accurate OCR on screen Accessibility API integration 33
  • 41. Thanks! http://sikuli.org or, just google sikuli New Sikuli X 1.0-rc2 released.