Robot Aided Test Automation · Robot Aided Test Automation Collecting and utilizing user data Testiautomaatiopäivä 2014 Jussi Liikka, Arttu Lämsä, Miska Seppänen VTT Technical

Robot Aided Test Automation Collecting and utilizing user data

Testiautomaatiopäivä 2014

Jussi Liikka, Arttu Lämsä, Miska Seppänen

VTT Technical Research Centre of Finland

Click to edit Master title style

Click to edit Master text styles

Second level

Third level

Fourth level

Fifth level

2 05/06/2014 2

Outline

1. Introduction

2. Architecture overview

3. Client and the collected data

4. Preprocessing the data

5. Generating the Markov Model

6. Robot controlling

7. More detailed UI usage tracking

8. Data collection overview

9. Examples

10. Future work & Conclusions



Second level

Third level

Fourth level

Fifth level

3 05/06/2014 3

Introduction

When testing extensive software components, such as mobile

platforms, number of test cases is high

How to prioritize testing?

Real-life user data can be collected and used to determine the most

used elements and patterns of the software platform

A model provides probabilities of transitions between user interface

elements

How to realistically test current consumption and device aging?

By generating usage patterns from the collected user data and by

using it in robot based testing via user interface



Second level

Third level

Fourth level

Fifth level

4 05/06/2014 4

Introduction (cont.)

A mobile client, operating completely

transparent to the user, collects user data

Statistical model of the information gathered

from multiple devices

Robot controlling utilizes the collected user

data

Image: http://www.lg.com/fi/matkapuhelimet/lg-Nexus-4-E960



Second level

Third level

Fourth level

Fifth level

5 05/06/2014 5

Architecture overview

1. Client devices collect

data and send it to the

server

2. Data is preprocessed

(periodically)

3. MarkovAnalyzer reads

preprocessed data

and generates Markov

models from the user

data

4. Markov models can

be used to control

automated testing

robots



Second level

Third level

Fourth level

Fifth level

6 05/06/2014 6

Client and the collected data

An Android client collects user data

The client is transparent to the user

The client collects the following data

Foreground activity name (timestamps tracked when an activity is

changed)

Touch events (coordinate and timestamp)

Hard button events

Screenshots (partial and whole screen)

Icons of the used applications

The collected data is stored into the internal memory of the

mobile device

The data is then converted into JSON format and transferred to

a server along with the screenshots once per day



Second level

Third level

Fourth level

Fifth level

7 05/06/2014 7

Preprocessing the data

The server receives the data and stores the

raw JSON data into a temporary database

table

A preprocessing utility processes the data and

stores it into multiple database tables

Tables for Android activity changes, screen

touch events, etc.

The preprocessing utility works as a batch

process that is executed e.g. once per day



Second level

Third level

Fourth level

Fifth level

8 05/06/2014 8

Generating the Markov Model

MarkovAnalyzer utility reads the

data from the database and

generates a Markov model

The model contains the Android

activities, transition probabilities

between them, touch paths

(coordinate sequences leading

from one activity to another), icon

of the activity and filename of the

activity screenshot

The model can be created from

single user’s data or by combining

data from multiple users Image from:

http://en.wikipedia.org/wiki/File:Markovkate_01.svg

(Creative Commons Attribution-Share Alike 3.0 Unported license,

author Joxemai4)

http://creativecommons.org/licenses/by-sa/3.0/deed.en



Second level

Third level

Fourth level

Fifth level

9 05/06/2014 9

Generating the Markov model (cont.) <MarkovModel>

<GuiState>

<ID>0</ID>

<ActivityClassName>eu.chainfire.supersu.PromptActivity</ActivityClassName>

<TimesVisited>9</TimesVisited>

<Transitions>

<Transition>

<ID>1</ID>

<Probability>1</Probability>

<Triggers>

<Trigger>

<Path>

<Coordinate>

<X>390</X>

<Y>1256</Y>

<Delay>1000</Delay>

<Orientation>1</Orientation>

</Coordinate></Path>

</Trigger>

<Trigger>

<Path>

<Coordinate>...

….

P1

P11

P12

…

…

… …

…



Second level

Third level

Fourth level

Fifth level

10 05/06/2014 10

Robot controlling

Open Source Modelling Objects (OSMO) is a set of tools and

libraries for model-based testing of systems and software

Robot controlling scripts can be generated from Markov models

by using OSMO or Markov2Robot utility

The generated scripts contain touch coordinates, delays

between them and the icon file names



Second level

Third level

Fourth level

Fifth level

11 05/06/2014 11

More detailed UI usage tracking

• Client software takes a small partial

screenshot from users’ last touched position

of the screen.

• Partial screenshot image is compared to

collection of images available in a test

model.

• A predefined test model contains information

about application views and transformations

between them.

• Test sequence generator is now able to

imitate the real user according to the

probabilities calculated from recorded data.



Second level

Third level

Fourth level

Fifth level

12 05/06/2014 12

Data collection overview

Five users over period of 6 months

~3.2 million touch event samples

150k application events

498 different Android activities



Second level

Third level

Fourth level

Fifth level

13 05/06/2014 13

Example analysis of the collected data –

Robot control

Robot controlling LG Google

Nexus 4 based on the collected

user data

The robot is hosted by

Optofidelity in Tampere

OptoFidelity. OptoFidelity HSUF One Finger Touch and Test System.

http://www.youtube.com/watch?v=nodOph7uZB0&feature=share&list=UUxZMoLkyOc7rMwKuTrNp6HQ



Second level

Third level

Fourth level

Fifth level

14 05/06/2014 14


Touch events

Heatmap was generated from all of

the collected touch events from all

users

Most used parts of the screen

shown in light colors



Second level

Third level

Fourth level

Fifth level

15 05/06/2014 15


Partial image analysis

Background image All touch positions Overlay



Second level

Third level

Fourth level

Fifth level

16 05/06/2014 16

Future work

Dynamic content recognition

Perceptual computing

Suitable metrics for behavioral testing



Second level

Third level

Fourth level

Fifth level

17 05/06/2014 17

Conclusion

How to prioritize testing? Our approach was the utilisation

of user data

Collect real-life user data of touch screen mobile devices

Model can be constructed from collected user data

Model can be utilized for example test case prioritization or user

interface navigation analysis

Contact information http://ca.vtt.fi

More information about VTT’s test automation research

http://www.testautomation.fi/

http://ca.vtt.fi/



TECHNOLOGY FOR BUSINESS

Documents

Robot Aided Test Automation · Robot Aided Test Automation Collecting and utilizing user data Testiautomaatiopäivä 2014 Jussi Liikka, Arttu Lämsä, Miska Seppänen VTT Technical