Upload
nguyenkiet
View
233
Download
5
Embed Size (px)
Citation preview
TFS/VS/MTM 2012 and Ranorex
Page | 1
Team Foundation Server /
Visual Studio /
Testmanager 2012
and
Ranorex
Documentation
Sample Virtual Maschine
(VM: TFS2012_U2_Ranorex)
Martin Sedlmeier
S3CC GmbH
Mai 2013
TFS/VS/MTM 2012 and Ranorex
Page | 2
Table of Contents
Overview ................................................................................................................. 3
Overview Infrastructure .......................................................................................... 5
Start the VM: ........................................................................................................... 6
Planning ................................................................................................................ 10
Implementation: .................................................................................................... 15
SCM & CI ............................................................................................................... 18
Integration (Part 1 and Part 2) ............................................................................... 22
Associate ............................................................................................................... 31
Troubleshooting .................................................................................................... 39
TFS/VS/MTM 2012 and Ranorex
Page | 3
Overview This document describes a Virtual Machine (VM) which contains a sample
integration of Ranorex into the Application Lifecycle Management (ALM) - Process
with Team Foundation Server (TFS), Visual Studio (VS) and Test Manager (MTM).
The VM TFS2012_U2_Ranorex extends a VM provided by Brian Keller (Microsoft
Principal Technical Evangelist for ALM). Many thanks to Brian for this VM
(downloaded from http://blogs.msdn.com/b/briankel/).
Figure 1: Overview Integration
Planning:
The story starts with the Planning of the User Stories (PBIs) for a Sprint. Part of this
PBI is a tiny Test Case. After manual Testing via Testmanager this Test Case is
planned for UI-Automation, should be used in our Build Definition of our
Application (CI) and the (automated) Test Case should be part of some Regression
Tests (later on).
TFS/VS/MTM 2012 and Ranorex
Page | 4
The UI-Test (Automated with Ranorex) should start inside Visual Studio
(Developer), inside a Build Definition (e.g. Nightly Build) and inside Microsoft Test
Manager (Tester).
Implementation:
The manual Testrun for this Test Case could be finished successfully. Now, we are
able to automate this Test Case via Ranorex.
SCM & CI:
The automated Test runs pretty well in Ranorex, so it is time to add the (Ranorex-)
Solution to Source Control in TFS. Additionally we create a Build Definition for this
(Ranorex-) Solution and copy the result to a shared Drop-Folder.
Integration (Part 1):
As next step we integrate the UI-Test into the (Unit)-Testing Framework. This allows
us to start the automated UI-Test via TestExplorer in Visual Studio.
Integration (Part 2):
Now, we are able to integrate this automated UI-Test into the nightly build of our
Application.
Associate:
Finally we associate the automated UI-Test with the Test Case. We created a tiny
Lab Environment, so we are able to start the automated UI-Test from Testmanager.
TFS/VS/MTM 2012 and Ranorex
Page | 5
Overview Infrastructure All Components are installed on a single VM. For sure, this shouldn’t be done in
real life, this VM is only for demonstrate reasons.
All Services are running under the Account Administrator. There is a Build- and Lab-
Environment configured (based on localhost).
All the work (e.g. automate UI-Tests, add to Source-Control) would be done under
the Account ./brian (Tribute on Brian Keller;-) ).
The VM is in a VMWARE-Format and runs quite well on a Win 8 Notebook. The VM
is configured with 4 GB Memory (should be enough) and 2 Processors. You need
VMWare-Player (free) for starting the VM locally.
All the Passwords (Accounts) are set to P2ssw0rd (Upper P – digit 2- lower ssw –
digit 0 (zero) – lower rd).
The software is provided with Trail-License Keys (10 days left). It is highly
recommended, that this keys will be replaced with valid MSDN-Licenses.
Please take a look to this document: http://download.microsoft.com/download/A/9/2/A9253B14-5F23-4BC8-9C7E-
F5199DB5F831/Working%20with%20the%20Visual%20Studio%202012%20Update%202%20RTM%20ALM%20Virtual%20Machine.docx
You will find some important information about the correct handling of the VM. It
describes the Hyper-V Version, but you will find some hints about time-settings in
the VM: The System Time in the VM is set to June 2012 (set by a tiny startup-task).
After using the VM, it is highly recommended, that you change the value in the File
C:\util\SetDateAndTime.bat to the current System Time in the VM (+ 5 min) before
you shutdown the VM.
The System Time will be set to this Value at the next startup of the VM.
TFS/VS/MTM 2012 and Ranorex
Page | 6
Start the VM: There are some steps to do to bring the VM (and Services) up and running: Note: please leave the accounts always in this State (e.g. logged on).
Description Screenshot
Copy VM to a local Folder
(e.g. C:\_MyVMs)
Start the VM
(DoubleClick on VM-Config-File
C:\_MyVMs\TFS2012_U2_Ranore
x \TFS2012_Update2.vmx)
Note: you have to install VMWare
Workstation or VMWare Player
Because of the configured Lab-
Environment after starting the
VM the user Administrator is
automatically logged in. The Test
Agent is started and Online.
If the Status is set to
disconnected, please refer to the
section Troubleshooting at the
end of this document.
Start the Build Service:
via Team Foundation
Administrator Console Build
Configuration Start
TFS/VS/MTM 2012 and Ranorex
Page | 7
The Build Service runs as a
“interactive process”: you have to
provide the Account-Password
(P2ssw0rd)
The Build-Service is starting:
Minimize the Windows
Important:
otherwise the UI-Tests will not
run
Close the TFS Admin Console:
File Exit
TFS/VS/MTM 2012 and Ranorex
Page | 8
Leave the Account Administrator
as logged on and switch to user
Brian:
Menu VM send Ctrl+Alt+Del
switch User
Menu VM send Ctrl+Alt+Del
DoubleClick Account Brian
Keller
Type in Password (P2ssw0rd)
Open a RDP-Session to the
Account Administrator on
localhost: DoubleClick RDP-Icon
on the Desktop of the Account
Brian,
Type in the Password (P2ssw0rd)
for Account Administrator
TFS/VS/MTM 2012 and Ranorex
Page | 9
Restore down the RDP-Window,
but do not minimize it.
Important:
otherwise the UI-Tests will not
run
TFS/VS/MTM 2012 and Ranorex
Page | 10
Planning Back to our Story: in this section we will go to the Story Backlog, take a look to our
PBI and the defined Tasks and Test Cases. We will see the Definition of a Test Plan
for Release 1 and a query based Test Suite. We will run our Test Case manually and
we take a look to the Test Results of our Test runs. After all, we change the
Automation Status on our Test Case from not automated to planned.
Description Screenshot
Under the Account Brian:
Open Browser IE and go to
the TFS FF Web Access
view Backlog
In the Sprint Backlog of the
Sprint 3 you will find the
PBI “Customer can create a
new Service Ticket” with
some Tasks.
That means:
- the Implementation is
planned
- the manual Test is
planned
- the automate Test is
planned
- the integration is planned
.., and trackable in the
Daily scrum Meeting
TFS/VS/MTM 2012 and Ranorex
Page | 11
The PBI contains a linked
Test Case.
In this State of the
Lifecycle of the PBI the
Automation State is set to
Not Automated.
After the Implementation
of the PBI (Task implement
UI) is done the manual
Tests could start
Open MTM (Testmanager)
Plan Contents
Release 1 Sprint 3
(query based)
You will find our Test Case.
Open Query behind this
Test Suite ( edit Query)
TFS/VS/MTM 2012 and Ranorex
Page | 12
The Query behind the Test
Suite is defined with the
Iteration Path and the
State of the Test Cases.
Under Test you will find
our Test Case 3 times,
because we defined 3
configurations for it
(Browser).
Start a manual Testrun via
Run Run with options
Note: there is a Build
associate with the current
Testrun
Check Box Run all the tests
manually for a manual
Testrun
(we will come back for
automated Testrun)
TFS/VS/MTM 2012 and Ranorex
Page | 13
Start the manual Test Run
(without Option Create
action recording) and save
the Result
( Save and Close)
You will find all Test Results
under View results
See all the Test Results
under Section Result
History.
DoubleClick on a specific
Test Result …
TFS/VS/MTM 2012 and Ranorex
Page | 14
… will open a detail view.
After the manual Tests
could be finished
successfully the
Automation Status could
be changed to Planned …
… and the Status of the
Task Test the implemented
UI could be set to Done.
TFS/VS/MTM 2012 and Ranorex
Page | 15
Implementation: In this Section we will automate the Testrun via Ranorex Test Suite.
Description Screenshot
Brian will find his Task
for automate the Test
Case.
Note:
The Task contains 3
Links (with LinkType):
Parent:
the PBI to this Task
Predecessor:
Task for manual Testing
(s. Planning)
Related:
the Test Case which
should be automated
The Test Steps in the
Test Case describes the
Test Scenario.
TFS/VS/MTM 2012 and Ranorex
Page | 16
Open the Ranorex
Studio and go to the
Test Suite
FabrikamFiberUITesting
Note:
We run the Tests with 3
different Browsers, but
we use always the same
Ranorex-Recording
Modules.
(The single Setups for
the Browser are
disabled, because we
will start this Testrun
outside of the Ranorex-
Studio.
At this point we are
able to run the
Testautomation via
Ranorex Studio Run
Test Suite
TFS/VS/MTM 2012 and Ranorex
Page | 17
This works fine…
… so Brian finished his
work quite well (and
could set the Status of
his Task to Done )
TFS/VS/MTM 2012 and Ranorex
Page | 18
SCM & CI At this Point we have an automated Test Case (with a Ranorex-Solution). Now we
will add this Solution to the TFS-Source Control (SCM) and create a Build-Definition
for this Ranorex-Solution (CI).
Description Screenshot
Brian will find his Task for
adding the Ranorex
Solution to the TFS-Source
Control.
Open the Visual Studio (VS),
connect to the local TFS
VSALM and the Team
Project FabrikamFiber in
the Collection
FabrikamFiberCollection
TFS/VS/MTM 2012 and Ranorex
Page | 19
Open the Source Control
Explorer and take a look to
the Ranorex-Solution.
You could work with this
Solution in Visual Studio …
… or in the Ranorex
Solution
TFS/VS/MTM 2012 and Ranorex
Page | 20
In the Team Explorer under
the Section Builds you will
find the Build Definition
Ranorex Solution (Main)
The Build Definition is using
a customized Build
Template called DefaultTemplate11.1forRanore
x
This Template builds the
Ranorex Solution and …
… copies the output (=
Ranorex Binaries) to a
shared Drop Location
TFS/VS/MTM 2012 and Ranorex
Page | 21
The first Part of the
Integration is done, Brian
leaves the Task-Status to In
Progress and set Remaining
Work to 3 (hours).
TFS/VS/MTM 2012 and Ranorex
Page | 22
Integration (Part 1 and Part 2) The next Step in our scenario is the Integration of the Ranorex Solution into the
(Unit)-Testing Framework. This means, we would like to start the UI-Tests inside of
Visual Studio and not only via Ranorex Studio (Part 1).
After this Integration we will be able to add this UI-Test to our Build Definition
Fabrikam (Main) with UI-Testing of our Application FabrikamFiber (Part 2).
Description Screenshot
Brian is still working
on this Task.
Under the Solution FabrikamFiber.CallCenter
you will find a new
Test Project.
TFS/VS/MTM 2012 and Ranorex
Page | 23
The Integration is
really simple: just
create a TestClass
with some Test
Methods (as always).
The Integration is
done via Ranorex
TestSuiteRunner.
Note:
This TestClass is using
the Ranorex-Binaries
located in the Drop
Location of the
Ranorex Build
Definition.
Open the Test
Explorer…
TFS/VS/MTM 2012 and Ranorex
Page | 24
… and run the Tests
You will find the
Result under Test
Explorer Passed
Tests Output
report.rxzlog (Ranorex
– Log File)
Note:
the TestMethod *_All
starts the UI-Tests
with the Browser IE
and FireFox,
TestMethod
*_All_WithChrome
starts IE, FireFox and
Chrome
TFS/VS/MTM 2012 and Ranorex
Page | 25
Now, we can add this
Tests to our new Build
Definition (in this
Case for the
Application-Solution)
Fabrikam (Main) with
UI.Testing
We are using the
Default Template.
Under Process
automated Tests 1.
Test Source we define
our TestClass (Test
Sources Spec) and
TestMethod (Test Case
Filter)
TFS/VS/MTM 2012 and Ranorex
Page | 26
Queue a new Build …
Note:
Be sure, that the Build
Service is started, the
RDP-Session is open
and not minimized.
Queue a new Build …
TFS/VS/MTM 2012 and Ranorex
Page | 27
As you can see in the
RDP-Window the UI-
Tests are running
under the Build
Service (Account
administrator)
You will find the Test
Results of this Build
under the Build-Log
(Team Explorer
Builds my Builds
DoubleClick on the
finished Build)
section test run
click on Test entry
opens the Test Result-
Window Right
Mouse Click View
Test Results Details
TFS/VS/MTM 2012 and Ranorex
Page | 28
You will find the
Ranorex-Log in the
Section Collected Files
(on the end of the
Log).
If the test goes to the
Status Failed , you
could quite simple
create a Bug –
Workitem.
TFS/VS/MTM 2012 and Ranorex
Page | 29
The Test Result
(including the
Ranorex- Log) will be
automatically
attached to the new
Bug.
Note:
Please set the
Iteration Path to the
current Release…
… and you will find
this Bug in your
Product Backlog (via
Web Access)
TFS/VS/MTM 2012 and Ranorex
Page | 30
You could queue a
Build also in the Web
Access.
The Task is still not
finished, so Brian let
the Status on In
Progress and changed
the Remaining Work
to 1 (hour)
TFS/VS/MTM 2012 and Ranorex
Page | 31
Associate Finally we have to combine our new TestMethod (implemented via Testing
Framework) and the original Test Case. This allows the Tester to run this automated
UI-Test within Microsoft Testmanager (MTM).
Description Screenshot
For associating a
TestMethod with a
Test Case, we have to
start Visual Studio
and open our tiny
Test Case.
There is a Query
available: (Team
Explorer Work
Items Queries
Shared Queries
Current Sprint
DoubleClick on
Query Test Case
Open the Test Case,
go to the Tab
Associated
Automation and take
a look to the set
Values.
Note:
The Automation
Status changed to
Automated
TFS/VS/MTM 2012 and Ranorex
Page | 32
Open the
Testmanager and
you will find our Test
Case in the Test Plan
Release 1 under the
Test Suite Regression
Testing (Testing
Center Plan
Content)
Note:
This is a Query Based
Test Suite. Click on
Edit query…
… and take a look to
the Query behind
the Test Suite.
TFS/VS/MTM 2012 and Ranorex
Page | 33
Run the Test (Testing
Center Test
Run Tests Release
1 Regression
Testing mark our
Test Case Run
with options …
We would like to run
this test
automatically (do
not check option
Run all the tests
manually leave the
Environment to
LabForRanorex and
start the Test run
Note:
The selected
Environment is on
the same VM, in Real
Life this should be on
different Maschines.
TFS/VS/MTM 2012 and Ranorex
Page | 34
As you can see in the
RDP-Window the UI-
Tests will start under
the Account
Administrator.
After the Test Run is
finished you will find
the Test Result under
the Section
Attachments.
DoubleClick the
attached *.trx-File…
TFS/VS/MTM 2012 and Ranorex
Page | 35
… will open the Test
Results inside Visual
Studio.
View Test Results
Details will open …
… a detail View of
the Test Results.
Under the section
Collected Files you
will find the Ranorex-
Log.
TFS/VS/MTM 2012 and Ranorex
Page | 36
You could create a
bug directly under
the Section Tests
Create bug)
As you can see under
the Tabs Test Cases
and Links some
additional Infos
would be
automatically linked
to the Bug (open it
with DoubleClick).
TFS/VS/MTM 2012 and Ranorex
Page | 37
You will find this bug
in the Product
Backlog.
You could find a
Summary about all
of the Test Runs in
Testing Center
Test Analyze Run.
Define the view
(Manual or
Automated run)
TFS/VS/MTM 2012 and Ranorex
Page | 38
Our tiny Test Case is
still manual
runnable. Check the
option Run all the
Tests manually
(Testing Center
Test run Tests
Release 1
Regression Testing
Run with Options
Finally all the work is
done, so Brian could
change the State of
his Task to Done.
TFS/VS/MTM 2012 and Ranorex
Page | 39
Troubleshooting
Testagent will not start
If the Test Agent does not start (State = disconnected), please go to the Test Agent
Configuration Tool (via Start search programs and files type in Test Agent)
and set the Password for the User of the Test Agent.
Ranorex will not start
If Ranorex will not start (Errormessage like : Ranorex License already in use…),
please restart the VM (but don’t forget to set the new time in
c:\util\SetDateAndTime.bat).