View
224
Download
0
Embed Size (px)
Citation preview
introduction to CTSJuly 7, 2010Jingtao
What is CTS
Compatibility Test System
open source
commercial-grade(20000+ cases)
Automation compatibility test tool
type of case
• Unit tests
• Functional tests
• Reference application tests
• Robustness tests
• Performance tests
what does it test
• Signature
• Platform API
• Dalvik VM
• Platform Data Model
• Intents
• Permissions
• Resources
basic topic
CTS history
CTS work flow
download, install and run CTS
check CTS report
CTS History
• Android1.0 - 16784 cases
• Cupcake - 21383 cases
• Donut - 22756 cases
• CTS2.1-early-look-r2(Eclair) - 22847 cases
• CTS2.1-r1 - 23107 cases
• CTS2.1-r2 - 23123 cases
• CTS2.1-r3 - 23138 cases
• Next ? Froyo
CTS work flow
download CTSbinary: http://source.android.com/compatibility/downloads.html
source: git ls-remote --heads git://android.git.kernel.org/platform/manifest.git
repo init -u git://android.git.kernel.org/platform/manifest.git -b android-cts-2.1_r3
repo sync
install CTS
install Android SDK
unzip CTS package
vi android-cts/tools/startcts, set SDK_ROOT=<absolute fold name of SDK>
run CTSconnect phone to PC
android-cts/tools/startcts to enter CTS shell
3 mode to start test:
plan mode
package mode
case mode
Sample of CTS commands
• start --plan CTS
• start --plan Android -p android.app.cts.ActivityManagerTest
• start --plan Android -t android.app.cts.ActivityManagerTest#testGetRunningServices
run CTS under Windows1. unzip CTS package and android SDK package to your windows PC.
2. Modified the SDK_ROOT in cts\android-cts\tools\startcts :
SDK_ROOT=D:/CTS/sdk/android-sdk_r05-windows/android-sdk-windows/tools
3. use below command to enter CTS shell:
java -Xmx512M -cp D:\CTS\cts\android-cts\tools\cts.jar; D:\CTS\cts\android-cts\tools\
hosttestlib.jar;D:\\lib\ddCTS\cts\android-cts\tools\junit.jar;D:\CTS\sdk\android-sdk_r05-windows\
android-sdk-windows\tools\lib\ddmlib.jar com.android.cts.TestHost D:\CTS\cts\android-cts\
repository\host_config.xml
Please change red marked to you current file path.
known issue:
The file (cts_result.css, cts_result.xsl, logo.gif, newrule-green.png ) under CTS result can not
automatically copy in new session folder.
Attention:
If you need to clear the data in addon emulator, Please use: emulator @addon -wipe-data
config phone before run CTS empty SD card
never Screen Timeout
English locale
install and set input method to LatinIME
open WIFI and GPRS
correct date/time
/opl/etc/mountd.conf, system.prop
CtsDelegatingAccessibilityService.apk
test report firefox repository/results/20xx.../testResult.xml
issue of CTS itself
Test Package Test Cases CTS Error
android.permission.cts.NoNetworkStatePermissionTest testStopUsingNetworkFeature will be fixed in future release
android.provider.cts.ContactsTest testGroupMembershipTable bug 2258907, needs investigation
android.util.cts.TimeUtilsTest testWorld bug 2323433, needs investigation
advance topic
design CTS plan
Modify testResult.xml
fix syntax error
run fail cases
Tool in CTS
design new cases
build CTS
Automation test on Android
CTS folder structure• |-- repository
| |-- ...• | |-- plans
| | |-- Android.xml| | |-- ...| | `-- CTS.xml| |-- results| | |-- 2009.12.29_14.48.20| | | |-- ...| | | `-- testResult.xml| | |-- 2009.12.29_14.48.20.zip| `-- testcases| |-- CtsAppTestCases.apk| |-- CtsAppTestCases.xml| |-- ...| |-- android.core.vm-tests.jar| `-- android.core.vm-tests.xml`-- tools |-- cts.jar `-- startcts
CTS cases structurePlan CTS
apidemos
Android Java Performance RefApp VMSignature
vm-testspackages…contentapp widget
Test cases …
You plan…
Design your own CTS test plan 20000+ cases, 24+ hours
most cases never fail before, plan “Java”
plan “Android”, 3000+ cases, 3+ hours
specific test package for plan.
specific test case for package
content of plan “sanity”<?xml version="1.0" encoding="UTF-8" ?>
<TestPlan version="1.0">
<Entry uri="android.tests.sigtest"/>
<Entry uri="android.app"/>
<Entry uri="android.content"/>
<Entry uri="android.database"/>
<Entry uri="android.graphics"/>
<Entry uri="android.hardware"/>
<Entry uri="android.location"/>
<Entry uri="android.net"/>
<Entry uri="android.os"/>
<Entry uri="android.permission"/>
<Entry uri="android.provider"/>
<Entry uri="android.text"/>
<Entry uri="android.util"/>
<Entry uri="android.view"/>
<Entry uri="android.widget"/>
</TestPlan>
fail to open test report by firefox?
wait for test finished
vi testResult.xml, delete the error line
save and open it with firefox again
repeat to delete error line until can open by firefox
how to run fail case again?
open testResult.xml by gedit
replace all the result=”fail” to result=”notExecuted”
reset “Summary failed=” and “notExecuted=”
run tools/startcts, start --plan CTS, select Choose a session, usually select 1
run CTS more faster?• In respository/host_config.xml of CTS
• <!-- Number of tests executed between reboots. A value <= 0 disables reboots. -->
• <IntValue name="maxTestCount" value="200" />
• <!-- Max size [tests] for a package to be run in batch mode. -->
• <IntValue name="maxTestsInBatchMode" value="5000" />
• <!-- Max time [ms] between test status updates. -->
• <IntValue name="testStatusTimeoutMs" value="300000" />
• <!-- Max time [ms] from start of package in batch mode and the first test status update. -->
• <IntValue name="batchStartTimeoutMs" value="1800000" />
• <!-- Max time [ms] from start of test in individual mode to the first test status update. -->
• <IntValue name="individualStartTimeoutMs" value="300000" />
• ...
Tools in CTS
• Annotation helper
• Device info collector
• Process controller
• Test plan generator
• Signature generator
• Host configer
• Script starter
• …
develop new cases
• build/core/tasks/cts.mkCTS_CASE_LIST
• tests/tests/<yourpackage>/Android.mk(LOCAL_MODULE_TAGS, LOCAL_CLASSPATH, LOCAL_PACKAGE_NAME)
AndroidManifest.xml(package, android:label)
generate test description
buildCts.py <testRoot> <ctsOutputDir>
<tempDir> <androidRootDir> <docletPath>
e.g.
cts/tools/utils/buildCts.py cts/tests/tests
out/host/linux-x86/cts/android-cts /tmp .
out/host/linux-x86/framework/descGen.jar
build CTS
• . build/envsetup.sh
• tapas
• make cts
• out/host/linux-x86/cts/android-cts.zip
CTS in-depth
• java com.android.cts.TestHost in startcts
• cts/tools/host
• cts/tools/host/src/com/android/cts/TestHost.java
32