Upload
naoki-ainoya
View
1.735
Download
0
Embed Size (px)
Citation preview
Run smoke test on AWS DeviceFarm@naokiainoya
About me• Naoki Ainoya(@naokiainoya)
• iOS/Server/Infra engineer
• Swift/Scala/Go/Docker...
• Android...?
• Recruit Marketing Partners Co, Ltd.
• Works for 英単語サプリ mainly
Support android development
• Maintain Jenkins CI system for building android
• Build apk with using Docker/Walter
• Ref: http://ainoya.io/docker-android-walter
Smoke testing• e.g. smoke test may ask basic
questions like "Does the program run?"
• Smoke testing performed on a particular build is also known as a build verification test
• Ref: http://www.slideshare.net/confiz/software-testing-methods-levels-and-types
How many test do I have to test?• There are a lot of environments!
• Different Screen resolution
• Different Architecture
• arm, x86, ...
• We want to test on as many devices as possible
But we don't have such a many devices
Actually even I want to use
AWS DeviceFarm!• Device Farm is an app testing
service that enables you to test your Android, iOS, and Fire OS apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS).
• Ref: http://www.slideshare.net/AmazonWebServicesJapan/20150722-black-belttechwebinarawsdevice-farm
• Ref: http://docs.aws.amazon.com/devicefarm/latest/developerguide/welcome.html
How does AWS DeviceFarm work
• create project
• schedule runs
• select device pools
• upload your '.apk'
• specify test types
• JUnit, Espresso, Robotium etc..
• Upload test package separeted from '.apk'
• view reports
Upload .apk to DeviceFarm• Once uploading ".apk" , DeviceFarm detects
configuration automatically
Select devices• You need to select/create "device pools" which
you want to test
Supported devices• Show device list with "aws-cli": "aws
devicefarm list-devices | jq '.devices |map(select(.platform=="ANDROID")|.model)[]' | wc -l"
over 103 devices!
Supported devices• Command result: "aws devicefarm list-devices |
jq '.devices |map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"
Supported devices• Command result: "aws devicefarm list-devices | jq '.devices |
map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"
• "DROID Turbo" "DROID Ultra" "G Flex2" "G Pad 7.0"
• "G2" "G3" "G4" "Galaxy Light"
• "Galaxy Note 2" "Galaxy Note 3" "Galaxy Note 4" "Galaxy Note 5"
• "Galaxy Note" "Galaxy S3 LTE" "Galaxy S3 Mini" "Galaxy S3"
• "Galaxy S4 Active" "Galaxy S4 mini" "Galaxy S4" "Galaxy S5 Active"
• "Galaxy S5" "Galaxy S6 Edge" "Galaxy S6 Edge+" "Galaxy S6"
Supported devices• Command result: "aws devicefarm list-devices | jq '.devices
map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"
• "Galaxy Tab 2 10.1" "Galaxy Tab 3 10.1" "Galaxy Tab 3 7.0" "Galaxy Tab 3 Lite 7.0"
• "Galaxy Tab 4 10.1 Nook" "Galaxy Tab 4 7.0 Nook" "Kindle Fire HD 6" "Kindle Fire HDX 7"
• "Moto E - 2nd Gen" "Moto G - 2nd Gen" "Moto G" "Nexus 10"
• "Nexus 4" "Nexus 5" "Nexus 6" "Nexus 7 - 1st Gen"
• "Nexus 7 - 2nd Gen" "One M7" "One M8" "Optimus Fuel"
• "Optimus G Pro" "Optimus L70" "Optimus L90" "Xperia Z2"
• "Xperia Z3 Compact" "Xperia Z3"
Supported devices
But it does't have a lot of Japanese popular devices
Supported Test frameworks• Appium, Calabash
• JUnit, Espresso, Robotium etc…
Test reports an result summary
Test reports an result with screen shots of device
Run smoke test with "Built-in Fuzz Test"
• Same kind of monkey test
Integrate smoke-test into CI
Jenkins AWS DeviceFarm
Developer Slack
Integrate smoke-test into CI
GHE
Jenkins AWS DeviceFarmDeveloper
Slack
push codes
webhook
Build & upload apk
Fetch Test results
Post results
Receive results
Slack integration• At First, Jenkins DeviceFarm plugin was good
to me…
Slack integrationIt has option allows to store all test results
(Screenshot, monkey-test.log, .logcat)
Slack integration• But Jenkins DeviceFarm plugin didn't work
well.. (Failed to store the test results)
• Finally I used "jedi4ever/devicefarm-cli" tool written in Go
Use AWS DeviceFarm API• e.g. "aws-‐sdk-‐go":
• "func (*DeviceFarm) CreateUpload"
• Uploads an app or test scripts.
• "func (*DeviceFarm) ScheduleRun"
• Schedules a run.
• "func (*DeviceFarm) ListArtifacts"
• Get information about artifacts.
• An artifacts includes a s3 URL of its contents.
• Ref: https://godoc.org/github.com/aws/aws-sdk-go/service/devicefarm
Slack integration example
Round up• AWS DeviceFarm makes tests easily
• On many real/physical devices
• But it does't have a lot of Japanese popular devices
• Try also SauceLabs for comparison :)
• I want to use STF with many devices