Embed Size (px)
DESCRIPTION
PHP User Group Minsk #4
Citation preview
Yet Another Continuous Integration Story
Who I Am● Anton Serdyuk
● Technical Leader at Startup Labs, Inc.
● @anton_serdyuk
● Skype: serdyuk.anton
Disclaimer
Disclaimer
Projects are different● Technical requirements
● Business requirements
● Length
● Budget
● Load
● Data size
● etc
Disclaimer
Teams are different● Level
● Size
● Structure
● etc
Disclaimer
There is no silver bullet● Worked for us != will work for you
● Adopt patterns and principles rather than the full
process
Who We Are
Who We Are
Project● b2b system
● Unexpected downtime is very expensive
● Bugs in production are also very expensive because of
the reputation loss
● Small amount of traffic (But expensive)
● 1.5 years
● 4-15 developers (+ layout, + qa)
● 15 Gb Database (Grows not very fast)
Who We Are
Technologies● PHP 5.4
● Java 7
● PostgreSQL
● RabbitMQ
Who We Are
Frameworks● Hibernate
● JUnit
● Symfony2
● Silex
● Behat
● PHPUnit
Goals and Principles
Goals and Principles
Agile● Small changes
● Frequent releases
● Early feedback
What Is NOT Our Way
What Is NOT Our Way
Git Flow
What Is NOT Our Way
Feature Branches
master
feature2
feature1
What Is NOT Our Way
Feature BranchesActive feature branches usage example
What Is NOT Our Way
Release Branches
master
1.0.x
1.1.x
What Is NOT Our Way
master/staging/develop branches
master
develop
What Is NOT Our Way
Release Tags
master
v1.1.2v1.1.1
What IS Our Way
What IS Our Way
Continuous Integration● Push every day
● Build every commit
● Run all tests on every build
● Fix red builds quickly
What IS Our Way
One Branch
master
What IS Our Way
Branch by Abstraction● Commit switch-off for this feature
● Develop, integrate
● … your code in production now, but switched-off
● Develop, integrate
● Test
● Switch-on this feature in production
What IS Our Way
Autotests● QAs write behat scenarios
● Developers implement
● Every feature covered by behat scenarios
● Great regression tests suite - easy refactoring, less
bugs
● + ~100% development cost
What IS Our Way
Every Build Is Release Candidate● Do not push if state after commit is not suitable for
production
What IS Our Way
Some Numbers● ~1000 behat scenarios (unit tests are not counted)
● ~15 minutes build duration
● 2 build servers (Intel Xeon, 32 Gb RAM, SSD)
● 5-6 build agents for tests (VMs)
● 3 build agents for build phase (VMs)
● ~15 builds per day
● 2 releases per week
Build Process
Build Process
Overview
build automatic tests manual tests load and prod migrate tests production
Build Process
Overview
testagentstest
agentstest
agentsbuild agentgit
repository
push
bamboo
pull
build agents
build
aptrepository
artifacts
testagents
puppetmasterdevelopers
release engineer (architect/admin)
tune
test
QAserver
preprod prod
Build Process
Use The Same Artifacts
testagentstest
agentstest
agentsbuild agentbuild
agents
aptrepository
artifacts
testagents
QAserver
preprod prod
Build Process
Autoconfigure All Environments
testagentstest
agentstest
agentstest
agents
puppetmaster
release engineer (architect/admin)
tune
QAserver
preprod prod
Build Process
Deploy by Pressing a Button
testagentstest
agentstest
agents
bamboo
testagents
QAserver
preprod prod
Build Phase
Build Phase
Goals● Create artifacts which will be used in all phases
● Run unit tests
Build Phase
Technologies● Phing
● Maven
● deb-packages (dpkg-buildpackage)
● PHPUnit
● JUnit
Build Phase
Example
Automatic Test Phase
Automatic Test Phase
Goals● Run functional tests
Automatic Test Phase
Deploy to Completely Clean System
agent 1
Automatic Test Phase
Run Tests in Parallel
behat process 1 application 1
...
agent 2
behat process 2 application 2
agent 5
behat process 5 application 5
Automatic Test Phase
Run Tests in Parallelfeature 1
feature 2
feature 3
feature 4
feature 5
feature 6
feature 7
feature 8
feature 9
feature 50
feature 51
...
job 01
job 02
job 03
Job 15
agent 1
agent 2
agent 3...
Automatic Test Phase
Run Tests in Parallelgithub: StartupLabs/behat-partial-runner
● bin/behat --count 15 --current 1
● bin/behat --count 15 --current 2
● bin/behat --count 15 --current 3
● bin/behat --count 15 --current 4
● bin/behat --count 15 --current 5
● …
● bin/behat --count 15 --current 15
Automatic Test Phase
Write Fast Tests● Continuously speed-up your tests
● Use curl instead of Selenium
● Put initial data directly to DB
● etc
Automatic Test Phase
Write Fast TestsMerge scenarios which do not modify data
Feature: Posts pagination
Background:
Given I am logged in as admin
And there is many posts
Scenario: Pagination works
Given I am at the posts list
Then I should see pagination
Scenario: Link “last” works
Given I am at the posts list
When I click at the “last” link
Then I should see last page
Feature: Posts pagination
Background:
Given I am logged in as admin
And there is many posts
Scenario: All pagination tests
# Scenario: Pagination works
Given I am at the posts list
Then I should see pagination
# Scenario: Link “last” works
When I click at the “last” link
Then I should see last page
Automatic Test Phase
Write Fast TestsTranslate outline to single scenarios with TableNode
Feature: Post validation
Background:
Given I am logged in as admin
And I am on post add page
Scenario Outline: Validation works
When I enter <value> into <field>
Then I should see error <error>
Examples:
| field | value | error |
| title | | Can not be empty |
| tags | %%%% | Should be letters |
Feature: Post validation
Background:
Given I am logged in as admin
And I am on post add page
Scenario: Validation works
Then I should see such errors with such input
| field | value | error |
| title | | Can not be empty |
| tags | %%%% | Should be letters |
server
Automatic Test Phase
Write Fast Tests
@parallel
Feature: Login
Background:
Given I am registered user
| username | alex |
| password | 123 |
Scenario: I can login
Given I am at the login page
When I fill form with
| username | alex |
| password | 123 |
Then I should be logged in
Parallel test execution
● Many problems with race
conditions
● Not worked for us :(
behat process 1
behat process 2application
alex = alex1
alex = alex2
Manual Test Phase
Manual Test Phase
Goals● Run manual tests
● Place where QA see new functionality
Manual Test Phase● Deploy to completely clean system
Load and Prod Migrate Tests
Load and Prod Migrate Tests
Load Tests Goals● Catch race conditions
● Catch serious performance degradation when database
grows or after adding new functionality
Load and Prod Migrate Tests
Prod Migrate Tests Goals● Test migrations on real data
● Test migrations time on real data
Load and Prod Migrate Tests● Migrate load test env to new version
● Rollback migrate test env to production snapshot
Deploy to Production
Deploy to Production● nb-prod - small production copy used by other teams for
integration purposes
Problems
Problems
Red Builds for a Long Time● QA can not deploy new version to staging
● New commits make new tests red like a snowball
● Nobody knows who are responsible for all those failed
tests
Solutions:
● If build can not be fixed in 15 minutes - revert commit
● Autoreverts on red builds (?)
Problems
Long Builds● Nobody wants to wait green build
● Nobody cares about green build after commit
● Red builds for a long time
Solutions:
● Speedup tests
● Parallel build
● Add more build servers
Problems
Blinking Failures● Failures are trustless - “it is not my fault - it is blinking,
just do rebuild”
● Time consuming
Solutions:
● Eliminate them ASAP
● Do not just run “Rebuild failed jobs”
Key Points, Again
Key Points, Again
Autotests● There is no sense in CI without tests
● Easy refactoring
● Less bugs
Key Points, Again
Fast Builds
● Fast tests
● Parallel run
● Hacks in tests
● etc
● Long builds => More changes => Red builds
Key Points, Again
One branch● There are no builds from other branches
● There are no hotfixes
● Easy deploy and build pipeline
Todo
Todo● Rollback strategy
● Use Bamboo deployment
Inspired By● Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment Automation (Jez
Humble, David Farley)
● http://timothyfitz.com/
Questions?
