Reliably Deploying Web Apps with Database
MigrationsPratik Vyas
To deploy new code
with a database migration
that takes 10 seconds to run per site
The Problem
in batches of five sites at a time
Then the 600th site would be running new code with
unpatched database for 20 minutes!
If we have to migrate 600 sites
run migrationfor 600th site
code deployed
danger zone
Deployments need to be handled more carefully
Create a new environment with the new code
Migrate sites one at a time by moving them to the new environment
An Intuitive Solution
build x
build x+1
site c
site d
site e
site b
site a
site f
site g
site h
site i
site j
● move to build x+1● run migrations
maintenance area(No HTTP requests)
site c
The Two Environments
The Solution - Part 1
Building an Environment
Use virtualenv and pip?
pip install would also need to compile C extensions
eg. lxml takes a lot of cpu time and memory
Makes creating new environmentsEXPENSIVE
Wheels are the new standard of python distribution (PEP 427)
Unzip the wheel in site-packages and the package is installed … like magic!
Build wheels on a different machine and watch a new environment get deployed
in a few seconds
Python Wheels
Cost of creating virtualenv is mostly unzipping
No compiler required on production server
PyPI availability is not a factor during deployments
Benefits of Build Step
buildfile.tar.gz├── scripts│ ├── virtualenv.py│ └── virtualenv_support│ ├── pip-1.5.2-py2.py3-none-any.whl│ └── setuptools-2.1-py2.py3-none-any.whl├── sites│ └── statics│ ├── css│ └── *.css│ ├── img│ └── *.png│ └── js│ └── *.js└── wheels └── *.whl
The Solution - Part 2
Running Processes
The new code will require us to start
a new wsgi process + workers
use Supervisor
Supervisor
Use Jinja and generate
process list per virtualenv
Protip: For adding/removing a group of processes, use reread and update instead of reload. It prevents non-related processes
from restarting.
The Solution - Part 3
Serving sites, both old and new
Serve unmigrated sites using old code,
migrated sites using new code.
nginx to the rescue
Configuring nginx awesomeness
Use Jinja to generate configuration and reload nginx
Fallback to old environment based on error condition
For performance, 404s should be cheap
upstream upstream-1 { server 127.0.0.1:8000 fail_timeout=0; }
upstream upstream-2 { server 127.0.0.1:8001 fail_timeout=0; }
server {
. . . .
location / {
error_page 404 = @upstream2handler;
proxy_redirect off;
proxy_intercept_errors on;
proxy_pass http://upstream-1;
}
location @upstream2handler {
proxy_redirect off;
proxy_intercept_errors on;
proxy_pass http://upstream-2;
}
}
Background Workers?
Stop executing scheduled jobs when in handover state
Preferably, use a fresh broker for every build
Track pending jobs per site(we prepend sitename to queue name for this)
The Final Picture
build x
build x+1
site c
site d
site e
site b
site a
site f
site g
site h
site i
site j
● Delay enqueuing scheduled tasks.● Continue if no pending jobs, else
retry after 60 seconds.
● Run migrations.● Move to build x+1.
site c
maintenance area(No HTTP requests)
ध यवाद@pdvyas, @frappe_io
Recommended
