Reliably Deploying Web Apps with Database Migrations

  • View

  • Download

Embed Size (px)




  • 1. Reliably Deploying Web Apps with Database Migrations Pratik Vyas

2. To deploy new code with a database migration that takes 10 seconds to run per site The Problem 3. in batches of five sites at a time Then the 600th site would be running new code with unpatched database for 20 minutes! If we have to migrate 600 sites run migration for 600th site code deployed danger zone 4. Deployments need to be handled more carefully 5. Create a new environment with the new code Migrate sites one at a time by moving them to the new environment An Intuitive Solution 6. build x build x+1 site c site d site e site b site a site f site g site h site i site j move to build x+1 run migrations maintenance area (No HTTP requests) site c The Two Environments 7. The Solution - Part 1 8. Building an Environment Use virtualenv and pip? pip install would also need to compile C extensions eg. lxml takes a lot of cpu time and memory Makes creating new environments EXPENSIVE 9. Wheels are the new standard of python distribution (PEP 427) Unzip the wheel in site-packages and the package is installed like magic! Build wheels on a different machine and watch a new environment get deployed in a few seconds Python Wheels 10. Cost of creating virtualenv is mostly unzipping No compiler required on production server PyPI availability is not a factor during deployments Benefits of Build Step 11. buildfile.tar.gz scripts virtualenv_support pip-1.5.2-py2.py3-none-any.whl setuptools-2.1-py2.py3-none-any.whl sites statics css *.css img *.png js *.js wheels *.whl 12. The Solution - Part 2 13. Running Processes The new code will require us to start a new wsgi process + workers use Supervisor 14. Supervisor Use Jinja and generate process list per virtualenv Protip: For adding/removing a group of processes, use reread and update instead of reload. It prevents non-related processes from restarting. 15. The Solution - Part 3 16. Serving sites, both old and new Serve unmigrated sites using old code, migrated sites using new code. nginx to the rescue 17. Configuring nginx awesomeness Use Jinja to generate configuration and reload nginx Fallback to old environment based on error condition For performance, 404s should be cheap 18. upstream upstream-1 { server fail_timeout=0; } upstream upstream-2 { server fail_timeout=0; } server { . . . . location / { error_page 404 = @upstream2handler; proxy_redirect off; proxy_intercept_errors on; proxy_pass http://upstream-1; } location @upstream2handler { proxy_redirect off; proxy_intercept_errors on; proxy_pass http://upstream-2; } } 19. Background Workers? Stop executing scheduled jobs when in handover state Preferably, use a fresh broker for every build Track pending jobs per site (we prepend sitename to queue name for this) 20. The Final Picture build x build x+1 site c site d site e site b site a site f site g site h site i site j Delay enqueuing scheduled tasks. Continue if no pending jobs, else retry after 60 seconds. Run migrations. Move to build x+1. site c maintenance area (No HTTP requests) 21. @pdvyas, @frappe_io