Embed Size (px)
Citation preview
Django Performance Recipes
September 2015
Jon Atkinson Technical Director FARM Digital
Me
• I am not a very good programmer.
• I’m quite a good problem solver.
• I have made a lot of mistakes.
• I prefer valuable solutions.
My Context
• “Fast-moving agency environment”.
• We often participate in launches; demand is spiky.
• Complex software.
Shared Context
• Traffic is unpredictable.
• We probably host in the cloud.
• The Django ecosystem.
Performance Matters
Speed is all about perception, but:
0-100ms Instant!100-300ms Small delay.
300-1000ms Something is happening.1000ms+ Likely task switch.
10000ms+ Abandoned task.
Rule of thumb
50% of users will abandon a task after 3 seconds.
Environment
A web server, talking to:
A managed python process, talking to:
A cache, and a database.
Request CycleWeb Server
ORM
Middleware
Views + Templates
Web Server
Tools
!
ORM
• Optimising your database is a separate presentation entirely.
• There are no ‘slow’ databases any more.
• In a read-heavy environment, caching Querysets is a huge advantage.
ORM
django-cachalot
Caches your Django ORM queries and automatically invalidates them.
ORM
$ pip install django-cachalot
INSTALLED_APPS += (‘cachalot’)
settings.CACHALOT_ENABLED = True
ORMfrom django.conf import settings
from django.test.utils import override_settings
with override_settings(CACHALOT_ENABLED=False):
# SQL queries are not cached in this block
@override_settings(CACHALOT_CACHE=‘second_cache')
def your_function():
# What’s in this function uses another cache
# Globally disables SQL caching until you set it back to True
settings.CACHALOT_ENABLED = False
ORM
• A few other data access tips:
• Is your database doing DNS lookups?
• Do you have a connection timeout set? The default is 0, and setup/teardown costs time.
settings.DATABASES[‘…’][‘CONN_MAX_AGE’] = 600
Middleware
Middleware is dumb.
Middleware
Middleware
• Think hard. Milliseconds add up.
• Middleware (and context processors!) often becomes a dumping ground for common features.
Middleware
Middleware is helpful.
Middleware
django.middleware.http.ConditionalGetMiddleware
Optimises GET requests from modern browsers
django.middleware.http.GZipMiddleware
Compresses responses. But be aware of BREACH!
Middleware
$ pip install django-cprofile-middleware
“Once you've installed it, log in as a user who has staff privileges and add ?prof to any URL to see the profiler's stats.”
eg. http://localhost:8000/foo/?prof.
Middleware7986 function calls (7850 primitive calls) in 1.725 CPU seconds
Ordered by: internal time, call count
List reduced from 392 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
2 1.570 0.785 1.570 0.785 /…/django/db/backends/__init__.py:36(_commit)
15 0.043 0.003 0.043 0.003 /…/linecache.py:68(updatecache)
1 0.020 0.020 0.027 0.027 /…/django/contrib/auth/models.py:1(<module>)
12 0.014 0.001 0.030 0.002 /…/django/utils/importlib.py:18(import_module)
1013 0.010 0.000 0.010 0.000 /…/posixpath.py:56(join)
Views & Templates
• View performance problems are usually obvious:
• Avoid nested loops (especially when generating QuerySets!)
• Cache where you can.
• Always return at the earliest possible moment.
Views & Templates
• Templates are more interesting.
• It’s easy to duplicate ORM calls already made in the view.
• It’s easy to traverse relationships in the template language.
• Templates are loaded from disk by default.
Views & Templatesclass Person(models.Model):
def friends(self):
# Hit the database here for something complex…
return friends
# View:
if person.friends():
# Do something here…
# Template:
{% for friend in person.friends %}
Views & Templatesfrom django.utils.functional import cached_property
@cached_property
def friends(self):
# Hit the database here for something complex…
return friends
Views & Templates• Caching template fragments is very powerful.
• Sometimes you need to do something expensive in a template, but:
{% load cache %}
{% cache 500 sidebar %}
… do something expensive here …
{% endcache %}
Views & Templates
• Where do your templates actually live?
• Cloud disk performance can be erratic.
Views & Templates# Default setting.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)
Views & TemplatesTEMPLATE_LOADERS = (
('django.template.loaders.cached.Loader', (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)),
)
nginx
This is my “one weird tip”…
… with a trade-off.
uwsgi_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m;
server { listen 80; server_name example.com; …
uwsgi_cache_use_stale error timeout invalid_header http_500; uwsgi_cache_valid 10m;
location / { include uwsgi_params; uwsgi_pass unix:///tmp/example.sock; uwsgi_cache cache; uwsgi_cache_key $scheme:$host$request_uri:$request_method; uwsgi_cache_bypass $http_pragma $http_authorization; uwsgi_no_cache $http_pragma $http_authorization; }
nginx
Tools• The tool ecosystem is richer now than ever
before.
$ pip install django-debug-toolbar$ pip install dogslow
• Measure twice, cut once.
• Remember, Schrödinger’s web app.
Tools
Tools
import newrelic.agent
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# This needs to be called after we bootstrapped the application
# otherwise the settings wouldn't be configured
from django.conf import settings # noqa
if hasattr(settings, 'NEWRELIC_CONFIG'):
newrelic.agent.initialize(settings.NEWRELIC_CONFIG \
getattr(settings, 'NEWRELIC_ENVIRONMENT', None))
application = newrelic.agent.WSGIApplicationWrapper(application)
Meta 1
Performance can be personal.
Meta 2
Meta 2
Meta 3
Performance is of huge value.
10x programmers probably don’t exist.
Keep your eyes up.
Concentrate on where the value lies.
