POST-MORTEM DEBUGGINGAND WEB DEVELOPMENT

Alessandro Molina@__amol__

amol@turbogears.org

Who am I

● CTO @ Axant.it, mostly Python company

(with some iOS and Android)

● TurboGears2 devteam member

● Contributions to web world python libraries

○ MING MongoDB ODM

○ ToscaWidgets2

○ Formencode

Why

● Debugging is a core part of the

development process.

● You can try to prevent issues as much as

possible, but users will find new ones.

● Gathering informations to replicate issues

is required to fix them

Debugging

At least know you have an issue.

● Debugging is the “process of finding and

reducing the number of bugs”

● Users are already doing half of the work.

● When users find a bug for you, make sure

to be aware that they found it

● Log it or it will be forgotten

Log what you really need● Log to replicate

○ Then you are able to write tests that actually verify

the issue has been solved.

● You probably want to log:

○ WSGI Environ

○ Traceback

○ Last N stack frames local variables

○ Request Headers & Body (when size permits)

Log in an useful way● Log data in an easy to use way

○ Log request in a way it’s quick to replay it

○ Log in a computer friendly format for test units

○ Request dump is good: netcat to replay it and is

already understood by computers.

● Organize your informations

○ Use a tool to group and avoid duplicates

○ Know what got solved and what didn’t

○ Log by email, so you don’t have to check yourself

Log or...

But log in a separate thread or...

Crash Report● Many middlewares and frameworks

provide exception reporting by email:

○ pyramid_exclog (Pyramid)

○ WebError (Pylons)

○ BackLash (TurboGears)

○ Django & Flask, framework provided

● Logging module has what you need:

Logger.exception + handlers.SMTPHandler

Ready-Made Email Reporting● Looking for a stand-alone solution?

○ Backlash can be used by itself, not bound to

TurboGears.

○ Only dependency is “WebOb”

● Supports both Python2 and Python3

from backlash.trace_errors import EmailReporter

email_reporter = EmailReporter(smtp_server="localhost", error_email="email@host.com", from_address="errors@host.com")app = backlash.TraceErrorsMiddleware(app, [email_reporter])

Try Sentry● Gathers data and detects duplicates

● Provides “Reply Request” button

● Mark exceptions as solved to keep track

● Can log multiple events, not only

exceptions

from backlash.trace_errors.sentry import SentryReporter

sentry_reporter = SentryReporter(sentry_dsn="http://public:secret@example.com/1")app = backlash.TraceErrorsMiddleware(app, [sentry_reporter])

Sentry

See what’s happening now

● On development and test environments

receiving errors by email is not convenient

● In case of freezes you don’t get an

exception that can be reported

● Some problems need to be

troubleshooted on the fly

Interactive Debugger● Python provides built-in Post Mortem

debugging through pdb module

● While it’s ready to use and works great it’s

not the most comfortable tool when

working on web development

try: return app(environ, start_response)except: import pdb pdb.post_mortem()

Browser Debugger● Many frameworks have browser debugger

○ Pyramid DebugBar

○ Werkzeug DebuggedApplication

○ TurboGears ErrorWare

● BackLash provides a stand-alone version

of the Werkzeug debugger.

import backlashapp = backlash.DebuggedApplication(app)

Werkzeug/BackLash Debugger

Browser Debugger Console● Browser debuggers usually implement a

special URL with an interactive console

that runs in the context of the application.

● Use it to see what’s happening right now

○ for threadId, stack in sys._current_frames().items()

● Try /__console__ on Werkzeug or Backlash

Production Debugging● In-browser debugger must be disabled on

production environments

○ Unless you are PythonAnyware you want to block

people from running code on your servers.

● So we are back again at the starting point

○ How do I debug issues that don’t provide an

exception traceback on production?

Attaching to running processesTo inspect running applications you can rely

on tools like ispyd and pyrasite that are able

to attach to a running python process

from ispyd.plugins.wsgi import WSGIApplicationWrapperapp = WSGIApplicationWrapper(app)

ispyd(wsgi:18630) requests

No active transactions.

(wsgi:18630) requests

==== 67 ====

thread_id = 140735076232384

start_time = Mon Apr 9 21:49:54 2012

duration = 0.013629 seconds

HTTP_HOST = 'localhost:5000'

QUERY_STRING = ''

…

File: "wsgi.py", line 19, in hello

time.sleep(0.05)

Pyrasite

Not only Debugging

● Debugging tools can also help in finding

and solving performance issues

● When a request is taking a lot of time, just

attach ipsyd and see what it’s happening

● There are specific tools that can

proactively notify you of slow requests

Slow Requests Tracing

● Backlash provides slow request tracing for

any WSGI framework.

● Dogslow: a Django tool created by

Bitbucket guys to monitor slow requests

● Both Notify you by email, no need to check

○ Backlash can also use any backlash reporter, for

example by reporting slow requests on sentry.

Slow Requests Tracing

● Setting up slow request reporting is

incredibly simple both with Backlash and

Dogslow

import backlashfrom backlash.trace_errors import EmailReporter

email_reporter = EmailReporter(smtp_server="localhost", error_email="email@host.com", from_address="errors@host.com")

app = backlash.TraceSlowRequestsMiddleware(app, [email_reporter])

New Relic● Full stack tracing

○ Controllers

○ SQLAlchemy queries

○ Background Tasks

○ External Services

○ Groups informations by controller, not URL

● On-Demand profiling

○ Turn On / Off controllers profiling from remote

New Relic

Slow requests stack trace might not be enough

Profiling

● Full profiling has a big cost, so it is usually

constrained to development environment

● If you want to profile on production,

perform sampling, don’t profile every

request

● Use a Low Overhead profiler like PLOP, not

built-in profile module.

Questions?