Upload
graham-dumpleton
View
187
Download
5
Embed Size (px)
Citation preview
Hear no evil, see no evil, patch no evil: Or, how to
monkey-patch safely.
Graham Dumpleton @GrahamDumpleton
PyCon Australia - August 2016
Decorators are useful!
Decorators are easy to implement?
Are you sure?
Typical decorator.
def function_wrapper(wrapped): def _wrapper(*args, **kwargs): return wrapped(*args, **kwargs) return _wrapper
@function_wrapper def function(): pass
This breaks introspection.
__name__ and __doc__ attributes are not
preserved.
Doesn’t @functools.wraps() help?
import functools
def function_wrapper(wrapped): @functools.wraps(wrapped) def _wrapper(*args, **kwargs): return wrapped(*args, **kwargs) return _wrapper
@function_wrapper def function(): pass
No, it doesn’t solve all problems.
Still issues with: introspection, wrapping decorators
implemented using descriptors, and more.
http://blog.dscpl.com.auQuick Link: Decorators and monkey patching.
Complicated details removed ….
Get all the details at:
Please try not to implement decorators
yourself.
What is the solution?
Use ‘wrapt’.
Basic decorator.
import wrapt
@wrapt.decorator def pass_through(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
@pass_through def function(): pass
Universal decorator.import wrapt import inspect
@wrapt.decorator def universal(wrapped, instance, args, kwargs): if instance is None: if inspect.isclass(wrapped): # Decorator was applied to a class. return wrapped(*args, **kwargs) else: # Decorator was applied to a function or staticmethod. return wrapped(*args, **kwargs) else: if inspect.isclass(instance): # Decorator was applied to a classmethod. return wrapped(*args, **kwargs) else: # Decorator was applied to an instancemethod. return wrapped(*args, **kwargs)
Bonus feature of wrapt if using multithreading.
Synchronise function calls.
from wrapt import synchronized
@synchronized # lock bound to function1 def function1(): pass
@synchronized # lock bound to function2 def function2(): pass
Methods of classes as well.from wrapt import synchronized
class Class(object):
@synchronized # lock bound to instance of Class def function_im(self): pass
@synchronized # lock bound to Class @classmethod def function_cm(cls): pass
@synchronized # lock bound to function_sm @staticmethod def function_sm(): pass
Synchronise block of code.from wrapt import synchronized
class Object(object): @synchronized def function_im_1(self): pass
def function_im_2(self): with synchronized(self): pass
def function_im_3(self): with synchronized(Object): pass
Don’t trust me when I say you should use wrapt?
Potential candidate for being included in the Python
standard library.
So it must be awesome.
Primary purpose of the wrapt package wasn’t as way to build decorators.
Primary reason for existence of wrapt was to
help with monkey patching.
Decorators rely on similar principles to monkey
patching.
Before decorators.# python 2.4+
@function_wrapper def function(): pass
# python 2.3
def function(): pass function = function_wrapper(function)
Decorators are applied when code is defined.
Monkey patching is performed after the fact,
… and can’t use the decorator syntax
Why monkey patch?
• Fix bugs in code you can’t modify.
• Replace/mock out code for testing.
• Add instrumentation for monitoring.
Monkey patching with wrapt.# example.py
class Example(object): def name(self): return 'name'
# patches.py
import wrapt
def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
from example import Example
wrapt.wrap_function_wrapper(Example, 'name', wrapper)
Don’t patch it yourself.# patches.py
import wrapt
@wrapt.decorator def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
from example import Example
# DON’T DO THIS.
Example.name = wrapper(Example.name)
Direct patching of methods breaks in
certain corner cases.
Let wrapt apply the wrapper for you.
Avoiding imports.
# patches.py
import wrapt
@wrapt.patch_function_wrapper('example', 'Example.name') def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
What about testing, where we do not want permanent patches?
Mock alternative.# example.py
class Storage(object): def lookup(self, key): return 'value' def clear(self): pass
# tests.py
import wrapt
@wrapt.transient_function_wrapper('example', 'Storage.lookup') def validate_storage_lookup(wrapped, instance, args, kwargs): assert len(args) == 1 and not kwargs return wrapped(*args, **kwargs)
@validate_storage_lookup def test_method(): storage = Storage() result = storage.lookup('key') storage.clear()
What if we need to intercept access to single
instance of an object?
Transparent object proxy.
# tests.py
import wrapt
class StorageProxy(wrapt.ObjectProxy): def lookup(self, key): assert isinstance(key, str) return self.__wrapped__.lookup(key)
def test_method(): storage = StorageProxy(Storage()) result = storage.lookup(‘key') storage.clear()
Beware though of ordering problems when
applying monkey patches.
Import from module.# example.py
def function(): pass
# module.py
from example import function
# patches.py
import wrapt
@wrapt.patch_function_wrapper('example', 'function') def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
Plus importing and patching of modules that the
application doesn’t need.
Post import hooks (PEP 369).
# patches.py
import wrapt
def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs)
@wrapt.when_imported('example') def apply_patches(module): wrapt.wrap_function_wrapper(module, 'Example.name', wrapper)
Better, but still requires patches module to be
imported before anything else in the main
application script.
Need a way to trigger monkey patches without
modifying application code.
The autowrapt package.
pip install autowrapt
Bundle patches as module.
from setuptools import setup
PATCHES = [ 'wsgiref.simple_server = wrapt_wsgiref_debugging:apply_patches' ]
setup( name = 'wrapt_wsgiref_debugging', version = '0.1', py_modules = ['wrapt_wsgiref_debugging'], entry_points = {‘wrapt_wsgiref_debugging': PATCHES} )
Patch to time functions.from __future__ import print_function from wrapt import wrap_function_wrapper from timeit import default_timer
def timed_function(wrapped, instance, args, kwargs): start = default_timer() print('start', wrapped.__name__) try: return wrapped(*args, **kwargs) finally: duration = default_timer() - start print('finish %s %.3fms' % ( wrapped.__name__, duration*1000.0))
def apply_patches(module): print('patching', module.__name__)
wrap_function_wrapper(module, 'WSGIRequestHandler.handle', timed_function)
Enabling patches.
$ AUTOWRAPT_BOOTSTRAP=wrapt_wsgiref_debugging $ export AUTOWRAPT_BOOTSTRAP
$ python app.py
patching wsgiref.simple_server start handle 127.0.0.1 - - [14/Jul/2016 10:18:46] "GET / HTTP/1.1" 200 12 finish handle 1.018ms
Packaging of patches means they could technically be
shared via PyPi.
Eg: instrumentation for monitoring.
Reasons to use wrapt.
• Create better decorators.
• Awesome thread synchronisation decorator.
• Safer mechanisms for monkey patching.
@GrahamDumpleton
wrapt.readthedocs.io
blog.dscpl.com.au