Embed Size (px)
Citation preview
ASTOpportunities and Threats
ITGM, Dec 2016
RDD
TDD
Slap this sh*t & release
ITGM, Dec 2016
Code Test
Test
Test
ITGM, Dec 2016
Code Test
Test
Test
def f(a,b): return a+b
5 == f(2,3)
0 == f(2,-2)
1 == f(1,0)
ITGM, Dec 2016
Code Test
Test
Test
def f(a,b): return a+b
5 == f(2,3)
0 == f(2,-2)
1 == f(1,0)
Test 1e15 == f(1,1e15-1)
ITGM, Dec 2016
DIS
ITGM, Dec 2016
>>> def foo(): ... a = 2 ... b = 3 ... return a + b ...
ITGM, Dec 2016
>>> import dis >>> dis.dis(foo) 2 0 LOAD_CONST 1 (2) 3 STORE_FAST 0 (a)
3 6 LOAD_CONST 2 (3) 9 STORE_FAST 1 (b)
4 12 LOAD_FAST 0 (a) 15 LOAD_FAST 1 (b) 18 BINARY_ADD 19 RETURN_VALUE
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
Zephyr ASDLParser/Python.asdl
ITGM, Dec 2016
Python/Python-ast.cimport _ast
ITGM, Dec 2016
Lib/ast.pyimport ast
ITGM, Dec 2016
• AST
• Add, And, Assert …
• NodeVisitor
• NodeTransformer
ITGM, Dec 2016
class DepthVisitor(ast.NodeVisitor): depth = 0
def generic_visit(self, node): parent = node.parent if parent: while parent: self.depth += 1 parent = parent.parent ast.NodeVisitor.generic_visit(self, node)
ITGM, Dec 2016
• parse
• copy_location
• fix_missing_locations
• dump
• increment_lineno
ITGM, Dec 2016
Модификация AST
ITGM, Dec 2016
class HackedImporter: def load_module(self, name): # do magic stuff return module
sys.path_hook.insert(0, HackedImporter)
ITGM, Dec 2016
https://github.com/alifanov/ast-spbpython-example
ITGM, Dec 2016
class ReturnIncrement(ast.NodeTransformer): def visit_Return(self, node): node.value = (ast.BinOp( left=node.value, op=ast.Add(), right=ast.Num(n=1) ) return node
ITGM, Dec 2016
def transform_with(transformer): def transform_decorator(func): node = func_to_node(func) node = transformer().visit(node) node = ast.fix_missing_locations(node) func = node_to_func(node, func) return func return transform_decorator
ITGM, Dec 2016
@transform_with(ReturnIncrementer) def f(a,b): return a+b
f(2,2) # 5
ITGM, Dec 2016
Threats
• Debug glitch
• Errors with multiple applies
• Strange errors
• Compiler version dependence
ITGM, Dec 2016
>>> parseprint("func(a, b=c, *d, **e)") # Python 3.4 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[Name(id='a', ctx=Load())], keywords=[keyword(arg='b', value=Name(id='c', ctx=Load()))], starargs=Name(id='d', ctx=Load()), # gone in 3.5 kwargs=Name(id='e', ctx=Load()))), # gone in 3.5 ])
>>> parseprint("func(a, b=c, *d, **e)") # Python 3.5 Module(body=[ Expr(value=Call(func=Name(id='func', ctx=Load()), args=[ Name(id='a', ctx=Load()), Starred(value=Name(id='d', ctx=Load()), ctx=Load()) # new in 3.5 ], keywords=[ keyword(arg='b', value=Name(id='c', ctx=Load())), keyword(arg=None, value=Name(id='e', ctx=Load())) # new in 3.5 ])) ])
ITGM, Dec 2016
Opportunities• Constant propagation
• Autologger
• Autoprofiler
• Macros (karnickel)
• Extend language
• MacroPy
• PonyORM
• Linters (auto codereview)ITGM, Dec 2016
>>> from macropy.tracing import macros, trace >>> trace[[x*2 for x in range(3)]] range(3) -> [0, 1, 2] x*2 -> 0 x*2 -> 2 x*2 -> 4 x*2 for x in range(3) -> [0, 2, 4] [0, 2, 4]
Macros
ITGM, Dec 2016
welcome = gettext("Welcome, {}!").format(user_name)
welcome = gettext("Welcome, {}!".format(user_name))
Linters
ITGM, Dec 2016
https://github.com/edx/edx-lint/blob/master/edx_lint/pylint/i18n_check.py
Linters
ITGM, Dec 2016
TRANSLATION_FUNCTIONS = set([ '_', 'gettext', 'ngettext', 'ngettext_lazy', 'npgettext', 'npgettext_lazy', 'pgettext', 'pgettext_lazy', 'ugettext', 'ugettext_lazy', 'ugettext_noop', 'ungettext', 'ungettext_lazy', ])
def visit_callfunc(self, node): if not isinstance(node.func, astroid.Name): # It isn't a simple name, can't deduce what function it is. return
if node.func.name not in self.TRANSLATION_FUNCTIONS: # Not a function we care about. return
if not self.linter.is_message_enabled(self.MESSAGE_ID): return
first = node.args[0] if isinstance(first, astroid.Const): if isinstance(first.value, basestring): # The first argument is a constant string! All is well! return
# Bad! self.add_message(self.MESSAGE_ID, args=node.func.name, node=node)
ITGM, Dec 2016
Tools• astdump
• astviewer
• ast_tool_box
• astroid
• baron
• redbaron
• astor
• meta
• astmonkey
ITGM, Dec 2016
Recommendation unit tests system
ITGM, Dec 2016
Pythoscope
ITGM, Dec 2016
Smother
ITGM, Dec 2016
app.views:add,app/tests.py::FunctionTest::test_add app.views:fact,app/tests.py::FunctionTest::test_fact app.views:hello,app/tests.py::FunctionTest::test_hello
ITGM, Dec 2016
Code1 Test1
Code1 Test2
Code1 Test3
Code2 Test1
Code2 Test2
ITGM, Dec 2016
def test_default_ordering(self): class OrderingListView(generics.ListAPIView): queryset = OrderingFilterModel.objects.all() serializer_class = OrderingFilterSerializer filter_backends = (filters.OrderingFilter,) ordering = ('title',) ordering_fields = ('text',)
view = OrderingListView.as_view() request = factory.get('') response = view(request) self.assertEqual( response.data, [ {'id': 3, 'title': 'xwv', 'text': 'cde'}, {'id': 2, 'title': 'yxw', 'text': 'bcd'}, {'id': 1, 'title': 'zyx', 'text': 'abc'}, ] ) # not in context
ITGM, Dec 2016
def fact(a): if a == 0: return 1 return a * fact(a - 1)
def test_fact(self): self.assertTrue(fact(0) == 1) self.assertTrue(fact(1) == 1) self.assertTrue(fact(2) == 2) self.assertTrue(fact(3) == 6) self.assertTrue(fact(4) == 24)
ITGM, Dec 2016
https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
ITGM, Dec 2016
Edges
ITGM, Dec 2016
BinOpLeftNum
ITGM, Dec 2016
Child Edge Parent Freq
Num left BinOp 12
Num value Assign 20
ITGM, Dec 2016
Depth
ITGM, Dec 2016
Builtin function
ITGM, Dec 2016
https://github.com/alifanov/testedio
ITGM, Dec 2016
Roadmap
ITGM, Dec 2016
Improve searchmodules names as features
ITGM, Dec 2016
Import github projectsunit-test monitoring
ITGM, Dec 2016
Fuzzy recommendations
hypothesis*pylint plugin
ITGM, Dec 2016
Performance optimizationmap -> list comprehension
*pylint plugin
ITGM, Dec 2016
Other languagesJavaScript
ITGM, Dec 2016
Real TDDtests -> code generation
ITGM, Dec 2016
Linkshttps://github.com/mkwiatkowski/pythoscope http://smother.readthedocs.io/en/latest/index.html http://testmon.org/ https://www.cosc.canterbury.ac.nz/research/reports/HonsReps/2015/hons_1508.pdf http://is.ifmo.ru/diploma-theses/2013/bachelor/rost/rost.pdf https://habrahabr.ru/post/65944/ http://pyke.sourceforge.net/
ITGM, Dec 2016
Contacts
Telegram: @jetbootsmaker
Alexander Lifanov
ITGM, Dec 2016
