Python’bindings’in’BornAgain’ - apps.jcns.fz ...apps.jcns.fz-juelich.de/doku/sc/_media/pospelov_pythonbindings.pdfBasic software architecture Main’features:’ o Programming’language’C++

Python bindings in BornAgain

Gennady Pospelov

Scien8fic Compu8ng at MLZ

October, 24 2013, ILL, Grenoble, France

BornAgain is a a free and open source soJware package to simulate and fit small angle scaMering at grazing incidence. o  see talk of Walter Van Herck

Scope

October, 24, 2013, ILL, Grenoble Python bindings in BornAgain, Gennady Pospelov 02

Basic software architecture

Main features: o Programming language C++ o Core libraries consist of 30k lines of code o Python bindings o Mac, Linux, Windows

o Depend on 4 well established libraries gsl, boost, eigen, Ww3

Why C++? o  Performance. o  Type safety. o  External dependencies. o  Prospec8ve use of GPU, parallel

compu8ng. o  Developers background.

libCore Samples and algorithms

libFit minimizers

BornAgain

Python bindings

GSL Ww3 ROOT Eigen

Python bindings


User C++ program



libFit minimizers

BornAgain

Python bindings

GSL Ww3 ROOT

External graphics

Eigen

Working with BornAgain. o Running user C++ program

Python bindings


User Python script



libFit minimizers

BornAgain

Python bindings

GSL Ww3 (Py)ROOT

Matplotlib

External graphics

Eigen

Working with BornAgain. o Running user Python script

Python bindings


o  C++ is a complex programming language to master. o  Several people find themselves more comfortable with higher-‐level interpreted

environments. Objec8ves o  create sample descrip8on and run the simula8on from Python o  extension of the framework func8onality with Python code new formfactors, pair-‐correla8on func8ons etc.

Motivation


C++ and Python integration

There are two basic methods for integra8ng C++ with Python. o  Extension wri8ng

Python access to C++

o  Embedding C++ access to the Python interpreter

Python

See h%p://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf for nice introduc=on to the problem

extending embedding

C++

We are primarily concerned with extension wri8ng


Extension example using Python-C-API

Python is wriMen on C Any third-‐party C/C++ code can be interfaced to Python using wrapper func8ons

int fact(int n) {

if(n<=1) return1; else return n*fact(n-1);

}

func8on.c #include <Python.h> PyObject *wrap_fact(PyObject *self, PyObject *args) {

int n, result; if (!PyArg_ParseTuple(args,”i:fact”,&n)) return NULL; result = fact(n); return Py_BuildValue(“i”,result);

} static PyMethodDef exampleMethods[] = {

{ "fact", wrap_fact, 1 }, { NULL, NULL }

}; void initexample() {

PyObject *m; m = Py_InitModule("example", exampleMethods);

}

wrapper.c

Wrapper func8on o  converts func8on arguments from

Python to C o  Returns results in Python expected

form Ini8aliza8on func8on o  Register new methods with the

Python interpreter


Extension example using Python-C-API

o  Func8on and wrappers code might be compiled into single shared library. g++ –I /usr/local/python2.7 function.c wrapper.c g++ -shared function.o wrapper.o –o mymodule.so

o  Resul8ng shared library can be imported in the python module import mymodule

Wrapping a C/C++ applica8on o  Write a Python wrapping func8on for every func8on,

variable, structure and class you want to access. o  Write ini8aliza8on func8on. o  Compile everything into the module.

Huge amount of work!


List of technologies

Python C API Strong candidate to wrap only a couple of func8ons

Boost::python Comprehensive mapping of C++ features to corresponding Python features Stl containers, excep8ons, proper8es

Swig Can create wrappers for other languages besides Python (Ruby, Java)

SIP Created for PyQt4 package

Shiboken Created for PySide package

Ctypes Provides C compa8ble data types

Cython Python like language to write C extensions for Python


How to select the one

o  Different technologies looks preMy easy while wrapping o  They are gekng preMy tricky to master while exposing classes o  And what about object ownership, shared_ptr, templates?

void hello_world() {

std::cout << “Hello World!” << std::endl; }

class IBase {

IBase(float x, float y); virtual ~IBase(); virtual init() = 0;

}

class Derived(IBase): def __init__(self): IBase.__init__(self)

def init(self): self.x = 0

It is hard to say if given exposing technology fits your C++ project before you actually try to apply it to the whole project

C++ Python

?


hMp://scipy-‐lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html

Disicussion of Python-‐C-‐API, Ctypes, SWIG, Cython from NumPy community

hMp://seal.web.cern.ch/seal/snapshot/work-‐packages/scrip8ng/evalua8on-‐report.html

Detailed evalua8on of boost-‐python vs SWIG (2003)

hMp://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf

SWIG tutorial

hMp://stackoverflow.com/ques8ons/1492755/python-‐c-‐binding-‐library-‐comparison

StackOverflow discussions

hMp://stackoverflow.com/ques8ons/135834/python-‐swig-‐vs-‐ctypes hMp://stackoverflow.com/ques8ons/456884/extending-‐python-‐to-‐swig-‐not-‐to-‐swig-‐or-‐cython




We have decided to go with boost::python

providing the higher level of integra8on between C++ and Python among all compe8tors.

o  What is the performance? o  Build system integra8on? o  Is wrapping code on Python side or on on C++ side? o  How much code should be wriMen addi8onally? o  Should I affect or duplicate exis8ng C++ code? o  How big is a community? o  Is it possible to fully automa8ze wrappers genera8on? o  Do I need bindings with another languages?


C++ template based library which enables interoperability between C++ and Python programming languages. o  References and pointers. o  Automa8c cross module type conversions. o  Efficient func8on overloading. o  C++ to Python excep8on transla8on. o  Default arguments. o  Keyword arguments. o  Expor8ng C++ iterators as Python iterators. o  Documenta8on strings. o  Manipula8ng Python objects in C++.

Boost::python


Boost::python

class TwoNumbers { public: Num(); float getA() const; void setA(float value); float getB() const; void setB(float value); private: double a; double b; };

Example I o  Exposing class and its access func8ons to Python

C++

Class to expose

boost::python::class_<TwoNumbers>(”TwoNumbers") .add_property(”A", &TwoNumbers::getA, &TwoNumbers::setA) .add_property(”B", &TwoNumbers::getB);

C++

Wrapper to create

>>> x = TwoNumbers() >>> x.A = 3.14 >>> x.A, x.B (3.14, 3.14) >>> x.B = 2.17 # error!

Use from Python

Class property “B” is read only since seMer member func8on not passed in

Python


Boost::python

class Base { virtual ~Base() {} virtual int f() { return 0; } };

Example II o  Inheritance and virtual func8ons

C++

Class to expose

struct BaseWrap : Base, wrapper<Base> { int f() { if (override f = this->get_override("f")) return f(); return Base::f(); } int default_f() { return this->Base::f(); } }; class_<BaseWrap, boost::noncopyable>("Base") .def("f", &Base::f, &BaseWrap::default_f) ;

C++

Wrapper to create

>>> base = Base() >>> class Derived(Base): ... def f(self): ... return 42 ... >>> derived = Derived() >>> base.f() 0 >>> derived.f() 42

Use from Python

Python


pyplusplus

o  Boost::python allows to expose to Python almost any C++ construc8on. o  However, this requires wri8ng of a large amount of wrapping code. o  Fortunately, this can be tackled with Pyplusplus Pyplusplus package An object oriented framework for crea8ng a code generator for boost::python library BornAgain

C++ source code

Pyplusplus codegenerator

Boost::python wrappers

compiler

o  We regenerate boost::python API only when public interface has changed

When required

When required


pyplusplus

codegenerator.py

o  Provides automa8c genera8on of boost::python code for the whole project o  Contains a number of sekngs to adjust ownership policies, list of classes to expose, etc o  About 300 lines of python code


Current status

Learning curve o  10% of 8me was devoted to Python bindings

during the first year o  <1% of 8me during the second

o  BornAgain core consist of 30k lines of code o  PythonAPI is generated automa8cally using

boost::python + pyplusplus o  Same shared library can be used from C++

and from Python

Number of lines of code in BornAgain git repository

Achieved Python/C++ interoperability o  Inheritance and overload o  Transfer of ownership o  Templates exposing o  Python lists to std::vector and back o  Export C++ data structures into NumPy arrays o  shared_ptr to Python and back


Current status

C++

Python


Summary

Advantages o  It is possible to to establish complete interoperability between C++ and python. o  Actually no limita8ons have been found in exposing C++ construc8ons into Python. o  No C++ code should be adjusted. o  Single script regenerates PythonAPI for the whole project if needed. Disadvantages o  boost::python wrappers double the amount of code. o  It is heavy templated code which double the compila8on 8me. o  pyplusplus and gccxml are not developed anymore, no C++11 support will follow. Future plans. o  Wait for ROOT version 6.0 which will use technology llvm + PyPy and have a look on it

boost::python + pyplusplus


Documents

Python’bindings’in’BornAgain’ - apps.jcns.fz ...apps.jcns.fz-juelich.de/doku/sc/_media/pospelov_pythonbindings.pdfBasic software architecture Main’features:’ o Programming’language’C++