Agile Computer Controlof a Complex Experiment

by

Gaël Varoquaux

presented by Jonathan Blakes

on18th September 2009

forInter-disciplinary Optimisation Laboratory

Journal Club

2

Outline of this talk

1. Why I chose this paper2. Outline of the paper3. Section by section discussion of

underlying technologies and practical implementation

4. Some useful references

3

Why this paper?Highly practicalMirrors common experimental

workflowNice example of an appropriate

degree of programming language evangelism

Introduces TraitsUI for building GUIs◦I use Mayavi2’s mlab module for 3D

plotting in our application, maintained by Gael

4

Shameless plug for the Infobiotics Workbench

5

Outline of the paperIntroductionInterfacing with specialist

hardwareUnit testing the experimentPerforming operations in parallelEvent-driven programmingBuilding GUIsData-driven programmingPutting it all together

6

IntroductionExperimentalists use (and build) specialist

equipmentThey are are computer literate and

mathematically competent but not computer scientists

Bad software design choices can hold up research, even though it is not the 'hard' part

Need software that evolves with the physical experiment

A 'flexible and reliable code base' can help the lab reorientate with goal and resource changes

7

Interfacing with hardwareCritical code often written in C (controller API /

performance)◦ Difficult for new people to contribute with steep

learning curve◦ New code can introduce memory-management

problemsAdvocates Python as high-level language for

connecting to devices◦ Python has modules for controlling VISA

instruments◦ Wrapping API calls (using 'ctypes' module) to a

generic interface enables new devices to be incorporated without changing data-processing/visualisation routines

◦ 'Pushes' memory-management problems out of C.

8

Unit testing the experiment

As complexity of task grows the elementary operations on which relies must be robust

Automated test harness ensures bugs are picked up early

Can replace temperamental hardware with 'mock objects'◦test software without them◦forces you to understand instruments

more fully

9

Parallel operationOften have to wait for devices or

data-processing – but how long?Should run aspects concurrently

using threads:◦One thread for user interface◦another for hard/long computations◦and one for each device

Rule of thumb: no two threads to modify the same object

10

ThreadingActually very easy in Python:

Prevents blocking UI

11

Event-driven programsProblem: Events occuring out of

sequence can bring down experiment

Solution: respond to events in the order they happen rather than in the order we program them to

How?◦Listen for events (with listener

thread), accumulate callbacks in queue, execute in order.

12

Super-simple event queueInherits deque, a fast

stack/queue: O(1) pop & insert(0, x)

dispatch time-consuming function calls to queue

queue executes them in order on a separate thread

13

Building GUIsWant to setup and run experiments

interactively◦ display and (sensible) editing of parameters

But designing and building GUIs is time-consuming and difficult

Code becomes messy as model, view and controller get merged

Python library TraitsUI can help by automatically generating GUIs based on underlying datamodel (Traits)◦ Removes GUI implementation from code

14

What are Traits?Static type definitions for Python variables

(otherwise dynamically typed)Example traits:

◦ aperture = Range(0,10,1)◦ plot = Instance(MLabSceneModel)

Traits facilitate: Variable initialization – simple, complex or deferred Validation – type checking and value checking Notification – trigger events when values change

(built-in, transparent Observer pattern) Visualisation – knowing types enables appropriate

widget choiceClasses inherit from HasTraits

15

TraitsUI example

16

Other benefits of TraitsUIMultiple views per object

◦Reuse objects in different contextsCompound views

◦Reuse views in larger applicationsDon't have to embed in GUI:

◦ configure_traits(view=object.view)

runs in its own event loopSerialize state easily:

◦ configure_traits(filename=~/app/ini)

17

Data-driven programsResponding to data changes from

◦ Devices◦ User input

Traits fire events when their values change

Capture events with methods:@aperture_changed@focus_changeddef update_display(self): self.camera.take_photo()

18

Putting it all together

19

Acknowledgements (plug)Traits are an innovation of Enthought Inc.

◦ sponsor SciPy (includes NumPy)◦ run free training webinars (US time)

Part of Enthought Tool Suite (ETS), which also includes:◦ Mayavi2 (TVTK) – 3D data visualisation◦ Chaco – 2D plotting, complements Matplotlib◦ Envisage – Eclipse-like application framework

All open source, BSD License (very liberal

20

ReferencesVaroquaux G. Agile Computer Control of a

Complex Experiment. Computing in Science & Engineering (2008) 20-25

Ramachandran P. Mayavi: Making 3D Data Visualization Reusable. Proc. SciPy 2008, 51-57

Varoquax G. Writing a graphical application for scientific programming using TraitsUI. http://gael-varoquaux.info/computers/

traits_tutorial/traits_tutorial.pdfEnthought Inc.

http://code.enthought.com/

21

Other resourcesSoftware Carpentry course

◦http://software-carpentry.org/◦For non computer scientists◦Uses Python◦Explains unit-testing, version control◦Mentioned in Nature:

Wilson G. Open-source offers solutions for science software education. Nature 436: 600 (27 July 2005) doi:10.1038/nj7050-600b

22

finally:'Passing that knowledge along'Check out .coma free programming Q&A wiki