SciPy IPython

IPython: modern tools for interactive & web-enabledscientific computing

Fernando Pérezhttp://fperez.org, @fperez_org

Fernando.Perez@berkeley.edu

Helen Wills Neuroscience Institute, UC Berkeley

NERSC User Day, LBL, Berkeley,Feb 13, 2013

SciPy IPython

Outline

1 Scientific Python

2 IPython: Interactive Python

FP (UC Berkeley) IPython 2/13/13 2 / 21

Beyond (Floating Point) Number Crunching

Hardwarefloating point

Arbitrary precisionintegers

Rationals

Interval arithmetic

Symbolic manipulation

FORTRAN

Extended precisionfloating point

Text processing

Databases

Graphical userinterfaces

Web interfacesHardwarecontrol

Multi-languageintegration

Data formats: HDF5, XML, ...

Scientific Python: a Rich Ecosystem

IPython

NetworkX

SciPy IPython

Outline

1 Scientific Python

2 IPython: Interactive Python

FP (UC Berkeley) IPython 2/13/13 5 / 21

SciPy IPython

The Lifecycle of a Scientific Idea (schematically)

1 Individual exploratory work2 Collaborative development3 Production work (HPC, cloud, parallel)4 Publication (with reproducible results!)5 Education6 Goto 1.

The Problem with most toolsBarriers and discontinuities in workflow in between all the steps

FP (UC Berkeley) IPython 2/13/13 6 / 21

SciPy IPython

The Lifecycle of a Scientific Idea (schematically)

1 Individual exploratory work2 Collaborative development3 Production work (HPC, cloud, parallel)4 Publication (with reproducible results!)5 Education6 Goto 1.

The Problem with most toolsBarriers and discontinuities in workflow in between all the steps

FP (UC Berkeley) IPython 2/13/13 6 / 21

IPython’s goal:Fluid transitions in all these steps

Demo

SciPy IPython

Pillar #1: An architecture for interactive computing

FP (UC Berkeley) IPython 2/13/13 9 / 21

SciPy IPython

Pillar #2: the Notebook Format

JSON but version control-friendlyEasy for machine processing, fixable by hand if need be.Lots of hooks for metadataNot Python-specific (R and Ruby notebooks exist, Julia planned)Produce Markdown, reST, LATEX, HTML, etc...

An open format for sharing, publishing andarchiving executable computational work

FP (UC Berkeley) IPython 2/13/13 10 / 21

Documented protocols and formats:a growing ecosystem around IPython

Microsoft Visual Studio 2010 integrated consoleDino Viehland and Shahrokh Mortazavi (Microsoft); http://pytools.codeplex.com

A vim client to control an IPython kernel/consolePaul Ivanov (Berkeley), https://github.com/ivanov/vim-ipython

Notebooks on Windows Azure CloudShahrokh Mortazavi (Microsoft), B.G., F.P.: http://bit.ly/JQeojD.

Star Cluster: IPython parallel+Notebook on Amazon EC2Justin Riley (MIT): http://web.mit.edu/star/cluster

One-click single notebook on Amazon EC2Carl Smith (UK): https://notebookcloud.appspot.com.

Other projects using IPython

ScientificSoftware CarpentryEPD: Enthought Python Distribution.Continuum: Anaconda, Wakari.Sage: open source mathematics.PyRAF: Space Telescope Science InstituteCASA: Nat. Radio Astronomy ObservatoryGanga: CERNPyMAD: neutron spectrom., Laue LangevinSardana: European Synchrotron RadiationASCEND: eng. modeling (Carnegie Mellon).JModelica: dynamical systems.DASH: Denver Aerosol Sources and Health.Trilinos: Sandia National Lab.DoD: baseline configuration.Mayavi: 3d visualization, Enthought.NiPype: computational pipelines, MIT.

...

Web/OtherVisual Studio 2010: MS.Django.Turbo Gears.Pylons web frameworkZope and Plone CMS.Axon Shell, BBCKamaelia.Schevo database.Pitz: distributedtask/bug tracking.iVR (interactive VirtualReality).Movable Python(portable Pythonenvironment)....

(Incomplete) Cast of CharactersBrian Granger - Physics, Cal State San Luis ObispoMin Ragan-Kelley - Nuclear Engineering, UC BerkeleyMatthias Bussonnier - Physics, Institut Curie, ParisBrad Froehle - Mathematics, UC BerkeleyPaul Ivanov - Neuroscience, UC Berkeley.Robert Kern - EnthoughtThomas Kluyver - Biology, U. SheffieldJonathan March- EnthoughtEvan Patterson - Physics, Caltech/EnthoughtJörgen Stenarson - Elect. Engineering, Sweden.John Hunter - TradeLink Securities, Chicago.Stefan van der Walt - Applied Math, U. StellenboschPrabhu Ramachandran - Aerospace Engineering, IIT Bombay.Satra Ghosh- MIT NeuroscienceGaël Varoquaux - Neurospin (Orsay, France)Ville Vainio - CS, Tampere University of Technology, FinlandBarry Wark - Neuroscience, U. Washington.Ondrej Certik - Physics, U Nevada RenoDarren Dale - CornellJustin Riley - MITMark Voorhies - UC San FranciscoNicholas Rougier - INRIA Nancy Grand EstThomas Spura - Fedora projectMany more! (~150 commit authors)

SupportThank you!

Enthought, Austin, TX: Lots!Microsoft: WinHPC support, Visual Studio integration, Azure(thanks to Shahrokh Mortazavi).DoD/DRC Inc: funding through Sept. 2012 (thanks to JoseUnpingco and Chris Keees).NIH: via NiPy grantNSF: via Sage compmath grantGoogle: summer of code 2005, 2010.Tech-X Corp., Boulder, CO: Parallel/notebook (previous versions)

Recent Stable Funding

2-year funding, core team (7 people)

Stanford: reproducibility & data sharing(neuroimaging, other data-intensive cases)

NumFOCUS: http://numfocus.orgFoundation to promote Open Source Scientific Computing

Support the development of multiple projects.Community-created and driven.A neutral ground for industry, academia and government to supportscientific open source.501(c)3 - donations are tax-exempt in the USA