Automatic Differentiation in

Finlay Scott & Iago Mosqueira

Structure

• What is R? – history, strengths, limitations• Current differentiation options in R• How we have used AD with R• Next steps – AD in R

What is R?

• R website - http://www.r-project.org/• “R is a free software environment for statistical

computing and graphics.”• Similar to S (Bell Laboratories)• Development started in 1993• Multi-platform (*nix, Windows, MacOS)• Scripting language• Object oriented and functional programming• Vectorised• Dynamically link to C / C++ / Fortran code

R - what is it good for?• Wide variety of statistical and graphical

techniques:– linear and nonlinear modelling– classical statistical tests– time-series analysis– classification– clustering– ...

• Highly extensible• Excellent graphical capabilities

– Presentation– Data exploration

Publication quality plots

EconomicsBiology

Epidemiology

Data mining

High performance computing Optimisation

Genomics

Pharmacokinetics

Business analysisEnvironmental modelling

Biostatistics

Forecasting

Bayesian networksSpatial modelling

At recent useR conference

NY Times article

Why the interest?• Open Source and therefore ‘Free’

– Not just as in ‘Free beer’ but as in ‘Free software’

• Released under GNU General Public License• This means:

– Transparent (source code available)– Distributed peer review– ‘given enough eyeballs, all bugs are shallow’ – Eric

Raymond, The Cathedral & The Bazaar– Blurs boundary between developer and user

• Director of technology product marketing at SAS: “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”

Why the interest (2)

• Highly extensible – ‘packages’• 1875 packages officially hosted on CRAN• Can ‘simply’ add particular functions or can

make more significant changes e.g. parrallelisation

• Link to C / C++ / Fortran• Link to databases (e.g. MySQL, SQLite)• People have added editors, GUIs, Sweave• Huge community• Costs nothing!

Source: Hornik, 2008

Differentiation and R• Default optimisation methods (including quasi-

Newton methods e.g. BFGS)• 20+ packages devoted to differentiation• Whenever derivatives are required, can supply

gradient function.• Otherwise, generally, numerical approximations

using finite differences are used.• All the associated problems – speed and

accuracy.• Some symbolic differentiation of expressions

(deriv) but simple (not Maple or Mathematica)• R would greatly benefit from generic AD

interface

Demo of Hessian calculation in R

• Speelpenning’s example from ADOL-C

∏−

=

==1

0

)(n

iixxfy

• Can we calculate the Hessian in R?

• Use the numDeriv package

How we have used AD and R

• Fisheries modelling requires projection under various constraints (e.g. catch must not exceed quota)

• Fishing Effort as independent variable• Used ADOL-C libraries (Walther, Kowarz & Griewank)• Implemented simple solvers (Newton-Raphson method)• R dynamically links to the C++ code• But… not generic. It’s application specific• Can pass parameter values from R to C• But not functions• ‘Hard wired and hard work’ - for ‘average’ R user, writing

C / C++ is hard

Example of using ADOL-C with R

User interface

C++

Function

ADOL-C Hessian calculation

Parameter values

Hessian

What we would like…

• Generic interface to AD methods, in R• E.g. write a function in R, use AD to get

first and higher derivatives of that function.• Need is certainly there – useR 2009• ‘Finding a solution was fast, working out

that we had found a solution was the problem’

What next?

• Work is very long term• Many skilled people in R community who want to

help drive this forward• But assistance from the AD community would be

very helpful• If you are interested:

– John Nash: nashjc@uottawa.ca– John Nash’s wiki:

http://macnash.telfer.uottawa.ca/optimx/doku.php– Martin Maechler: maechler@stat.math.ethz.ch– Developers mailing list: http://www.r-project.org/– finlay.scott@cefas.co.uk