Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin

Duke University Marine Geospatial Ecology Lab4-Mar-2009

Talk outlineOverview of MGETQuick tour of the MGET tool collectionExample application: habitat modeling

What is MGET?A collection of geoprocessing tools for marine ecology

Oceanographic data management and analysisHabitat modeling, connectivity modeling, statisticsHighly modular; designed to be used in many scenariosEmphasis on batch processing and interoperability

Free, open source softwareWritten in Python, R, MATLAB, C#, and C++Minimum requirements: Win XP, Python 2.4 ArcGIS 9.1 or later currently needed for many toolsArcGIS and Windows are only non-free requirements

Also useful for terrestrial problems!

Set goals & priorities

Make & implement EBM decisions

Monitor &

assess

Develop conceptual

models

Analyze data & develop models and scenarios

Collect physical, biological, and

socioeconomic data

Visualize scenarios

Set goals & priorities

Make & implement EBM decisions

Monitor &

assess

Develop conceptual

models

Analyze data & develop models and scenarios

Collect physical, biological, and

socioeconomic data

Visualize scenarios

Monitoring & Monitoring & assessment toolsassessment tools

Project Project management toolsmanagement tools

Data collection Data collection and management and management

toolstools

Conceptual Conceptual modeling modeling

toolstools

Data processing Data processing toolstools

Modeling toolsModeling tools-Model development tools

-Watershed models-Dispersal and habitat models

-Marine ecosystem models-Social science models

Sector-specific decision support toolsSector-specific decision support tools-Conservation and restoration site selection

-Coastal zone management tools-Fisheries management tools

-Hazard assessment and resiliency planning tools-Land use planning tools

Stakeholder Stakeholder communication communication & engagement & engagement

toolstoolsScenario Scenario

visualization visualization toolstools

Set goals & priorities

Make & implement EBM decisions

Monitor &

assess

Develop conceptual

models

Analyze data & develop models and scenarios

Collect physical, biological, and

socioeconomic data

Visualize scenarios

Monitoring & Monitoring & assessment toolsassessment tools

Project Project management toolsmanagement tools

Conceptual Conceptual modeling modeling

toolstools

Sector-specific decision support toolsSector-specific decision support tools-Conservation and restoration site selection

-Coastal zone management tools-Fisheries management tools

-Hazard assessment and resiliency planning tools-Land use planning tools

Stakeholder Stakeholder communication communication & engagement & engagement

toolstoolsScenario Scenario

visualization visualization toolstools

Data collection Data collection and management and management

toolstools

Data processing Data processing toolstools

Modeling toolsModeling tools-Model development tools

-Watershed models-Dispersal and habitat models

-Marine ecosystem models-Social science models

Data collection Data collection and management and management

toolstools

Data processing Data processing toolstools

Modeling toolsModeling tools-Model development tools

-Watershed models-Dispersal and habitat models

-Marine ecosystem models-Social science models

MGET

MGET’s software architectureMGET “tools” are really just Python functions, e.g.:

pythoncom2x.dll

IDispatchIMyTool

MyToolCOM class

MyTool.py

MGET ArcGIS Toolbox

Python programs

ArcGIS 9.xEarly-bound COM clients (e.g. C++)

Late-bound COM clients

(e.g. VBScript)

MGET

External callers

win32com module

MGET COM module

MGET exposes them to several types of external callers:

def MyTool(input1, input2, input3, output1)

MGET interface in ArcGISThe MGET toolbox appears in the ArcToolbox window

MGET interface in ArcGISDrill into the toolbox to find the toolsDouble-click tools to execute directly, or drag

to geoprocessing models to create a workflow

Integration The Python functions can invoke C++, MATLAB, R, ArcGIS, and COM classes.

R interpreter

MyTool.m MyTool.r

MyTool.py

Python extension DLL

MyTool.cpp

C++

MyTool.pyd

Python extension DLL

MyToolMatlab.pyd

MATLAB Component Runtime (MCR)

rpy module

MGET COM module

win32com module

R packagesMATLAB toolboxes

IDispatch

COM Automation

classes

MGET ArcGIS module

arcgisscripting or win32com

module

ArcGIS geoprocessor

C libraries

ArcGIS toolboxes

Python packages

MGET R module

MGET utilizes a lot of other softwareInterpreters / Runtimes

PythonMATLAB Component RuntimeR

R Packages

gamMASSmgcvrgdalROCR

Applications

ArcGISNOAA CoastWatch Utilities

Python Packages

docutilshttplib2lxmlnetcdf4numpyosgeopydappyparsingpyprojpywin32rpysetuptools

C Libraries

GDAL/OGRgziphdflibxmllibxsltnetcdfproj4zlib

All but one of these (pywin32) are installed automatically

Quick tour of the tools

Analyzing larval connectivityCoral reef ID and % cover maps Ocean currents data

Tool downloads data for the region and dates you specify

Larval density time series rasters

Edge list feature class representing dispersal network

Original research by Eric A. Treml

Converting data

Batch processingCopy one raster at a time

Batch processingCopy rasters that you list in a table

Batch processingCopy rasters from a directory tree

Tools for specific products

Downloads sea surface height data from http://opendap.aviso.oceanobs.com/thredds

Identifying SST fronts

~120 km

AVHRR Daytime SST 03-Jan-2005

28.0 °C

25.8 °C

Mexico

Front

Cayula and Cornillion (1992) edge detection algorithm

Freq

uenc

y

Temperature

Optimal break 27.0 °C

Strong cohesion front present

Step 1: Histogram analysis

Step 2: Spatial cohesion test

Weak cohesion no front

Bimodal

Example output

Mexico

ArcGIS model

Identifying geostrophic eddies

Aviso DT-MSLA 27-Jan-1993 Red: Anticyclonic Blue: Cyclonic

Negative W at eddy core

SS

H a

nom

aly

Available in MGET 0.8

Example output

Mapping species biodiversity

Invoking R from ArcGIS

Example application: habitat modeling

ChlorophyllSST

Bathymetry

Presence/absence observations

Sampled environmental data

Multivariate statistical model

Probability of occurrence predicted from environmental covariates

Binary classification

Warning: Habitat modeling is complicated! This simplified example is meant to briefly illustrate tools. Consult the literature for best practices!

Focal species: Stenella frontalis

Photo: Garth Mix

Map: OBIS-SEAMAP

Common name:Atlantic Spotted Dolphin

Distribution:Tropical and warm temperate Atlantic

Study area: Eastern U.S.

Species observation data

The Ocean Biogeographic Information System (OBIS) is a global database of marine species observations.The OBIS-SEAMAP system at Duke University holds the records for seabirds, marine mammals, and sea turtles, including records gathered during NOAA cruises.

Environmental predictor variables

Bathymetry:ETOPO2V2 from NOAA NGDC

SST:Monthly climatological 4km AVHRR Pathfinder from NOAA NODC

Chlorophyll:Monthly climatological SeaWiFS chlorophyll-a from NASA GSFC

Images shown above are for month of March

Step 1: Download species pointsDownload points using MGET tool:Presence: Records of Stenella frontalisAbsence: Records of other cetaceans

The tool uses the DiGIR protocol to retrieve data from OBIS servers

Red: PresenceGreen: Absence

Step 2: Convert oceanography to Arc rasters1.Download with FTP from NOAA and NASA:

ETOPO2 bathymetry – 1 binary fileAVHRR Pathfinder monthly climatological SST – 12 HDF filesSeaWiFS monthly climatological chlorophyll – 12 HDF files

2.Convert to ArcGIS rasters using MGET tools:

Step 3: Sample oceanography at points

• Need to sample rasters and populate fields

• Must sample SST and chlorophyll by date

Step 3: Sample oceanography at points

Sampling bathymetry is easy because it is static

To sample dynamic data such as SST and chlorophyll, you must first calculate the paths to rasters to sample from the points’ dates

Then use an MGET batch sampling tool

Step 4: Create exploratory plots

Best predictors: SST and Chl

Step 5: Fit, evaluate, and predict model

Presence ~ s(SST) + s(log10(Chlorophyll))Presence ~ s(SST) + s(log10(Chlorophyll))

Partial plots produced by the Fit GAM tool

SST SSTlog10(Chlorophyll)

s(S

ST

,8.9

7)

s(lo

g10

(Ch

loro

ph

yll)

,5.6

)

Presence more likely at higher SST Presence more likely at lower Chl

Plotting a receiver operating characteristic curve

The ROC plot

False positive rate

Tru

e p

ositi

ve r

ate Cutoff = 0.020

By default, tool selects the cutoff closest to the point of perfect classification (0, 1)

Model summary statistics:

Area under the ROC curve (auc) = 0.960779Mean cross-entropy (mxe) = 0.030566Precision-recall break-even point (prbe) = 0.001866Root-mean square error (rmse) = 0.087781

Contingency table for cutoff = 0.019638:

Actual P Actual N TotalPredicted P 287 3541 3828Predicted N 26 32408 32434 Total 313 35949 36262

Accuracy (acc) = 0.901633Error rate (err) = 0.098367Rate of positive predictions (rpp) = 0.105565Rate of negative predictions (rnp) = 0.894435

True positive rate (tpr, or sensitivity) = 0.916933False positive rate (fpr, or fallout) = 0.098501True negative rate (tnr, or specificity) = 0.901499False negative rate (fnr, or miss) = 0.083067

Positive prediction value (ppv, or precision) = 0.074974Negative prediction value (npv) = 0.999198Prediction-conditioned fallout (pcfall) = 0.925026Prediction-conditioned miss (pcmiss) = 0.000802

Matthews correlation coefficient (mcc) = 0.246384Odds ratio (odds) = 101.026394SAR = 0.650065

ROC summary stats for cutoff:

Predicting presence for oceanographic rasters

Rasters output by the Predict GAM tool

Predicted presence:Range: 0 - 0.25

Standard errors:Range: 0 - 0.11

Binary classification:Species range map produced by classifying presence into 0 or 1 according to ROC cutoff

Similar to OBIS-SEAMAP range map?

Predictions for October

AcknowledgementsA special thanks to the many developers of the open source software that MGET is built upon, including:

Guido van Rossum and his many collaborators; Mark Hammond; Travis Oliphant and his collaborators; Walter Moreira and Gregory Warnes; Peter Hollemans; David Ullman, Jean-Francois Cayula, and Peter Cornillon; Stephanie Henson; Tobias Sing, Oliver Sander, Niko Beerenwinkel, and Thomas Lengauer; Frank Warmerdam and his collaborators, Howard Butler; Timothy H. Keitt, Roger Bivand, Edzer Pebesma, and Barry Rowlingson; Gerald Evenden; Jeff Whitaker; Roberto De Almeida and his collaborators; Joe Gregorio; David Goodger and his collaborators; Daniel Veillard and his collaborators; Stefan Behnel, Martijn Faassen, and their collaborators; Paul McGuire and his collaborators; Phillip Eby, Bob Ippolito, and their collaborators; Jean-loup Gailly and Mark Adler; the developers of netCDF; the developers of HDFThanks to our funders:

For more informationDownload MGET:

http://code.env.duke.edu/projects/mget

Email us:

jason.roberts@duke.edu, bbest@duke.edu

Learn more about habitat modeling:Guisan, A., Zimmermann, N.E. (2000) Predictive habitat distribution models in ecology. Ecological Modelling 135, 147–186.

Thanks for attending!