Segmentation TrainerA Robust and User-friendly Machine Learning Image Segmentation Solution

Presented by Mike Marsh, Ph.D.Dragonfly Product Manager

Thursday, March 2, 201710th FIB-SEM Users’ Group MeetingGaithersburg, MD

About ORS▸ Headquartered in Montreal, Canada.▸ Founded in 2004.▸ Registered users in 80 countries.▸ Practicing ISO and IEC standards compliant processes

Visual SI ORS Visual

About ORS▸ Headquartered in Montreal, Canada.▸ Founded in 2004.▸ Registered users in 80 countries.▸ Practicing ISO and IEC standards compliant processes

Dragonfly ORS Visual

Rapid InnovationV2.0 launched September 2016V2.1 coming in April 2017▸ macro engine▸ superpixel segmentation▸ machine learning

segmentation engine▸ In-application store

V2.2 coming in fall 2017(coming to Linux)

TechnologyAnaconda Python 3.5 for scientific computingState-of-the-art image segmentationHigh-impact rendering engine

Extensibility and CommunitySockets for extensions:▸ Embedded online console▸ Object analysis

measurements▸ Image filters▸ Menu-actions▸ Macros▸ Machine Learning classifiers▸ (and more)

App store for sharing and versioning

LicensingFlexible licensing options for various institutional needsFree licensing for non-commercial use in most countries

About Dragonfly

Image SegmentationThe hard way and the easy wayPainstaking:▸ Painting▸ Constrained Painting

▹ Threshold-gated painting▹ Superpixel-bloc painting

Easy, but never good enough:▸ Point-and-click▸ Thresholding (interactive)▸ Thresholding (algorithmically, eg. Otsu’s method)▸ Other tools▸ Automated?

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

Input image

Class 1Class 2

Segmentation

Classifier

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

FIB-SEM of fuel cell Segmentation

ElectrolyteElectrode

Pore spaceClassifier

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

FIB-SEM of fuel cell Segmentation

ElectrolyteElectrode

Pore space

Filter Banks

Use any of the filters in the Image Processing toolbox▸ Smoothing▸ Edge Enhancement▸ Texture

▹ Gabor▹ HoG▹ DoG▹ Standard deviation

Aggregate into filter banks

Spatial Discretization

Spatial Discretization

Spatial Discretization

Pixel classificationSmartGrid cell classification:▸ Superpixel▸ Watershed on Grid▸ Superixel (Scikit-learn)▸ Watershed on Grid (Scikit-learn)

Spatial Discretization

Machine Learning Core

▸ Random Forest▸ Extra-Trees▸ Adaboost▸ Gradient Boosting▸ Bagging▸ K-Nearest Neighbors

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

FIB-SEM of fuel cell Segmentation

ElectrolyteElectrode

Pore space

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

ElectrolyteElectrode

Segmentation

Pore space

SE

BSE

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

Black box classifier segmentation

ElectrolyteElectrode

Segmentation

Pore space

SE

BSE

Machine learning coreEngineParameters

Spatial discretization settingsSmart Grid (Region) engineParameters

Filter bank (Feature Presets)

Train it

Classifier

ElectrolyteElectrode

Pore space

ElectrolyteElectrode

Segmentation

Pore space

SEimage

BSEimage

Machine learning coreEngineParameters

Spatial discretization settingsSmart Grid (Region) engineParameters

Filter bank (Feature Presets)

Apply it

Classifier ElectrolyteElectrode

Segmentation

Pore space

SEimage

BSEimage

Machine learning coreEngineParameters

Spatial discretization settingsSmart Grid (Region) engineParameters

Filter bank (Feature Presets)

Apply it

Classifier

Mask

ElectrolyteElectrode

Segmentation

Pore space

SEimage

BSEimage

ClassifierMachine learning core

● Engine● Parameters

Spatial DiscretizationSmart Grid (Region)

● Engine● Parameters

Signal TexturesFilter bank (Feature Presets)

It’s modular

ElectrolyteElectrode

Segmentation

Pore space

SEimage

BSEimage

ClassifierDeep Learning core

● Engine● Parameters

Spatial DiscretizationNot necessary

Signal TexturesNot necessary

It’s modular (Deep Learning CNN) Late 2017

ElectrolyteElectrode

Segmentation

Pore space

SEimage

BSEimage

Encourage re-use of Classifiers

▸ Share classifiers with the community in the App Store (Infinite Toolbox) April 2017

▸ Preview classifiers onlineLate 2017

Acknowledgments

▸ Isabelle Bouchard▸ Nicolas Piche

▸ scikit-learn.org (Machine Learning in Python)

Workflow for Using Classifiers

Build the classifierTrain itTune itRe-use it

Workflow for Using Classifiers

Build the classifierTrain itTune it▸ Iterate:

▹ Update training classes▹ Tweak engine parameters▹ Add / remove filter banks

review coefficients▹ Retrain

▸ PreviewRe-use it

Segmenting Systematically(and with multiple signals)

1D thresholding: Use range2D thresholding: Histographic segmentation3D, 4D, ... : ???▸ BSE, ESB▸ Elemental maps: Cu, Mb, Sn, Ni, ▸ More common than that: beyond simple signal intensity,

you may have spatially correlated signal (e.g. texture)