LIGHTNING, A LIBRARY FORLARGE-SCALE MACHINELEARNING IN PYTHON

, Fabian Pedregosa (1) Mathieu Blondel (2)

(1) Chaire Havas-Dauphine / INRIA, Paris France

(2) NTT Communication Science Laboratories, Kyoto Japan

SCIKIT-LEARN: WITH GREAT CODECOMES GREAT RESPONSABILITY

# lines of code in scikit-learn

Very selective for new algorithms/models.

LIGHTNINGIncorporate recent progress in large-scale optimization.

scikit-learn compatible .scalable on large datasets.support for dense and sparse input.emphasis on structured sparsity penalties.

dependencies = Python + Cython + scikit-learn.

SCIKIT-LEARN COMPATIBLE

mix lightning with scikit-learn Pipeline, GridSearchCV,etc.

⟹

FROM LARGE DATA TO LARGEOPTIMIZATION

Big data comes in different flavors.

n{⎛

⎝

⎜⎜⎜⎜

DA

TA

⎞

⎠

⎟⎟⎟⎟

pLarge sample:

Computer vision, advertising,etc.

Large dimension:Biology, neuroscience, etc.

LEARNING FROM LARGE SAMPLESUsual methods (gradient descent, BFGS, etc.):

Pass through the data at each iteration.Prohibitive for large datasets.

Back to simple methods:

Stochastic gradient descent (Robbins and Monro, 1951).

LEARNING FROM LARGE SAMPLES

lighting example, n=100.000

In last 5 years, flurry ofnew stochastic methods:

Stochastic variance-reduced gradient(SVRG)Stochastic DualCoordinate Ascent(SDCA)Stochastic AverageGradient (SAG/SAGA)

They are all in lightning!

LEARNING FROM LARGE FEATURESIterate through the columns.Coordinate Descent-like algorithms.Very efficient for sparse models.

(Blondel et al. 2013) , multiclass classification with group-lasso penalty

STRUCTURED SPARSITYThere's so much more than the Lasso ...

Group sparse penalty.Total variation.Trace norm (low rank).

APISimilarities and differences with scikit-learn

scikit-learn: (penalty = 'l1', )LogisticRegression

loss function

solver='liblinear' algorithm

lightning: (penalty = 'l1', ) CDClassifier

algorithm

loss='log' loss function

API based on algorithms, not models.

EXTENSIBILITYTypical loss and penalties available.Possible to pass custom loss or penalty function

clf = FistaClassifier( loss=my_loss, penalty=my_penalty)

(available for Fista* and SAGA*)

FUTURE CHALLENGESParallel stochastic methods

(Leblond, Pedregosa, Lacoste-Julien 2016)

Out of core (scale beyond computer memory).

SCIKIT-LEARN-CONTRIBlightning is just the beginning.

Welcome projects that are:

Your browser does not support SVG

scikit-learn compatible.Documented.Test coverage > 80%.

THANKS FOR YOUR ATTENTIONhttp://contrib.scikit-learn.org/lightning/

(We're hiring!)