Scaling up Generic Optimization usingautoBLAS

Joachim Giesen Lars Kühne Sören Laue Jens Müller

Friedrich-Schiller-University Jena, Germany

Big Data Analytics

About 600-700 papers published at NIPS and ICML per year.

~30% design / implement a new optimization algorithm

Big Data Analytics

About 600-700 papers published at NIPS and ICML per year.

~30% design / implement a new optimization algorithm

Motivation

Look at big data analytics through the lens of optimization.

I Least squares regression

I SVMs

I Kernel learning

I k-means

I . . .

Motivation

Look at big data analytics through the lens of optimization.

I Least squares regression

I SVMs

I Kernel learning

I k-means

I . . .

Generic Optimization (GENO)

matrix A

vector b

minimize norm(A*x - b, 2)

subject to

x >= 0

GENO

non-negativeleast squares

solver

matrix X sparse

vector y

scalar c

minimize

1/2 * w’*w + c * sum(xi)

subject to

y.*(X*w+vector(b)) >= 1-xi

xi >= 0

SVM solver

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

Matrix Calculus

gradient computation

example: f (x) = xT Ax

gradient: ∇f (x) = (AT + A)x

example: logistic regressionf (w) = 1

2wT w + C · sum(log(1 + exp(−y . ∗ (Xw + b))))gradient: ∇f (w) = ?

I Maple, Mathematica, Sage cannot do it

Matrix Calculus

gradient computation

example: f (x) = xT Axgradient: ∇f (x) = (AT + A)x

example: logistic regressionf (w) = 1

2wT w + C · sum(log(1 + exp(−y . ∗ (Xw + b))))gradient: ∇f (w) = ?

I Maple, Mathematica, Sage cannot do it

Matrix Calculus

gradient computation

example: f (x) = xT Axgradient: ∇f (x) = (AT + A)x

example: logistic regressionf (w) = 1

2wT w + C · sum(log(1 + exp(−y . ∗ (Xw + b))))

gradient: ∇f (w) = ?

I Maple, Mathematica, Sage cannot do it

Matrix Calculus

gradient computation

example: f (x) = xT Axgradient: ∇f (x) = (AT + A)x

example: logistic regressionf (w) = 1

2wT w + C · sum(log(1 + exp(−y . ∗ (Xw + b))))gradient: ∇f (w) = ?

I Maple, Mathematica, Sage cannot do it

Matrix Calculus

gradient computation

example: f (x) = xT Axgradient: ∇f (x) = (AT + A)x

example: logistic regressionf (w) = 1

2wT w + C · sum(log(1 + exp(−y . ∗ (Xw + b))))gradient: ∇f (w) = ?

I Maple, Mathematica, Sage cannot do it

Matrix Calculus – Demo

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

Optimization Problem Solver

I tested on many data analytics problems

I tested on CUTEst benchmark (1150 optimizationproblems)

I competitive on large problems

I room for improvement on small problems

Optimization Problem Solver

I tested on many data analytics problems

I tested on CUTEst benchmark (1150 optimizationproblems)

I competitive on large problems

I room for improvement on small problems

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsEigen

GENO stack

Matrix CalculusSymbolic Computation

Optimization ProblemSolver

Linear Algebra ExpressionsautoBLAS

Motivation autoBLAS

GENO compute time

solver timefunction value

&gradient

What takes most of the time?

EN KDSVM LR LSQ NNLSQ SPCA0

10

20

30

40

50

60

70

80

90

100 98

9296 95 95

100

Time spent on function value & gradient

Computing gradient and function value

Example: logistic regression

t1 = ones(rows(X))t2 = exp(-y .* (X * w))t3 = t1 + t2g = w + c*X’ * ((t1 ./ t3) .* t2 .* -y)f = 0.5 * w’ * w + c * t1’ * log(t3)

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

autoBLAS

I linear algebra expression pre-compiler

I host language independent

I decouples expression generation from evaluation

I performs symbolic expression optimization

I efficient temporary allocation and reuse

I expressive error-messages

I mapping to BLASI standard interface: portable

I efficient implementations available for all major hardwareplatforms and frameworks

Declarative Language

Example: logistic regression

t1 = Vector(1, (rows(X)));t2 = exp(-y .* (X * w));t3 = t1 + t2;g = w + c*X’ * ((t1 ./ t3) .* t2 .* -y);f = 0.5 * w’ * w + c * t1’ * log(t3);

autoBLAS

• C• Eigen-Classes• SQL-Tables• Python array

• expression to statementmapping

• expression optimzation• temporary allocation

• BLAS• Eigen3• MKL• NumPy• sqlBLAS

Frontend Core Backend

Frontend

#autoblas {Vector y, w;Vector g f=c data=gPtr size=N;Matrix X sparse=csc;...

}

autoblas -f eigen -b mkl < file.cpp.in >file.cpp

Frontend

#autoblas {Vector y, w;Vector g f=c data=gPtr size=N;Matrix X sparse=csc;...

}

autoblas -f eigen -b mkl < file.cpp.in >file.cpp

Experiments

KDSVM LR0

2

4

6

8

10

12

Eigen with multicore MKL Eigen with autoBLAS TemporariesautoBLAS with multicore MKL

Sp

ee

du

p

Experiments

1 2 3 4 5 6 7 8 Xeon Phi0

1

2

3

4

5

6

LSQ - 4GB dataset

# of cores

spe

ed

up

Conclusion

I with autoBLAS, GENO scales to medium-sized datasets(a few dozen GB) within a single compute node

I exploits all available compute resources within a node(CPU, Xeon Phi, GPU)

I Outlook: go open source

Workshop (Sparse) Linear Algebra for Graphs andOptimization

Organizers: Joachim Giesen, Sören Laue,Henning Meyerhenke

Venue: Institut für Informatik,Friedrich-Schiller-Universität Jena

Date: Spring 2016 (TBD)

If interested please send an email tojoachim.giesen@uni-jena.de.

Workshop (Sparse) Linear Algebra for Graphs andOptimization

Schedule: Friday14:15 - 15:45: Linear Algebra, Graph Algorithms, and theGraphBLAS (Henning Meyerhenke)14:45 - 16:15: Coffee break16:15 - 17:45: Non-linear Optimization (Sören Laue/JoachimGiesen)19:00 Joint Dinner

Saturday9:15 - 10:45: A Practical Introduction to the Intel Xeon PhiArchitecture (Lars Kühne)10:45 - 11:15: Coffee break11:15 - 12:45: Linear Algebra on the Intel Xeon Phi (LarsKühne)

If interested please send an email tojoachim.giesen@uni-jena.de.