We circumvent the expensive evaluation with a

deterministic RBF surrogate defined as:

We tackle the high-dimensionality by reducing the

probability φ of searching along dimension:

We escape local minima by computing a compound score

for each candidate point. The score is a dynamic weighted

average of a metric based on the distance from the best

found solution and of a metric based on the candidate

surrogate value.

Efficient Hyperparameter Optimization of Deep Learning

Algorithms using Deterministic RBF Surrogates

Ilija Ilievski a Taimoor Akhtar b Jiashi Feng c Christine Annette Shoemaker b

ilija.ilievski@u.nus.edu erita@nus.edu.sg elefjia@nus.edu.sg isesca@nus.edu.sg

➢In

sh

ort

➢M

ore

: bit.ly/hord-aaai Supplement and more at: ilija139.github.ioa) Graduate School for Integrative Sciences and Engineering b) Industrial and Systems Engineering c) Electrical and Computer Engineering

➢D

etai

ls

➢R

esu

lts

Optimization of 19 CNN hyperparameters Optimization of 8 CNN hyperparameters

Optimization of 6 MLP hyperparametersOptimization of 15 CNN hyperparameters

➢A

lgo

rith

m

HORD: Hyperparameter Optimization using deterministic RBF surrogate and DYCORS Input: n0 init evals, Nmaxmax evals, m #candidates

Sample n0 points Xn0≔{xi} Populate An0≔{Xn0,f(Xn0)} while n < Nmaxupdate surrogate Sn with An≔{Xn,f(Xn)}set xbest≔argmin{f(Xn)}sample m candidate points ti around xbest according to probabilities φncompute Vev, Vdm, and Wn for all tiset x*≔argmin{Wn(ti)}set An+1≔{ An ⋃ (x*,f(x*)}

end while return xbest ➢

Co

mp

aris

on

Evaluations required by HORD to match the best found error by state-of-the-art hyperparameter optimization algorithms

Deep learning algorithms are powerful but are very

sensitive to the many hyperparameters they have: number

of layers and nodes, learning rate, weights initialization...

Optimizing the validation error with respect to the

hyperparameters involves the minimization of highly

multimodal and expensive function in high dimensions.

We propose an algorithm that matches the performance of

the state-of-the-art hyperparameter optimization

algorithms while using up to 6 times fewer evaluations.

Distance metric:

Surrogate value metric:

Final candidate score: