Backup: Machine Learning - UniTrento Enver Sangineto... · 2020. 6. 9. · Backup: Machine Learning ... –Weakly-supervised Object Detection –Anomaly Detection –Few-Shot Learning

Backup: Machine Learning

Enver Sangineto DISI

University of Trento, Italy

• My (general) research interests

• Backup: Hybrid biological-artificial networks

• Backup: Neuromorphic computing

Overview




Overview

My Research Area

• Deep Learning:

– Discriminative methods

– Generative methods

Current (non-Backup) Research Interests

• Discriminative training with minimal human supervision – Weakly-supervised Object Detection

– Anomaly Detection

– Few-Shot Learning

– Domain Adaptation

• GAN-based Image Generation – GANs conditioned on structured input

– Improving GAN stability




Overview

Goals: 1. Use an ANN to predict what a biological net "thinks"

2. Perform hybrid, joint artificial-biological computations

Hybrid biological-artificial nets




Predicting the behaviour of a biological network

• Can we read and predict what a brain thinks?

• There are some in-vivo experiments using human beings

9

"Mind reading"

• Reconstructing the brain signal (e.g., fMRI) using a neural network

Image reconstruction examples

• Common problems of the brain-signal reconstruction approaches: – Voxels have a low spatial resolution

– Small datasets

• A possible alternative: in-vitro experiments

• Goal: – To access each biological neuron

– To collect large training datasets

Brain reading in the Backup project

Photonic circuits and optogenetics used to stimulate and read biological neuron activations

Accessing individual neurons

• Let B be the biological network with n neurons

• Let A be the ANN with m neurons

• A and B do NOT need to share the same structure

Biological net’s activation prediction: a possible schema

• Let st be the light-based stimulus of B at time t

• E.g., st Rn represents the individual-neuron stimulation

• In a sparse stimulus, for most i (1 <= i <= n), st,i = 0


• After a time delay (k), let rt+k, be the activation state of B, i.e., B’s "response" induced by st as measured at time t + k.

• rt+k Rn

• rt+k is the light-based “readout” of B


• We can collect a virtually unlimited dataset

D = {(st , rt+k)}

• D is used to train A


• A learns to predict rt+k from st

• Formally: A(st) = rt+k

• A is a "functional copy" (a backup…) of the memories of B





Hybrid computational systems

• Can we develop hybrid computational systems?

• Can we replace arbitrary parts of the biological network with an artificial network, still preserving the same functional behaviour?

Hybrid computational systems: a possible schema

• B is arbitrarily split in two sub-nets

• Analogously:

st = (st(1), st

(2))

rt+k = (rt+k(1), rt+k

(2))

• B1 is inhibited or removed

• B1 is replaced with an ANN A

• A is connected with B2


• A and B2 exchange information

• Goal: B2 ‘s response (rt+k

(2)) should be statistically similar to what is obtained without amputation





Overview

Neuromorphic Computing

Goal: to implement ANNs using photonic circuits

This is motivated by the much higher speed and lower power consumption of a photonic circuit w.r.t. an electric circuit

Our current solutions

• MultiLayer Perceptron (MLP)

• Reservoir Computing

Our current solutions



MLP using silicon photonics

• Our MLP is based on a microring resonator whose (thermal) nonlinear response corresponds to the neuron activation function


• Only light intensity is used (no phase information)


• Constraints:

– All net’s weights and activations should be positive

– The sum of the weights associated with the connections exiting from a neuron should be limited by 1


• We solved this constrained-optimization problem using a technique called Projected Gradient Descent

• In each SGD step, w is projected onto the admissible area defined by our constraints.


Preliminary Results:

• We used a software-based simulation

• MNIST dataset (60,000 28X28 digit images)

• MLP structure: 784-200-10

• Our simulation: 92% accuracy

• Standard, non-constrained ANN (same structure): 97%

• MNIST “linear regime”: 88%

Our solutions



Reservoir Computing (RC): Introduction

• It is an RNN with:

– input-to-hidden layer weights randomly fixed

– hidden-to-hidden layer weights randomly fixed

– hidden-to-output layer weights (readout) learned

• Main advantage: it is easy to train

• Can be implemented using photonic circuits


• Sparsity is obtained by setting most of the connection weights to 0:


• Difference with a standard RNN:

– Testing: no difference

– Training: only Who needs to be trained


RC: Our solution

• Our trainable last layer (Who) is a Perceptron

• Specifically, using both phase and intensity, we have complex-valued activations and weights (CV-Perceptron)

Our CV-Perceptron

• The output neuron activation function is the squared light intensity

• All computations, except the last activation function are performed using photonic circuits

• This is different from common implementations [1], in which Who ht is computed electronically

[1] Amin et al., Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing (72) 2009

Our CV-Perceptron: Simulation Results

Real datasets (normalized test error):

Symmetry detection (@ 0 error):

[1] Amin et al., Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing (72) 2009

Dataset Amin [1] (act-fun. 1)

Amin [1] (act-fun. 2)

RV Perceptron

Ours

Fisheriris 0.19 - 0.15 0.11

Diabetes 0.37 0.47 0.23 0.27

Cancer 0.026 0.025 0.014 0.08

Dataset Amin [1] (act-fun. 1)

Amin [1] (act-fun. 2)

RV Perceptron

Ours

Longest seq. 7 6 1 6

Thank you!

Questions?

Documents

Backup: Machine Learning - UniTrento Enver Sangineto... · 2020. 6. 9. · Backup: Machine Learning ... –Weakly-supervised Object Detection –Anomaly Detection –Few-Shot Learning