Semi-Supervised Classification with Graph Convolutional ... · 3/27/2019 · CNN 3 CNNs are...

Semi-Supervised Classification with Graph Convolutional Networks

Authors: Kipf, Thomas N., and Max Welling

Presenter: Parham Hamouni

Facilitators: Karim Khayrat

27 March 2019

Lunch and Learn

Overview● CNN

● When CNNs are powerful?

● Euclidean vs Non-Euclidean

● CNN on Euclidean Data

● GCN

● Why GCN works?

● GCN as Weisefeiler-Lehman (WL)

● Experiment and results

● CNNs are powerful to solve high dimensional learning problems.

● Is it possible to apply CNNs on Graphs?

Convolutional Neural Networks on Graphs, Xavier Bresson ,IPAM Workshop “New Deep Learning Techniques”, Feb 7th 2017, School of Computer Science and Engineering NTU, Singapore

When CNNs are powerful?● Main assumption

○ Euclidean Data (images, videos, sounds) that are

compositional

○ Extracting compositional features and feed them

to classifier, etc. (end-to-end).

● Euclidean domain

○ These domains have nice regular spatial

structures. (e.g. the distance between adjacent

pixels are the same)

○ All CNN operations are math well defined and

fast (convolution, pooling)

Non-Euclidean Data Domain

Examples include:

• Social Networks

• Web

• Molecule representation

• Knowledge graphs

Euclidean vs Non-Euclidean Data Domain

Source: Generalizing Convolutions for Deep Learning by Max Welling

CNN on Euclidean Data Domain

Graph Convolutional Network (GCN)

Why GCN works?Spectral Graph Convolution on arbitrary kernel

Spectral Graph Convolution on Graph Laplacian

A graph can be clustered by this operation, yet computationally expensive, as multiplication with the

eigenvector matrix U is O(N2).

Node Embeddings

Spectral Graph Clustering

10J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

How to reduce the computational cost?

Approximation of Spectral Graph Clustering● Well-approximated by a truncated expansion in terms of Chebyshev polynomials, ChebNet

● GCN is the approximation of ChebNet!

Chebyshev coefficients

● Renormalization trick:

● The complexity is now:

C-dimensional feature vector for every node,F filters or feature maps , E linear in the number of edges.

Graph Convolutional Network (GCN)

Semi-Supervised node classification task● Assumption: Connected nodes can possibly share labels

○ Input: some labelled and unlabelled data in a graph

○ Output: predict the node label of unlabeled nodes

● Embedding based approaches:

○ Embed each node to a numerical Euclidean Space

○ Train the classifier on them

● With GCN, the training would be end to end!

Karate club graph, colors denote communities obtained via modularity-based clustering (Brandes et al., 2008).

Overall View of the GCN

Loss Function

14Source: https://tkipf.github.io/graph-convolutional-networks/

GCN as WL?

Experiment setup

Results

Limitations

○ Memory requirement

○ Directed edges and edge features

○ Limiting assumptions

Key takeaways

● Graph and non-Euclidean domain needs special considerations to apply deep learning success

● GCN has mathematical explanation

● GCN is one of the most successful algorithms in deep learning on graphs

Discussion Points1. Why not more layers, just two layers of convolution?

2. Why spectral graph clustering uses Graph Laplacian matrix?

3. How to make GCN scalable to big data?

4. How is it different from gated graph neural network?

5. More recent frameworks such as graph attention networks have improved the state-of-the-art…

Thank youQ&A

Spectral graph clusteringThree basic stages:

1. Pre-processing

■ Construct a matrix representation of the graph

2. Decomposition

■ Compute eigenvalues and eigenvectors of the matrix

■ Map each point to a lower-dimensional representation based on one or more eigenvectors

3. Grouping

■ Assign points to two or more clusters, based on the new representation

Why eigenvalues and clustering?● For a symmetric matrix:

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

Why eigenvalues and clustering?●

Balance to minimize

Multiple partitioning● Two basic approaches:

○ Recursive bi-partitioning [Hagen et al., ’92]

■ Recursively apply bi-partitioning algorithm in a hierarchical divisive manner

■ Disadvantages: Inefficient, unstable

○ Cluster multiple eigenvectors [Shi-Malik, ’00]

■ Build a reduced space from multiple eigenvectors

■ Commonly used in recent papers

■ A preferable approach…

Semi-Supervised Classification with Graph Convolutional ... · 3/27/2019 · CNN 3 CNNs are...

Documents

AnApproachtoSemanticandStructuralFeaturesLearningfor ...downloads.hindawi.com/journals/mpe/2020/6038619.pdf3.2. Convolutional Neural Network. Convolutional neural network(CNN)isone

Convolutional Neural Networkscis581/Lectures/Fall2018/CNN-intr… · Convolutional Neural Network (CNN) Pipeline: A Monitor. Convolutional Neural Nets (CNNs) in a nutshell: • A

Architecture-aware design and implementation of CNN ...liacs.leidenuniv.nl/~stefanovtp/pdf/ICM_18.pdf · convolution layers in Convolutional Neural Networks (CNNs). Two successful

Deep Convolutional Neural Networks for Forensic Age ...blog.hakzone.info/.../uploads/2020/05/CNN-for-Forensic-Age-Estimation.… · 3. The Convolutional Neural Network (CNN) The most

CS11-747 Neural Networks for NLP Convolutional …...Outline 1. Feature Combinations 2. CNNs and Key Concepts 3. Case Study on Sentiment Classification 4. CNN Variants and Applications

Convolutional Neural Networks (CNN) in Medical and General ...€¦ · Convolutional Neural Network -CNN • Imaging physics and convolution mathematics to understand how each component

Interpretable Convolutional Neural Networkssczhu/papers/Conf_2018/CVPR... · 1. Introduction Convolutional neural networks (CNNs) [14,12,7] have achieved superior performance in many

Tube Convolutional Neural Network (T-CNN) for Action ...openaccess.thecvf.com/...Tube_Convolutional_Neural... · propose Tube Convolutional Neural Network (T-CNN) for action detection

Deep Learning - Convolutional Neural Network (CNNs) · Deep Learning Convolutional Neural Network (CNNs) Ali Ghodsi University of Waterloo October 30, 2015 Slides are partially based

Convolutional Neural Networks (CNNs) and Recurrent Neural

Deep Learning - Convolutional Neural Network (CNNs)i-systems.github.io/HSE545/machine learning all/16 Deep learning... · Deep Learning Convolutional Neural Network (CNNs) Ali Ghodsi

A Regularized Convolutional Neural Network for Semantic ...A REGULARIZED CNN FOR SEMANTIC IMAGE SEGMENTATION 3 We give experiments of our proposed method on two CNNs for image segmentation,

Convolutional Neural Networks CNN Partie I

Research Trends on Convolutional Neural Network (CNN

CNN-PDM: A Convolutional Neural Network Framework For

Rain Prediction Using Convolutional Neural Network (CNN

O-CNN: Octree-based Convolutional Neural Networks for … · O-CNN: Octree-based Convolutional Neural Networks for 3D ... object classification, ... Octree-based Convolutional Neural

Convolutional Neural Networks (CNNs) Structured Prediction …mlcenter.postech.ac.kr/files/attach/workshop_fall_2015... · · 2015-10-15Structured Prediction using CNNs By Prof

Maximizing CNN Accelerator Efficiency Through Resource ...mferdman/...1 INTRODUCTION The rapid adoption of convolutional neural networks (CNNs) has transformed machine learning. CNNs

Render for CNN: Viewpoint Estimation in Images Using CNNs … · 2015-09-25 · Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views Supplementary