Embed Size (px)
Citation preview
Graph Neural NetworksIrene Li @ LILY Group Meeting
25th, Oct
OutlineQuick Introduction
A Brief History, Earlier Research
Recent Papers: 3-4
Future Directions
Discussion
Introduction: Graphs
https://images.app.goo.gl/mnaK9vc1547Sd5UU8 Karate Network
Webpages
Citation networks
Social networks...
A brief history...
We will cover… and more
Early Research: DeepWalkLearning latent representations of vertices in a network.
Skip-Gram Model
GCN: the graph convolutional network
Focus on Graph structure data. How to understand traditional CNN?
From CNN to GCN (1)
http://petar-v.com/GAT/
Filter: shared with all locations
From CNN to GCN (2)
http://petar-v.com/GAT/
Find out rules to transform to the next layer
GCN: formulation
Input:
A is adjacency matrix
X is the feature matrix, shape N×D, can be one-hot
H is the outputs of the current layer l
Then choosing f.
Every neural network layer:
GCN: formulation
A single layer NN, with a ReLU as f.
W is the parameter matrix.
For the first layer, H0:
GCN: formulation
Two tricks:
1) About A (adjacency matrix, diagonal elements? A = A + I2) Normalize the adjacency matrix A, all rows sum up to 1?
Laplacian matrix normalization
From spectrum graph theory
Degree matrix D
A symmetric normalization?
GCN: formulation
GCN: formulation
PyTorch: https://github.com/tkipf/pygcn
http://opennmt.net/OpenNMT-py/main.html
Results: [supervised learning] classify the nodes.
Graph Attention Networks (ICLR, 2019) Self-attention: by looking at the neighbors with probabilities.
Which neighbor is having a larger impact/similar to it?Calculate a node pair (i,j):
Normalize over all the neighbors:
Use neighbors to represent the current node: weighted average
Multi-head
Results
Inductive experiments: protein-protein interaction, PPI (where in test graphs remain unseen during training).
About GATsBetter representation ability than GCN with attention mechanism.
Shared attention module: locally, efficient.
More: http://petar-v.com/GAT/
Variational Graph Auto-Encoders for link predictionRecall: encoder and decoder
Similar models on texts, images, ...
Can we do the same thing on graph-structure data?
Move to graphs...
Details of VGAETwo layers of GCN as the encoder.
An inner product as the Decoder
A simple version (without ‘variation’) ...
Results: [Semi-supervised] Classify edges/non-edges
The models are trained on an incomplete version of these datasets where parts of the citation links (edges) have been removed, while all node features are kept. Complete X; incomplete A → A
Keep It Simple: Graph Autoencoders Without Graph Convolutional Networks
Replace GCN encoder with a linear model wrt the adjacency matrix of the graph and a unique weight matrix.
Takeaways:
Dense datasets: Blogs, Google pages, etc. GCN encoder performance increases with the size of the graph.
Nature of the dataset is crucial: in citation graphs, if a reference A in an article B cited by some authors is relevant to their work, authors will likely also cite this reference A (creating a first order link)
Recent ResearchWe covered:
Conv GNNs +
Graph Autoencoders
Future DirectionsHeterogeneity: handle different types of nodes and edges; various forms of inputs, such as images and texts!
Multi-filters paper
Multi-edge Aggregation (MAE) : Using Multi-view of edge relations
Multiple node representations: Is a Single Embedding Enough? Learning Node Representations..
Dynamicity: node/edge inputs may change time by time (dynamic spatial relations).
Social network in a period of time...
Discussion
What applications do neural graph networks can solve? (Anti-Money Laundering...)
What tasks in NLP can be solved? (Parse trees, knowledge graphs…)
Do you think GCNs can be very deep? (100+ layers?)
Related Linkshttps://sites.google.com/view/deep-gcns
Leaderboard: https://paperswithcode.com/sota
https://www.inference.vc/how-powerful-are-graph-convolutions-review-of-kipf-welling-2016-2/
http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf
ThanksQ&A
