Measuring and Relieving the Over-smoothing Problem for Graph NeuralNetworks from the Topological View

Deli Chen,1 Yankai Lin,2 Wei Li,1 Peng Li,2 Jie Zhou,2 Xu Sun1

1MOE Key Lab of Computational Linguistics, School of EECS, Peking University2Pattern Recognition Center, WeChat AI, Tencent Inc., China

{chendeli,liweitj47,xusun}@pku.edu.cn, {yankailin,patrickpli,withtomzhou}@tencent.com,

Abstract

Graph Neural Networks (GNNs) have achieved promisingperformance on a wide range of graph-based tasks. De-spite their success, one severe limitation of GNNs is theover-smoothing issue (indistinguishable representations ofnodes in different classes). In this work, we present a sys-tematic and quantitative study on the over-smoothing is-sue of GNNs. First, we introduce two quantitative metrics,MAD and MADGap, to measure the smoothness and over-smoothness of the graph nodes representations, respectively.Then, we verify that smoothing is the nature of GNNs andthe critical factor leading to over-smoothness is the lowinformation-to-noise ratio of the message received by thenodes, which is partially determined by the graph topol-ogy. Finally, we propose two methods to alleviate the over-smoothing issue from the topological view: (1) MADRegwhich adds a MADGap-based regularizer to the training ob-jective; (2) AdaEdge which optimizes the graph topologybased on the model predictions. Extensive experiments on7 widely-used graph datasets with 10 typical GNN modelsshow that the two proposed methods are effective for re-lieving the over-smoothing issue, thus improving the perfor-mance of various GNN models.

Introduction1Graph Neural Networks form an effective framework forlearning graph representation, which have proven powerfulin various graph-based tasks (Yang, Cohen, and Salakhut-dinov 2016; Velickovic et al. 2018; Zitnik and Leskovec2017). Despite their success in graph modeling, over-smoothing is a common issue faced by GNNs (Li, Han, andWu 2018; Zhou et al. 2018), which means that the represen-tations of the graph nodes of different classes would becomeindistinguishable when stacking multiple layers, which se-riously hurts the model performance (e.g., classification ac-curacy). However, there is limited study on explaining whyand how over-smoothing happens. In this work, we conducta systematic and quantitative study of the over-smoothingissue of GNNs on 7 widely-used graph datasets with 10 typ-ical GNN models, aiming to reveal what is the crucial factor

1Accepted by AAAI 2020. This complete version contains theappendix.

Figure 1: The prediction accuracy (Acc) and MADGap ofGCNs (Kipf and Welling 2017) on the CORA dataset. Wecan observe a significantly high correlation between the ac-curacy and MADGap in two different situations: (a) differentmodels: results of GCNs with different number of layers; (b)different training periods: results after each epoch in the 2-layer GCN. The Pearson correlation coefficient is shown inthe title and ** means statistically significant with p < 0.01.

bringing in the over-smoothing problem of GNNs and findout a reasonable direction to alleviate it.

We first propose a quantitative metric Mean Average Dis-tance (MAD), which calculates the mean average distanceamong node representations in the graph to measure thesmoothness of the graph (smoothness means similarity ofgraph nodes representation in this paper). We observe thatthe MAD values of various GNNs become smaller as thenumber of GNN layers increases, which supports the argu-ment that smoothing is the essential nature of GNNs. Hence,the node interaction through the GNN message propaga-tion would make their representations closer, and the wholegraph representation would inevitably become smoothingwhen stacking multiple layers.

Furthermore, we argue that one key factor leading to theover-smoothing issue is the over-mixing of information andnoise. The interaction message from other nodes may be ei-ther helpful information or harmful noise. For example, inthe node classification task, intra-class interaction can bringuseful information, while inter-class interaction may leadto indistinguishable representations across classes. To mea-sure the quality of the received message by the nodes, wedefine the information-to-noise ratio as the proportion of

arX

iv:1

909.

0321

1v2

[cs

.LG

] 1

8 N

ov 2

019

intra-class node pairs in all node pairs that have interac-tions through GNN model. Based on our hypothesis, we ex-tend MAD to MADGap to measure the over-smoothness ofgraph (over-smoothness means similarity of representationsamong different classes’ nodes in this paper). We notice thattwo nodes with close topological distance (can reach with afew hops) are more likely to belong to the same class, andvice versa. Therefore, we differentiate the role between re-mote and neighboring nodes and calculate the gap of MADvalues (MADGap) between remote and neighboring nodesto estimate the over-smoothness of graph representation. Ex-perimental results prove that MADGap does have a signifi-cantly high correlation with the model performance in gen-eral situations, and an example is shown in Figure 1. Fur-ther experiments show that both the model performance andthe MADGap value rise as the information-to-noise ratio in-creases, which verifies our assumption that the information-to-noise ratio affects the smoothness of graph representationto a great extent.

After more in-depth analysis, we propose that lowinformation-to-noise ratio is caused by the discrepancy be-tween the graph topology and the objective of the down-stream task. In the node classification task, if there are toomany inter-class edges, the nodes will receive too much mes-sage from nodes of other classes after several propagationsteps, which would result in over-smoothing. To prove ourassumption, we optimize the graph topology by removinginter-class edges and adding intra-class edges based on thegold labels, which proves very effective in relieving over-smoothing and improving model performance. Hence, thegraph topology has a great influence on the smoothness ofgraph representation and model performance. That is to say,there is a deviation from the natural graph to the down-stream task. However, in the previous graph-related stud-ies (Velickovic et al. 2018; Li et al. 2016; Bianchi et al.2019), researchers mainly focus on designing novel GNN ar-chitectures but pay less attention to improve the establishedgraph topology.

Based on our observations, we propose two methods torelieve the over-smoothing issue from the topological view:(a) MADReg: we add a MADGap-based regularizer tothe training objective to directly increase received infor-mation and reduce noise; (b) Adaptive Edge Optimization(AdaEdge2): we iteratively train GNN models and conductedge remove/add operations based on the prediction to ad-just the graph adaptively for the learning target. Experimen-tal results show that our two proposed methods can signifi-cantly relieve the over-smoothing issue and improve modelperformance in general cases, which further verifies our con-clusions and provides a compelling perspective towards bet-ter GNNs performance.

The contributions of this work are threefold:

• We conduct a systematic and quantitative study of theover-smoothing issue on a wide range of graph datasetsand models. We propose and verify that a key factor be-

2The algorithm name is changed from the previous AdaGraphto AdaEdge since the conflicting using of AdaGraph with otherwork.

Model PropagateGCN (Kipf and Welling 2017) ConvolutionChebGCN (Defferrard, Bresson, and Vandergheynst 2016) Convolution

HyperGraph (Bai, Zhang, and Torr 2019) Convolution&Attention

FeaSt (Verma, Boyer, and Verbeek 2018) ConvolutionGraphSAGE (Hamilton, Ying, and Leskovec 2017) ConvolutionGAT (Velickovic et al. 2018) AttentionARMA (Bianchi et al. 2019) ConvolutionGraphSAGE (Hamilton, Ying, and Leskovec 2017) ConvolutionHighOrder (Morris et al. 2019) AttentionGGNN (Li et al. 2016) Gated

Table 1: Introduction of baseline GNN models. The infor-mation propagation method is also displayed.

hind the over-smoothing issue is the information-to-noiseratio which is influenced by the graph topology.

• We design two quantitative metrics: MAD for smoothnessand MADGap for over-smoothness of graph representa-tion. Statistical analysis shows that MADGap has a sig-nificantly high correlation with model performance.

• We propose two methods: MADReg and AdaEdge to re-lieve the over-smoothing issue of GNNs. Experimental re-sults show that our proposed methods can significantlyreduce over-smoothness and improve the performance ofmultiple GNNs on various datasets.

Datasets and Modelsnode classification task, one of the most basic graph-based tasks, is usually conducted to verify the effective-ness of GNN architectures (Velickovic et al. 2018; Hamil-ton, Ying, and Leskovec 2017) or analyze the character-istics of GNNs (Li, Han, and Wu 2018; Maehara 2019).Therefore, we select the node classification task for our ex-periments. We conduct experiments on 7 public datasetsin three types, namely, (1) citation network: CORA, Cite-Seer, PubMed (Sen et al. 2008); (2) coauthor network:CS, Physics;3 (3) Amazon product network: Computers,Photo (McAuley et al. 2015). We conduct our detailed anal-ysis on the three citation networks, which are usually takenas the benchmarks for graph-related studies (Li, Han, andWu 2018; Maehara 2019) and verify the effectiveness of theproposed method on all these datasets.

To guarantee the generalizability of our conclusion, weconduct experiments with 10 typical GNN models in thiswork. The GNN models and their propagation methods arelisted in Table 1, in which the propagation taxonomy fol-lows Zhou et al. (2018). The implementation of the base-lines is partly based on Fey and Lenssen (2019) and Mae-hara (2019). More details about the datasets and experimen-tal settings are given in Appendix A.

Measuring Over-smoothing Problem from theTopological View

In this section, we aim to investigate what is the key factorleading to the over-smoothing problem. To this end, we pro-pose two quantitative metrics MAD and MADGap to mea-

3https://kddcup2016.azurewebsites.net

sure the smoothness and over-smoothness of graph repre-sentation, which are further used to analyze why and howthe over-smoothing issue happens.

MAD: Metric for SmoothnessTo measure the smoothness of the graph representation, wefirst propose a quantitative metric: Mean Average Distance(MAD). MAD reflects the smoothness of graph representa-tion by calculating the mean of the average distance fromnodes to other nodes. Formally, given the graph representa-tion matrix H ∈ Rn×h (we use the hidden representation ofthe final layer of GNN. Term h is the hidden size), we firstobtain the distance matrix D ∈ Rn×n for H by computingthe cosine distance between each node pair:

Dij = 1− Hi,: ·Hj,:

|Hi,:| · |Hj,:|i, j ∈ [1, 2, · · · , n], (1)

where Hk,: is the k-th row of H . The reason to use cosinedistance is that cosine distance is not affected by the absolutevalue of the node vector, thus better reflecting the smooth-ness of graph representation. Then we filter the target nodepairs by element-wise multiplication D with a mask matrixM tgt

Dtgt = D ◦M tgt, (2)where ◦ denotes element-wise multiplication; M tgt ∈{0, 1}n×n; M tgt

ij = 1 only if node pair (i, j) is the targetone. Next we access the average distance Dtgt for non-zerovalues along each row in Dtgt:

Dtgti =

∑nj=0 D

tgtij∑n

j=0 1(Dtgt

ij

) , (3)

where 1(x) = 1 if x > 0 otherwise 0. Finally, the MADvalue given the target node pairs is calculated by averagingthe non-zero values in Dtgt:

MADtgt =

∑ni=0 D

tgti∑n

i=0 1(Dtgt

i

) . (4)

Li, Han, and Wu (2018) perform a theoretical analysison the graph convolution network (GCN), and conclude thatperforming smoothing operation on node representations isthe key mechanism why GCN works. We extend the con-clusion empirically to the 10 typical GNNs listed in Table 1with the help of the proposed MAD metric. To this end, foreach GNN model with different number of layers, we com-pute the MAD value MADglobal by taking all node pairs intoaccount, i.e., all values in M tgt are 1, to measure the globalsmoothness of the learned graph representation.

The results on the CORA dataset are shown in Figure 2.We can observe that as the number of GNN layers increases,the MAD values become smaller. Apart from this, the MADvalue of high-layer GNNs gets close to 0, which means thatall the node representations become indistinguishable. GNNmodels update the node representation based on the featuresfrom neighboring nodes. We observe that the interactionbetween nodes makes their representations similar to eachother. Similar phenomenons that the smoothness rises as thelayer increases are also observed in other datasets as pre-sented in Appendix B. Therefore, we conclude that smooth-ing is an essential nature for GNNs.

2 3 4 5 6#Model Layer

ARMA

ChebGCN

DNA

FeaSt

GAT

GCN

GGNN

GraphSAGE

HighOrder

HyperGraph

Mod

el

0.629 0.860 0.608 0.305 0.004

0.557 0.756 0.138 0.024 0.018

0.665 0.352 0.347 0.172 0.096

0.778 0.770 0.677 0.182 0.072

0.794 0.704 0.232 0.047 0.005

0.796 0.765 0.714 0.602 0.289

0.661 0.078 0.021 0.033 0.039

0.925 0.816 0.632 0.303 0.053

0.629 0.145 0.023 0.004 0.012

0.828 0.742 0.493 0.046 0.0230.2

0.0

0.2

0.4

0.6

0.8

1.0

Figure 2: The MAD values of various GNNs with differ-ent layers on the CORA dataset. Darker color means largerMAD value. We can find that the smoothness of graph rep-resentation rises as the model layer increases.

Information-to-noise Ratio Largely AffectsOver-smoothnessWith the help of MAD, we can quantitatively measure thesmoothness of graph representation. Here come two newquestions: since smoothing is the nature of GNNs, what isover-smoothing, and what results in over-smoothing?

We assume that the over-smoothing problem is caused bythe over-mixing of information and noise, which is influ-enced by the quality of the nodes received message. The in-teraction message from other nodes by GNN operation maybe either helpful information or interference noise. For ex-ample, in the node classification task, interaction betweennodes of the same class brings useful information, whichmakes their representations more similar to each other andthe probability of being classified into the same class is in-creased. On the contrary, the contact of nodes from otherclasses brings the noise. Hence, the reason why GNNs workis that the received useful information is more than noise.On the other hand, when the noise is more than the infor-mation, the learned graph representation will become over-smoothing.

To quantitatively measure the quality of the received mes-sage of the nodes, we define the information-to-noise ratioas the proportion of intra-class node pairs in all contactablenode pairs that have interactions through the GNN model.For example, at the second-order, the information-to-noiseratio for each node is the proportion of nodes of the sameclass in all the first-order and second-order neighbors; theinformation-to-noise ratio for the whole graph is the propor-tion of the intra-class pairs in all the node pairs that can becontacted in 2 steps. In Figure 3, we display the information-to-noise ratio of the whole graph for the CORA, CiteSeerand Pubmed datasets. We can find that there are more intra-class node pairs at low order and vice versa. When the modellayer number gets large where the information-to-noise ra-

0 2 4 6 8 10 12 14 16 18 20Order

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Info

rmat

ion-

to-n

oise

Rat

ioCORACiteSeerPubmed

Figure 3: The information-to-noise ratio at different neigh-bor orders (accumulated) for the CORA/CiteSeer/PubMeddatasets. We can find that the information-to-noise ratio de-clines as the orders increases in all these three datasets.

tio is small, the interaction between high-order neighborsbrings too much noise and dilutes the useful information,which is the reason for the over-smoothing issue. Based onthis observation, we extend MAD to MADGap to measurethe over-smoothness in the graph representation. From Fig-ure 3 we notice that two nodes with small topological dis-tance (low-order neighbours) are more likely to belong to thesame category. Hence, we propose to utilize the graph topol-ogy to approximate the node category, and calculate the gapof MAD values differentiating remote and neighbour nodesto estimate the over-smoothness of the graph representation,

MADGap = MADrmt −MADneb, (5)

where MADrmt is the MAD value of the remote nodes inthe graph topology and MADneb is the MAD value of theneighbouring nodes.

According to our assumption, large MADGap value in-dicates that the useful information received by the node ismore than noise. At this time, GNNs perform reasonableextent of smoothing, and the model would perform well.On the contrary, small or negative MADGap means over-smoothing and inferior performance. To verify the effective-ness of MADGap, we calculate the MADGap value4 andcompute the Pearson coefficient between the MADGap andthe prediction accuracy for various GNN models. We reportthe Pearson coefficient for GNNs with different layers onCORA, CiteSeer and PubMed datasets in Table 2. Accord-ing to the table, we can find that there exists a significantlyhigh correlation between MADGap and the model perfor-mance, which validates that MADGap is a reliable metricto measure graph representation over-smoothness. Besides,MADGap can also be used as an observation indicator to es-timate the model performance based on the graph topologywithout seeing the gold label. It is worth noting that 1-layerGNN usually has small MADGap and prediction accuracy(Figure 1), which is caused by the insufficient information

4In this work, we calculate MADneb based on nodes with or-ders ≤ 3 and MADrmt based on nodes with orders ≥ 8.

Model CORA CiteSeer PubMedGCN 0.986∗∗ 0.948∗∗ 0.971∗∗

ChebGCN 0.945∗∗ 0.984∗∗ 0.969∗∗

HyperGraph 0.990∗∗ 0.965∗∗ 0.932∗∗

FeaSt 0.993∗∗ 0.986∗∗ 0.906∗

GraphSAGE 0.965∗∗ 0.995∗ 0.883GAT 0.960∗∗ 0.998∗∗ 0.965∗∗

ARMA 0.909∗ 0.780 0.787HighOrder 0.986∗∗ 0.800 0.999∗∗

DNA 0.945∗∗ 0.884∗ 0.887∗

GGNN 0.940∗ 0.900∗ 0.998∗∗

Table 2: The Pearson coefficient between accuracy andMADGap for various models on CORA/CiteSeer/PubMeddatasets. Pearson coefficient is calculated based on the re-sults of models with different layers (1-6). * means statisti-cally significant with p < 0.05 and ** means p < 0.01.

transfer, while the over-smoothing issue of high-layer GNNis caused by receiving too much noise.

In Figure 4, we show the MADGap and prediction ac-curacy for node sets with different information-to-noise ra-tios in the same model. We can find that even with thesame model and propagation step, nodes with higher rate ofinformation-to-noise ratio generally have higher predictionaccuracy with smaller over-smoothing degree. We also ob-serve similar phenomena on other datasets, which are shownin Appendix C. This way, we further verify that it is theinformation-to-noise ratio that affects the graph representa-tion over-smoothness to a great extent, thus influencing themodel performance.

Topology Affects the Information-to-noise RatioFrom the previous analysis, we can find that the key fac-tor influencing the smoothness of graph representation is theinformation-to-noise ratio. Then the following question is:what affects the information-to-noise ratio? We argue thatit is the graph topology that affects the information-to-noiseratio. The reason for the node receiving too much noise isrelated to the discordance between the natural graph andthe task objective. Take node classification as an example. Ifthere are too many inter-class edges, the nodes will receivetoo much noise after multiple steps of message propagation,which results in over-smoothing and bad performance.

The graph topology is constructed based on the naturallinks. For example, the edges in the citation network rep-resent the citing behavior between papers and edges in theproduct network represent the products co-purchasing rela-tions. GNN models rely on these natural links to learn noderepresentations. However, natural links between nodes ofdifferent classes are harmful to the node classification task.Therefore, we propose to alleviate the over-smoothing issueof GNNs and improve their performance by optimizing thegraph topology to match the downstream task.

To verify our assumption, we optimize the graph topologyby removing inter-class edges and adding intra-class edgesbased on the gold labels. The results on the CORA datasetare shown in Figure 5. We can find that the MADGap valuerises consistently as more inter-class edges are removed and

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

0.2

0.4

0.6

0.8

valu

emodel = GCN

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GAT

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GraphSAGE

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = ChebGCN

typeAccMADGap

Figure 4: Performance (accuracy) and over-smoothness (MADGap) of node sets with different information-to-noise ratio (e.g.,0.1 means ratio≤ 0.1) on the CORA dataset (We display 4 out of 10 models results due to the limited space. We observe similarresults in other models). All models have 2 layers. Results prove that nodes with higher information-to-noise ratio would haveless over-smoothness degree and better prediction result.

Figure 5: The gold label based topology adjustment experiment on the CORA dataset. We show the results of both removinginter-class edges (first row, where the X-axis represents the removing rate) and adding intra-class edges (second row, where theX-axis represents the intra-class edge ratio compared to the raw graph) on GCN, GAT, GraphSAGE and ChebGCN. Resultsshow that both of these methods are very helpful for relieving the over-smoothing issue and improving model performance.

more intra-class edges are added, resulting in better modelperformance. Therefore, optimizing graph topology is help-ful in relieving the over-smoothing problem and improvingmodel performance.

In summary, we find that the graph topology has a greatinfluence on the smoothness of graph representation andmodel performance. However, there is still discordance be-tween the natural links and the downstream tasks. Most ex-isting works mainly focus on designing novel GNN architec-tures but pay less attention to the established graph topology.Hence, we further investigate to improve the performance ofGNNs by optimizing the graph topology.

Relieving Over-smoothing Problem from theTopological View

Inspired by the previous analysis, we propose two meth-ods to relieve the over-smoothing issue from the topologi-cal view: (1) MADReg: we add a MADGap-based regular-izer to the training objective; (2) Adaptive Edge Optimiza-tion (AdaEdge): we adjust the graph topology adaptivelyby iteratively training GNN models and conducting edge re-move/add operations based on the prediction result. Neither

of these two methods is restricted to specific model archi-tectures and can be used in the training process of generalGNN models. Experiments demonstrate their effectivenessin a variety of GNNs.

MADReg: MADGap as RegularizerIn the previous experiments, we find that MADGap showsa significantly high correlation with model performance.Hence, we add MADGap to the training objective to makethe graph nodes receive more useful information and lessinterference noise:

L =∑−l log p( l |X,A,Θ)− λMADGap, (6)

where X is the input feature matrix, A is the adjacency ma-trix, l and l are the predicted and gold labels of the noderespectively. Θ is the parameters of GNN and λ is the reg-ularization coefficient to control the influence of MADReg.We calculate MADGap on the training set to be consistentwith the cross-entropy loss.

AdaEdge: Adaptive Edge OptimizationAs discussed in the previous section, after optimizing thetopology based on gold label (adding the intra-class edges

Figure 6: MADReg and AdaEdge results on the CORA/CiteSeer/PubMed datasets. The number of GNN layers is 4, where theover-smoothing issue is severe. The box plot shows the mean value and the standard deviation of the prediction accuracy andthe MADGap values of 50 turns results (5 dataset splitting methods and 10 random seeds for each splitting following Shchuret al. (2018) and Sun, Koniusz, and Wang (2019). More details can be found in Appendix A). And we can find that the twoproposed methods can effectively relieve the over-smoothing issue and improve model performance in most cases.

and removing the inter-class edges), the over-smoothingissue is notably alleviated, and the model performanceis greatly improved. Inspired by this, we propose a self-training algorithm called AdaEdge to optimize the graphtopology based on the prediction result of the model to adap-tively adjust the topology of the graph to make it more rea-sonable for the specific task objective. Specifically, we firsttrain GNN on the original graph and adjust the graph topol-ogy based on the prediction result of the model by deletinginter-class edges and adding intra-class edges. Then we re-train the GNN model on the updated graph from scratch.We perform the above graph topology optimization opera-tion multiple times. The details of the AdaEdge algorithmare introduced in Appendix D.

Relieving Over-smoothing in High-order Layers

To verify the effectiveness of the two proposed methods, weconduct controlled experiments for all the 10 baseline GNNmodels on CORA/CiteSeer/PubMed datasets. We calculatethe prediction accuracy and MADGap value for the GNNmodels with 4 layers, where the over-smoothing issue is se-rious. The results are shown in Figure 6. We present 6 outof 10 models results due to the space limit; the other mod-els can be found in Appendix E. We can find that in thehigh-order layer situation where the over-smoothing issueis severe, the MADReg and AdaEdge methods can effec-tively relieve the over-smoothing issue and improve modelperformance for most models in all three datasets. The ef-fectiveness of MADReg and AdaEdge further validates ourassumption and provides a general and effective solution torelieve the over-smoothing problem.

Improving Performance of GNNsIn Table 3, we show the controlled experiments for GNNmodels trained on the original graph and the updated graphobtained by the AdaEdge method on all the 7 datasets. Weselect the best hyper-parameters when training GNN on theoriginal graph and fix all these hyper-parameters when train-ing on the updated graph. Experimental results show thatthe AdaEdge method can effectively improve the model per-formance in most cases, which proves that optimizing thegraph topology is quite helpful for improving model perfor-mance. We analyze the cases of the AdaEdge method withlittle or no improvement and find that this is caused by theincorrect operations when adjusting the topology. Therefore,when the ratio of incorrect operations is too large, it willbring serious interference to the model training and bringin little or no improvement. Due to the space limit, the re-sults of MADReg are shown in Appendix F. Typically, thebaselines achieve their best performance with small num-ber of GNN layers, where the over-smoothing issue is notsevere. Under this condition, MADReg can hardly improvethe performance by enlarging the MADGap value. However,when the over-smoothing issue becomes more severe whilethe GNN layer number grows larger, MADReg is still ca-pable of improving the performance of the baselines signifi-cantly. Above all, both AdaEdge and MADReg are effectivefor improving GNNs performance, and AdaEdge general-izes better when the over-smoothing issue is not severe.

Related WorkGraph Neural Networks (GNNs)GNNs have proven effective in various non-Euclidean graphstructures, such as social network (Hamilton, Ying, andLeskovec 2017), biology network (Zitnik and Leskovec

Acc(%) CORA CiteSeer PubMed Amazon Photo Amazon Comp. Coauthor CS Coauthor Phy.Model baseline +AE baseline +AE baseline +AE baseline +AE baseline +AE baseline +AE baseline +AEGCN 81.2±0.8 82.3∗∗±0.8 69.3±0.7 69.7∗∗±0.9 76.3±0.5 77.4∗∗±0.5 90.6±0.7 91.5∗∗±0.5 81.7±0.7 82.4∗∗±1.1 89.8±0.3 90.3∗∗±0.4 92.8±1.6 93.0∗∗±1.1

ChebGCN 78.6±0.6 80.1∗∗±0.5 67.4±1.0 67.8∗±1.2 76.7±0.1 77.5∗∗±0.6 89.6±1.6 89.4∗±1.2 80.8±2.4 81.3∗∗±1.1 90.5±0.4 90.7∗±0.3 \ \HyperGraph 80.5±0.6 81.4∗∗±0.8 67.9±0.5 68.5∗∗±0.5 77.4±0.2 77.3±0.7 87.5±0.7 88.6∗∗±0.3 58.7±22.1 61.4∗∗±26.8 86.9±0.5 87.3∗∗±0.4 91.9±2.0 92.2∗∗±1.4

FeaSt 80.4±0.7 81.6∗∗±0.7 69.3±1.1 69.4∗±1.0 76.6±0.6 77.2∗±0.4 90.5±0.6 90.8∗∗±0.6 80.8±1.3 81.7∗∗±0.9 88.4±0.2 88.9∗∗±0.2 \ \GraphSAGE 78.5±1.7 80.2∗∗±1.2 68.4±0.9 69.4∗∗±0.8 75.2±1.1 77.2∗∗±0.8 90.1±1.4 90.6∗∗±0.5 80.2±1.0 81.1∗∗±1.0 90.1±0.4 90.3∗∗±0.4 93.0±0.4 92.7±0.2

GAT 76.3±3.1 77.9∗∗±2.0 68.9±0.6 69.1∗±0.8 75.9±0.5 76.6∗∗±0.2 89.7±1.7 90.8∗∗±0.9 81.4±1.5 81.1∗±1.6 85.5±1.9 86.6∗∗±1.6 91.1±1.0 91.4∗±1.0

ARMA 74.9±10.6 76.4∗∗±5.6 65.3±4.1 66.1∗∗±4.3 68.5±11.4 68.9∗±12.2 86.4±3.0 87.0∗∗±1.9 63.8±18.9 71.7∗∗±8.1 90.6±1.1 90.9∗∗±0.6 92.2±1.8 92.6∗±1.0

HighOrder 76.6±1.2 72.5∗∗±4.1 64.2±1.0 63.3∗∗±1.0 75.0±2.6 76.9∗∗±1.3 26.1±12.4 30.3∗∗±10.2 26.3±12.7 23.9∗±13.4 84.2±1.0 85.6∗∗±0.7 90.8±0.8 90.9±0.6

DNA 58.2±14.4 60.1∗±10.8 60.9±2.7 61.3∗∗±2.2 65.8±7.8 66.8∗±9.6 89.1±1.3 89.8∗∗±0.6 78.2±2.9 79.8∗∗±2.0 88.2±0.9 90.0∗∗±0.6 93.0±0.5 93.3∗±0.4

GGNN 47.3±6.1 44.7∗∗±3.5 55.5±2.8 47.9∗∗±3.4 66.1±4.4 69.5∗∗±1.2 74.1±12.3 80.6∗∗±7.2 42.4±26.7 61.5∗∗±20.8 86.6±1.4 88.2∗∗±0.8 91.2±1.2 91.6∗∗±0.7

Table 3: Controlled experiments of AdaEdge (+AE) on all the 7 datasets. We show the mean value, the standard deviation andthe t-test significance of 50 turns results. * means statistically significance with p < 0.05 and ** means p < 0.01. Darker colormeans larger improvement. The missing results are due to the huge consumption of GPU memory of large graphs.

2017), business graph (McAuley et al. 2015) and academicgraph (Sen et al. 2008). Recently, many novel GNN architec-tures have been developed for graph-based tasks. Velickovicet al. (2018) propose the graph attention network to useself-attention to aggregate information from neighboringnodes. Hamilton, Ying, and Leskovec (2017) propose ageneral inductive framework to generate node embeddingby sampling and aggregating features from the neighbor-ing nodes. There are also other GNNs proposed, such asARMA (Bianchi et al. 2019), FeaSt (Verma, Boyer, and Ver-beek 2018), HyperGraph (Bai, Zhang, and Torr 2019) and soon. Xu et al. (2018) propose jumping knowledge networks tohelp the GNN model to leverage the information from high-order neighbours for a better node representation. However,all these models focus on improving the information prop-agation and aggregation operation on the static graph whilepaying less attention to the graph topology. In this work, wepropose to explicitly optimize the graph topology to make itmore suitable for the downstream task.

Pareja et al. (2019) propose the EvolveGCN that uses theRNN to evolve the graph model itself over time. Fey (2019)allow for a selective and node-adaptive aggregation ofthe neighboring embeddings of potentially differing local-ity. Yang et al. (2019b) propose a new variation of GCN byjointly refining the topology and training the fully connectednetwork. These existing works about dynamic graph rely onthe adaptive ability of the model itself and focus on specialGNN architecture (e.g., GCN), while our AdaEdge methodoptimizes the graph topology with a clear target (addingintra-class edges and removing inter-class edges) and canbe used in general GNN architectures. Rong et al. (2019)propose DropEdge method to drop edges randomly at eachtraining epoch for data augmentation while our AdaEdgemethod adjusts edges before training to optimize the graphtopology.

Over-smoothing Problem in GNNsPrevious works (Li, Han, and Wu 2018; Zhou et al. 2018)have proven that over-smoothing is a common phenomenonin GNNs. Li, Han, and Wu (2018) prove that the graph con-volution of the GCN model is actually a special form ofLaplacian smoothing. Deng, Dong, and Zhu (2019) pro-pose that smoothness is helpful for node classification anddesign methods to encourage the smoothness of the outputdistribution, while Yang et al. (2019a) propose that nodes

may be mis-classified by topology based attribute smooth-ing and try to overcome this issue. In this work, we provethat smoothing is the essential feature of GNNs, and thenclassify the smoothing into two kinds by the information-to-noise ratio: reasonable smoothing that makes GNN work,and over-smoothing that causes the bad performance. Fromthis view, the methods from Deng, Dong, and Zhu (2019)and Yang et al. (2019a) can be regarded as improving rea-sonable smoothing and relieve over-smoothing, respectively.Besides, Li et al. (2019a) propose to use LSTM in GNNto solve over-smoothing issue in text classification. How-ever, existing works usually mention the over-smoothingphenomenon, but there lacks systematic or quantitative re-search about it.

Conclusion and Future WorkIn this work, we conduct a systematic and quantitative studyof the over-smoothing issue faced by GNNs. We first de-sign two quantitative metrics: MAD for smoothness andMADGap for over-smoothness. From the quantitative mea-surement results on multiple GNNs and graph datasets, wefind that smoothing is the essential nature of GNNs; over-smoothness is caused by the over-mixing of information andthe noise. Furthermore, we find that there is a significantlyhigh correlation between the MADGap and the model per-formance. Besides, we prove that the information-to-noiseratio is related to the graph topology, and we can relieve theover-smoothing issue by optimizing the graph topology tomake it more suitable for downstream tasks. Followingly,we propose two methods to relieve the over-smoothing is-sue in GNNs: the MADReg and the AdaEdge methods. Ex-tensive results prove that these two methods can effectivelyrelieve the over-smoothing problem and improve model per-formance in general situations.

Although we have shown optimizing graph topology isan effective way of improving GNNs performance, our pro-posed AdaEdge method still suffers from the wrong graphadjustment operation problem. How to reduce these opera-tions is a promising research direction.

AcknowledgementThis work was supported in part by a Tencent ResearchGrant and National Natural Science Foundation of China(No. 61673028). Xu Sun is the corresponding author of thispaper.

ReferencesBai, S.; Zhang, F.; and Torr, P. H. 2019. HypergraphConvolution and Hypergraph Attention. arXiv preprintarXiv:1901.08150.Bianchi, F. M.; Grattarola, D.; Livi, L.; and Alippi, C. 2019.Graph Neural Networks with Convolutional Arma Filters.arXiv preprint arXiv:1901.01343.Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016.Convolutional Neural Networks on Graphs with Fast Local-ized Spectral Filtering. In Advances in Neural InformationProcessing Systems, 3837–3845.Deng, Z.; Dong, Y.; and Zhu, J. 2019. Batch virtual ad-versarial training for graph convolutional networks. arXivpreprint arXiv:1902.09192.Fey, M., and Lenssen, J. E. 2019. Fast Graph RepresentationLearning with PyTorch Geometric. In ICLR Workshop onRepresentation Learning on Graphs and Manifolds.Fey, M. 2019. Just Jump: Dynamic NeighborhoodAggregation in Graph Neural Networks. arXiv preprintarXiv:1904.04849.Hamilton, W. L.; Ying, Z.; and Leskovec, J. 2017. InductiveRepresentation Learning on Large Graphs. In Advances inNeural Information Processing Systems, 1024–1034.Kipf, T. N., and Welling, M. 2017. Semi-supervised Clas-sification with Graph Convolutional Networks. In 5th In-ternational Conference on Learning Representations, ICLR2017.Li, Y.; Tarlow, D.; Brockschmidt, M.; and Zemel, R. S. 2016.Gated Graph Sequence Neural Networks. In 4th Interna-tional Conference on Learning Representations, ICLR 2016.Li, W.; Li, S.; Ma, S.; He, Y.; Chen, D.; and Sun, X. 2019a.Recursive graphical neural networks for text classification.arXiv preprint arXiv:1909.08166.Li, W.; Xu, J.; He, Y.; Yan, S.; Wu, Y.; and Sun, X. 2019b.Coherent Comment Generation for Chinese Articles with aGraph-to-Sequence Model. In Proceedings of the 57th Con-ference of the Association for Computational Linguistics,4843–4852.Li, Q.; Han, Z.; and Wu, X.-M. 2018. Deeper Insights intoGraph Convolutional Networks for Semi-supervised Learn-ing. In Thirty-Second AAAI Conference on Artificial Intelli-gence.Maehara, T. 2019. Revisiting graph neural networks: All wehave is low-pass filters. arXiv preprint arXiv:1905.09550.McAuley, J.; Targett, C.; Shi, Q.; and Van Den Hengel, A.2015. Image-based Recommendations on Styles and Substi-tutes. In Proceedings of the 38th International ACM SIGIRConference on Research and Development in InformationRetrieval, 43–52. ACM.Morris, C.; Ritzert, M.; Fey, M.; Hamilton, W. L.; Lenssen,J. E.; Rattan, G.; and Grohe, M. 2019. Weisfeiler and Lemango Neural: Higher-order Graph Neural Networks. In Pro-ceedings of the AAAI Conference on Artificial Intelligence,volume 33, 4602–4609.

Pareja, A.; Domeniconi, G.; Chen, J.; Ma, T.; Suzumura,T.; Kanezashi, H.; Kaler, T.; and Leisersen, C. E. 2019.Evolvegcn: Evolving graph convolutional networks for dy-namic graphs. arXiv preprint arXiv:1902.10191.Rong, Y.; Huang, W.; Xu, T.; and Huang, J. 2019. The trulydeep graph convolutional networks for node classification.arXiv preprint arXiv:1907.10903.Sen, P.; Namata, G.; Bilgic, M.; Getoor, L.; Galligher, B.;and Eliassi-Rad, T. 2008. Collective classification in net-work data. AI magazine 29(3):93–93.Shchur, O.; Mumme, M.; Bojchevski, A.; and Gunnemann,S. 2018. Pitfalls of Graph Neural Network Evaluation. arXivpreprint arXiv:1811.05868.Sun, K.; Koniusz, P.; and Wang, J. 2019. Fisher-Bures Ad-versary Graph Convolutional Networks. In Proceedings ofthe Thirty-Fifth Conference on Uncertainty in Artificial In-telligence, 161.Velickovic, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio,P.; and Bengio, Y. 2018. Graph Attention Networks. In6th International Conference on Learning Representations,ICLR 2018.Verma, N.; Boyer, E.; and Verbeek, J. 2018. Feastnet:Feature-steered Graph Convolutions for 3d Shape Analysis.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, 2598–2606.Xu, K.; Li, C.; Tian, Y.; Sonobe, T.; Kawarabayashi, K.; andJegelka, S. 2018. Representation learning on graphs withjumping knowledge networks. In Proceedings of the 35th In-ternational Conference on Machine Learning, ICML 2018,5449–5458.Yang, L.; Chen, Z.; Gu, J.; and Guo, Y. 2019a. DualSelf-Paced Graph Convolutional Network: Towards Reduc-ing Attribute Distortions Induced by Topology. In Proceed-ings of the Twenty-Eighth International Joint Conference onArtificial Intelligence, IJCAI 2019, 4062–4069.Yang, L.; Kang, Z.; Cao, X.; Jin, D.; Yang, B.; and Guo,Y. 2019b. Topology Optimization based Graph Convolu-tional Network. In Proceedings of the Twenty-Eighth Inter-national Joint Conference on Artificial Intelligence, IJCAI2019, 4054–4061.Yang, Z.; Cohen, W. W.; and Salakhutdinov, R. 2016. Re-visiting Semi-supervised Learning with Graph Embeddings.In Proceedings of the 33nd International Conference on Ma-chine Learning,ICML 2016, 40–48.Zhou, J.; Cui, G.; Zhang, Z.; Yang, C.; Liu, Z.; and Sun, M.2018. Graph Neural Networks: A Review of Methods andApplications. arXiv preprint arXiv:1812.08434.Zitnik, M., and Leskovec, J. 2017. Predicting MulticellularFunction through Multi-layer Tissue Networks. Bioinfor-matics 33(14):i190–i198.

A Experimental SettingsIn this section, we will introduce the graph task datasets andthe baseline GNN models considered in this work first. Andthen we will introduce the experiment details and notationsused in this paper.

Class Feature Node EdgeCORA 7 1,433 2,708 5,278CiteSeer 6 3,703 4,732 13,703PubMed 3 500 44,338 500Coauthor CS 15 6,805 18,333 81,894Coauthor Phy. 5 8,415 34,493 247,962Amazon Comp. 10 767 13,381 245,778Amazon Photo 8 745 7,487 119,043

Table 4: Statistical information about datasets.

Experiment DatasetWe conduct experiment on 7 node classification datasets in3 types:

• Citation Network The CORA, CiteSeer, PubMeddatasets (Sen et al. 2008) are citation networks which areusually taken as be benchmarks of graph related stud-ies (Li, Han, and Wu 2018; Maehara 2019). The nodes arepapers in computer science field with bag-of-word fea-tures of paper title. The edges represent the citation rela-tion amaong papers and the label is paper category.

• Coauthor Network Coauthor CS and Coauthor Physicsare co-authorship graphs based on the Microsoft Aca-demic Graph from the KDD Cup 2016 challenge 3.5 Here,nodes are authors, that are connected by an edge if theyco-authored a paper; node features represent paper key-words for each authors papers, and class labels indicatemost active fields of study for each author.

• Amazon Network Amazon Computers and AmazonPhoto (McAuley et al. 2015) are segments of the Amazonco-purchase graph, where nodes represent goods, edgesindicate that two goods are frequently bought together,node features are bag-of-words encoded product reviews,and class labels are given by the product category.

Datasets related statistical information are shown in the Ta-ble 4. All these graphs are undirected graphs with no edgeweight.

Experimental DetailsPrevious works (Velickovic et al. 2018; Bai, Zhang, and Torr2019; Li et al. 2019b) on GNN study usually run experi-ment multi-turns to eliminate random effects. And Shchuret al. (2018) pointed out that in the semi-supervised nodeclassification task, the train/valid/test split of dataset has asignificant influence on the final result. Following Shchuret al.; Sun, Koniusz, and Wang (2018; 2019) we apply the20/30/rest splitting method, which means we randomly sam-ple 20 nodes in each category for training set and 30 forvalidation set; all the rest nodes are as test data. In orderto ensure the credibility of the results, we select 5 randomtrain/valid/test split of dataset and run 10 turns with differ-ent random seeds in each split. Then we measure the averagenumber of all the 50 turns’ results.

Besides, in order to avoid the random effects caused bydataset split or initial seeds and observe the influence of theproposed methods more clearly, we use the same random

5https://kddcup2016.azurewebsites.net

dataset split and initial seed list for the baseline method andthe proposed method in each controlled experiment, and thedataset split and seed list was randomly generated beforeeach controlled experiment. We also fix all the other hyper-parameters in each controlled experiment.

Notations Used in This Paper

Given a undirected graph G = (V ,E), where the V is thenode set with |V | = n and E is the edge set. The adjacencymatrix is denoted by A = [aij ] ∈ Rn×n. The raw nodefeatures are denoted by X = [x1,x2, . . . ,xn]> ∈ Rn×c

and each xi ∈ Rc is the input feature vector for the i-thnode. Each node has a corresponding label l indicating thenode class. In the semi-supervised node classification task,the gold labels of nodes in the trainset are known and thetarget is to predict the labels for nodes in the testset. We useGNNk to represent a k-layer (propagate step) graph neuralnetwork and the label predicted by this GNN is representedby l. Besides, the hidden states of all node after j layer(s) isdenoted by Hj ∈ Rn×hj (hj represents the hidden-size ofGNN j-th layer) and hji denotes the hidden state vector ofthe i-th node representation after j layers.

B MAD Global Values on More Datasets

In Figure 9, we display the MAD values of various GNNswith different layers on CiteSeer and PubMed dataset. Wecan observe that in these two datasets, the MAD values of allbaseline GNN models decreases with the increase of modellayer.

C Infomation-to-noise Ratio Experiment onMore Datasets

In Figure 7, we display the model performance and theMADGap value of node sets with different infomation-to-noise ratio at 2-order neighbours in CiteSeer and PubMeddataset. We can find that in these four GNN models, themodel performance and the MADGap value rises with theincrease of the intra-class node ratio, which will bringmore useful information for the graph nodes. So it is theinformation-to-noise ratio that largely affects the node repre-sentation over-smoothness thus has a influence on the modelperformance.

D AdaEdge Algorithm

The details of AdaEdge Algorithm is displayed in Algo-rithm 1. Besides, we use several heuristic approaches toadjust graph topology more wisely: (1) Operation edge fornodes with high prediction confidence (the max value aftersoftmax operation); (2) Operation edge for nodes belong toclasses with high prediction precision; (3) Skip operation bya certain probability to control the sparsity of the added andremoved edges; (4) Operation edge for nodes with certaindegrees.

CORA CiteSeer PubMedLAYER Metric GCN +MADReg GCN +MADReg GCN +MADRegLayer1 acc(%) 54.6±14 54.9∗±13 39.7±12 41.6∗∗±12 52.8±20 57.1∗∗±16

MADGap 0.13±0.04 0.14∗±0.04 0.07±0.04 0.10∗∗±0.03 0.33±0.20 0.44∗∗±0.21

Layer2 acc(%) 80.8±0.8 80.6±0.88 68.7±0.96 67.1∗∗±1.7 76.6±0.66 75.2∗∗±1.2

MADGap 0.59±0.03 0.66∗∗±0.04 0.51±0.08 0.53∗±0.07 0.74±0.03 0.73±0.05

Layer3 acc(%) 79.0±1.2 79.9∗∗±1.1 65.9±1.6 65.4∗±2.2 76.2±0.65 76.5∗±0.64

MADGap 0.57±0.03 0.59∗±0.04 0.52±0.04 0.53∗±0.03 0.70±0.04 0.74∗∗±0.07

Layer4 acc(%) 70.8±12 72.1∗∗±8.2 55.6±11 62.2∗∗±1.9 67.3±11 69.4∗∗±9.9

MADGap 0.52±0.05 0.53∗±0.04 0.47±0.07 0.54∗∗±0.04 0.63±0.22 0.71∗∗±0.13

Layer5 acc(%) 46.9±9.7 48.0∗∗±11 35.8±10 46.5∗∗±7.7 55.1±9.4 56.1∗∗±9.6

MADGap 0.24±0.07 0.28∗±0.04 0.331±0.18 0.46∗∗±0.07 0.45±0.22 0.51∗∗±0.24

Layer6 acc(%) 36.2±9.1 38.5∗∗±8.2 28.9±9.0 37.5∗∗±4.7 49.5±9.1 52.3∗∗±10.5

MADGap -0.22±0.20 −0.20∗∗±0.16 0.18±0.23 0.36∗∗±0.07 0.12±0.25 0.18∗±0.24

Table 5: Results of MADReg method for GCN with different layers on CORA/CiteSeer/PubMed dataset. Darker color meanslarger improvement over the baseline.

E Supplementary Result of RelievingOver-smoothing

In Figure 8, we display more models results of relievingover-smoothing in high layer GNNs.

F Error Analysis of MADRegIn Table 5, we show the results for GCN with differentnumber of layers on the CORA/CiteSeer/PubMed datasetsand we observe that the MADReg method can relieve theover-smoothing problem and improve model performanceeffectively especially for GNNs with more layers wherethe over-smoothing issue is more severe. In the shallowGCN, our MADReg method can not effectively improve themodel performance because the model has already been welltrained and the over-smoothing issue is not serious.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

0.2

0.4

0.6

0.8

valu

e

model = GCN

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GAT

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GraphSAGE

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = ChebGCN

typeAccMADGap

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

0.2

0.4

0.6

0.8

valu

e

model = GCN

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GAT

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = GraphSAGE

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0range

model = ChebGCN

typeAccMADGap

Figure 7: Infomation-to-noise Ratio Experiment on CiteSeer (top plot) and PubMed (bottom plot) datasets.

Figure 8: More Results on the CORA/CiteSeer/PubMed datasets. The number of model layer is 4, where the over-smoothingissue is serious. The box graph shows the mean value and the standard deviation of the prediction accuracy and the MADGapvalues. And we can find that the two proposed methods can effectively relieve the over-smoothing issue and improve modelperformance in most cases.

2 3 4 5 6#Model Layer

ARMA

ChebGCN

DNA

FeaSt

GAT

GCN

GGNN

GraphSAGE

HighOrder

HyperGraph

Mod

el

0.521 0.751 0.321 0.205 0.164

0.539 0.676 0.228 0.015 0.007

0.679 0.266 0.182 0.140 0.107

0.830 0.726 0.626 0.119 0.084

0.791 0.700 0.427 0.049 0.004

0.856 0.787 0.772 0.604 0.419

0.720 0.198 0.101 0.084 0.021

0.913 0.817 0.770 0.046 0.010

0.678 0.148 0.007 0.023 0.044

0.818 0.722 0.396 0.118 0.0410.2

0.0

0.2

0.4

0.6

0.8

1.0

2 3 4 5 6#Model Layer

ARMA

ChebGCN

DNA

FeaSt

GAT

GCN

GGNN

GraphSAGE

HighOrder

HyperGraph

Mod

el

0.473 0.723 0.693 0.100 0.007

0.859 0.767 0.291 0.015 0.002

0.481 0.595 0.255 0.118 0.141

0.896 0.848 0.523 0.416 0.045

0.849 0.749 0.168 0.021 0.011

0.890 0.830 0.756 0.561 0.275

0.608 0.251 0.060 0.041 0.044

0.914 0.764 0.712 0.273 0.000

0.844 0.040 0.032 0.003 0.003

0.914 0.798 0.238 0.018 0.0090.2

0.0

0.2

0.4

0.6

0.8

1.0

Figure 9: MAD results on CiteSeer (top plot) and PubMed(bottom plot) datasets.

Algorithm 1 AdaEdgeRequire: The GNN model GNNk, the feature matrix X , the

raw adjacency matrix A, the node size N ,the operation orderorder, the limit number num+ and num−, the limit confi-dence conf+ and conf− and max training times maxt.

1: function ADDEDGE(A,pred, conf )2: add count← 03: A′ ← A4: for node1 ni ∈ [0, N) do5: for node2 nj ∈ [ni, N) do6: if Aij==0 and pred[ni]==pred[nj] and

conf [ni]≥conf+ and conf [nj]≥conf+ then7: A′ij ← 1, A′ji ← 18: add count← add count + 19: if add count ≥ num+ then

10: reutrnA′

11: end if12: end if13: end for14: end for15: reutrnA′

16: end function17:18: function REMOVEDGE(A,pred, conf )19: rmv count← 020: A′ ← A21: for edge (ni, nj) ∈A do22: if Aij==1 and pred[ni]!=pred[nj] and

conf [ni]≥conf− and conf [nj]≥conf− then23: A′ij ← 0, A′ji ← 024: rmv count← rmv count + 125: if rmv count ≥ num− then26: reutrnA′

27: end if28: end if29: end for30: reutrnA′

31: end function32:33: function ADJUSTGRAPH(pred, conf )34: if order==’add first’ then35: A′ ← ADDEDGE(A,pred, conf)36: A′ ← REMOVEEDGE(A′,pred, conf)37: else38: A′ ← REMOVEEDGE(A,pred, conf)39: A′ ← ADDEDGE(A′,pred, conf)40: end if41: reutrnA′

42: end function43:44: AdaEdge45: acc0,pred0, conf0 ← GNN0

k(A,X)46: A′0 ← ADJUSTGRAPH(pred0, conf0)47: for iter times i ∈ [1,maxt) do48: acci,predi, confi ← GNNi

k(A′i−1,X)

49: if acci ≤ acci−1 then50: reutrn acci−1

51: end if52: A′i ← ADJUSTGRAPH(predi, confi)53: end for54: reutrn accmaxt−1