Embed Size (px)
Citation preview
Efficient Identification of Overlapping Communities
Jeffrey BaumesMark Goldberg
Malik Magdon-Ismail
Rensselaer Polytechnic Institute, Troy, NY
Outline
• Communities as clusters • What is a cluster? • Cluster seed procedure (LA) • Cluster refinement procedure (IS2) • Experimental results • Conclusions and future work
Communities as clusters
• Malicious groups use large communication networks for planning and coordination
• Their goal: remain undetected• Our goal: sift through
communications for suspicious patterns, using structure only, not content
Communities as clusters
• Detecting all social groups (malicious or not) will aide in searching for “hidden” groups
• Social groups tend to communicate densely
• Approach: Find social groups by finding clusters in the graph of the communication network
actor Aactor B
A communicates with Blikely a social group
likely not a social group
Add external edges
What is a cluster?
• Many partitioning algorithms exist• Social groups often overlap• Instead define clusters as locally
optimal with respect to density
partitioning overlapping clustering
Two-stage process
seed procedure
refinement procedure
communication network
seed clusters
final clusters
Original procedures
Rank Removal(RaRe)
Iterative Scan(IS)
communication network
seed clusters
final clusters
Jeffrey Baumes, Mark Goldberg, Mukkai Krishnamoorthy, Malik Magdon-Ismail,
Nathan Preston. "Finding Communities by Clustering a Graph into
Overlapping Subgraphs", International Conference on Applied Computing (IADIS
2005), Feb 22-25, Algarve, Portugal.
Proposed new procedures
Link Aggregate(LA)
Iterative Scan 2(IS2)
communication network
seed clusters
final clusters
Link Aggregate (LA)
• Order the nodes (two routines are used)
• Pass through the nodes– For each node, add it to the clusters it
improves, or start a new cluster
LA procedure
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
LA procedure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 17
18
1920
21
22
2324
25
26
27
28
29
30
31
32
33
34
35
Iterative Scan (IS)
• Old refinement procedure– Traverses entire node list, adding /
removing nodes which increase the density
– Repeats the process until no improvements are possible
• May be inefficient in sparse networks\
• Guaranteed to be locally optimal
Iterative Scan 2 (IS2)
• New refinement procedure– Traverses neighborhood of cluster
only, adding / removing nodes which increase the density
– Repeats the process until no improvements are possible
• More efficient in sparse networks in spite of overhead, less efficient in dense networks
IS2 procedure
IS2 procedure
IS2 procedure
IS2 procedure
IS2 procedure
Experimental results
• Compare run time of new vs. old• Compare cluster quality of new vs.
old• Compare on different network types
– Random– Preferential attachment– Real-world
• Compare possible actor orderings for LA
RaRe vs. LA run time
New RaRe
LA
Original RaReNew RaRe
LA
IS vs. IS2 run time
Define IS* = IS for dense graphs, IS2 for sparse graphs
Old vs. new quality
New RaRe → IS
LA → IS2
New RaRe → IS
LA → IS2
Preferential attachment
New RaRe → IS
LA → IS2
New RaRe → IS
LA → IS2
Real-World Networks
Ratio = new/old = (LA→IS*)/(RaRe→IS)
Quality Ratio
0
0.5
1
1.5
2
2.5
E-mail Web Newsgroup Fortune 500
Run-time Ratio
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
E-mail Web Newsgroup Fortune 500
IS2 IS IS2 IS2IS* =
LA ordering
Conclusions and future work
• Overlapping clustering may be used to discover social groups in communication networks
• The new algorithm is more efficient in many cases, while keeping the same or better quality
• A unified algorithm should choose strategies and parameters based on network properties
Questions
Rank Removal
• Existing seed procedure– Removes highly connected nodes until network is
broken into small clusters– Adds removed nodes back into clusters it is well-
connected to
• Two main inefficiencies– Computed Page Rank at each iteration– Computed connected components at each iteration
• Page Rank could be computed once, but reprocessing connected components is crucial
LA procedure detail
IS2 procedure detail
RaRe vs. LA
RaRe vs. LA
RaRe vs. LA
IS vs. IS2
IS vs. IS2
IS vs. IS2
Run time RaRe vs. LA
Run time IS vs. IS2
Cluster quality
Cluster quality
Preferential attachment run time
Preferential attachment quality
LA ordering run time
LA ordering quality
