Upload
-
View
614
Download
1
Tags:
Embed Size (px)
DESCRIPTION
GraphX is the blue ocean for scala engineers @ Scala Matsuri 2014
Citation preview
GraphX is the blue ocean for Scala Engineers
Scala Matsuri 2014 LT
@teppei_tosahttps://www.flickr.com/photos/exalthim/337922734
@ t e p p e i _ t o s a F i n a n c e I T E n g i n e e r !A s a k u s a / H a d o o p / S c a l a / P l a y F r a m e w o r k / S p a r k / G r a p h X
Who am I ?
https://www.flickr.com/photos/exalthim/337922734
• One of Spark Components
• Graph-parallel computation system.
• Unify graph-parallel and data-parallel
computation in one system with a single
composable API.https://www.flickr.com/photos/exalthim/337922734
Example graph computation : Page Rank
0.33 0.33 0.33Set the values which are divided 1 with the number of vertex
0.170.17
0.33 0.33
Divide the values of each vertex with the number of degrees and send neighbors the values
0.17 0.50 0.33Summarize the values which are sent from neighbors and Set the summarized value
Until the values are converged, repeat these steps
https://www.flickr.com/photos/exalthim/337922734
Difficulty of graph-parallel computation
Because of connection between vertices, distributed computation of vertices needs to communicate between nodes ( Apache Giraph communicates by Zookeeper )
https://www.flickr.com/photos/exalthim/337922734
Unify graph-parallel and data-parallel computation
10
20
30
1
100
3
110
120200
2
[1,10,[2,100]][2,20,[3,110]][3,30,[1,200],[2,120]]
Apache GiraphI D VA L
1 1 0
2 2 0
3 3 0
S R C T G T VA L1 2 1 0 02 3 1 1 03 2 1 2 03 1 2 0 0
GraphX
val graph = Graph.fromEdgesgraph.joinVertices(…)
https://www.flickr.com/photos/exalthim/337922734
Graph data around you
Social Network Train Network Data Network
https://www.flickr.com/photos/exalthim/337922734
What you will be able to do with graph data
Eveluate Vertex Clustering Graph Shape
Flow on Graph Predict Link
GraphX is Still young
• Not enough information on web
• Much less functions than other graph lib like igraph of R
https://www.flickr.com/photos/exalthim/337922734
https://www.flickr.com/photos/katedot/8272997562
My work about GraphX
• Translated GraphX document in Japanese
• https://gist.github.com/ironpeace/9306874
• Graph utility
• https://github.com/ironpeace/graph-web
https://www.flickr.com/photos/exalthim/337922734
Advantage for Scala Engineers
• Handling graph data with API like Scala’s collection’s API
• Easy to implement recursive computation
• Easy to implement function to handle graph data in iteration
https://www.flickr.com/photos/exalthim/337922734
GraphX is the blue ocean for YOU !
• GraphX is the good solution for graph-parallel computation
• Handling Graph structure data gives you power to work out something which you have never been able to
• GraphX is still Young
• Scala engineers have advantage for graph data
https://www.flickr.com/photos/exalthim/337922734
Get the Graph Power!Thank you !
@teppei_tosahttps://www.flickr.com/photos/exalthim/337922734