Upload
lydien
View
251
Download
1
Embed Size (px)
Citation preview
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 1
SAFARIArchitecture and Software Stack
Alberto Garcia-Robledo, Abel Sanchez, Rongsha Li, Juan-Carlos Murillo-Torres, John Williams and Sascha Boheme
Massachusetts Institute of TechnologyMIT Geospatial Data Center
z
CSVReaderPandas
REST serverFlask
RepositoryMongoDB
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
Fuzzy MatchingDetectorsJellyfish
Fuzzy MatchingDetectorsJellyfish
Geolocation Matching
Detectors*PyGeo
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
Exact MatchingDetectorsMongoDB
AggregationFramework
Fuzzy MatchingDetectorsJellyfish
SessionMongoDB
SessionMongoDB
SessionMongoDB
TaskQueueCelery
REST serverFlask
FlagsMongoDB
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
RFNetIntegrators
igraph
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
NetworkCentralityRankers
igraph
BBN Rankers
ebay-bayesian
TaskQueueCelery
REST serverFlask
RFNetsMongoDB
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
TreemapFormatter
igraph
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
LinkAnomalyDetectorNetworkXPy2neo
LinkAnomalyDetectorsNetworkXPy2neo
RFNetFormatter
igraph
MapFormatter
igraph
TaskQueueCelery
REST serverFlask
... ... ...
XLSReader*Pandas
RDBReader*Pandas
...Data
Integrator*
CSVFile
XLSDoc.
RDB
Treemap ViewJIT InfoVis
Network ViewJIT InfoVis
Map ViewLeaflet
...GUI
JQuery EasyUI
Data Layer(DAL)Python
Flag Layer(FAL)
Python
Integration Layer & Rank Layer(NIL & RAL)
Python
View Layer(VAL)Python
Web Layer(WEL)
JavaScript
SAFARI Software Architecture
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 2
* = to be developed
MongoDBhttps://www.mongodb.org/
MongoDB is an open-source NoSQL document database.
JSON-style documents with dynamic schemas.
Rich, document-based queries.
Flexible aggregation and MapReduce data processing.
RepositoryMongoDB
RepositoryMongoDB
SessionMongoDB
SessionMongoDB
Who's using it:
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 3
Flaskhttp://flask.pocoo.org/
Lightweight Web application framework for Python
Microframework: it keeps the core simple but extensible.
RESTful request dispatching.
Extensions available to enhance features as desired.
REST serverFlask
REST serverFlask
Who's using it:
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 4
Pandashttp://pandas.pydata.org/
Easy-to-use data structures and data analysis tools.
Efficient DataFrame object for data manipulation.
R/W between in-memory data and text files, CSV,
Microsoft Excel, SQL DBs, and HDF5.
High performance merging and joining of data sets.
RDBReader*Pandas
RDBReader*Pandas
XLSReader*Pandas
CSVReaderPandas
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 5
Celeryhttp://www.celeryproject.org/
Asynchronous task queue based on distributed message passing.
Tasks are executed concurrently on a single or more worker servers.
Support for RabitMQ, Redis, Beanstalk and MongoDB brokers.
TaskQueueCelery
TaskQueueCelery
Who's using it:
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 6
igraphhttp://igraph.sourceforge.net/
High performance library for complex network research and SNA.
Algorithms for measuring structural properties, node centrality, K-
decomposition and community detection.
Algorithms for generating 2D/3D layouts: Fruchterman-Reingold,
Kamada-Kawai, Reingold-Tilford, Distributed Recursive Layout, etc.
RFNetIntegrator
igraph
RFNetFormatters
igraph
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 7
eBay Bayesian Belief Networkshttps://github.com/eBay/bayesian-belief-networks
BBNRankers
igraph
Who's using it:
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 8
Pythonic Bayesian Belief Network package.
Exact inference on BBNs specified as pure python functions.
Discrete and continuous variables.
Different inference engines: junction tree, sum product, etc.
JS InfoVis Toolkithttp://philogb.github.io/jit/
Tools for creating Interactive Data Visualizations for the Web.
Based on the HTML5 canvas.
Graph, radial and hierarchical network visualizations.
Treemap, stacked sunburst, area, bar and pie charts.
Treemap ViewJS InfoVis
Treemap ViewJS InfoVis
Network ViewJS InfoVis
Network ViewJS InfoVis
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 9
jQuery EasyUIhttp://www.jeasyui.com/
Collection of user-interface plugin based on HTML5 and jQuery.
Essential functionality for building modern, interactive, javascript
applications.
Datagrid, treegrid, panel, combo and more for building cross-browser
web page.
GUIjQuery EasyUI
GUIjQuery EasyUI
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 10
SAFARI: Web-Based Visual Analytics
WebGUI
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 11
Conclusions
• Analysis integration. Enable SMEs to integrate different analysis techniques for processing large amounts of payment documents.
• Big data analysis. Help SMEs to make sense of a large amount of RFs spread across data.
• Focus. Help SMEs to focus on the most suspicious payments by exploiting modern high-performance multi-core computers and visualization techniques.
False positive minimization
Novelty:
IntegrationRanking
Visualization
MIT Geospatial Data Center SAFARI: Situational Awareness Framework for Risk Ranking 12