Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
Big Data: GeoEvent & GeoAnalytics, Leveraging the Spatiotemporal Big Data Store
Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri
@amollenkopf
Ricardo TrujilloReal-Time & Big Data GIS Developer, Esri
@rtrujill007
1
2
3
4
5
6
Agenda
why a new type of data store?
visualizing observation data
writing observation data from GeoEvent Server
on-the-fly aggregations
writing analytic results from GeoAnalytics Server
deeper dive
R&D projects sneak-peek7
why a new type of data store?1
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
thousands3K e/s
3K e/s 200 e/s
BYO-RDBMS
X
X
200 e/s
GeoEvent
Server
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
GeoEvent
Server
thousands3K e/s
3K e/s 200 e/s
relational
data store
X
X
200 e/s
Web GISwith real-time & big data capabilities
Web GIS
spatiotemporal
big data store
ingest visualize
analyze store
real-time & big data
GeoEvent
Server
millions4K e/s
4K e/s 10Ks e/s
4K e/s
Web GISwith real-time & big data capabilities
Web GIS
spatiotemporal
big data store
ingest visualize
analyze store
real-time & big data
millions4-12K e/s
4-12K e/s 10Ks e/s
GeoEvent
Server
4K e/s
4K e/s
4K e/s
bring-your ownmessage broker
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
GeoAnalytics
Server
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
1
2 43
MINIMUM environment
functional servers & spatiotemporal big data store
SHOULD BE on ISOLATED machines!!!
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
1
2 3 64 75 8
RECOMMENDED environment
functional servers & spatiotemporal big data store
SHOULD BE on ISOLATED machines!!!
ArcGIS Enterprisewith real-time & big data capabilities
DesktopWeb Device
visualization
live & historic
aggregates & features
map & feature service
• Ingest high velocity real-time
data into ArcGIS
• Perform continuous analytics
on events as they are received
• Store observations in a
spatiotemporal big data store
• Run batch analytics on
stored observations
• Visualize high velocity &
volume data:
- as an aggregation
- as discrete features
• Notify those who need to
know about patterns of
interest
stream service
live features
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
storage
ingestion
analytics
GeoAnalytics
Server
analytics
visualizing observation data2
Observation datatypes of observation data
movingsomething
that moves
• planes
• vehicles
• animals
• satellites
• storms
stationarystands still but
attributes change
• water gauges
• weather stations
• traffic sensors
• air quality
discretesomething that
“just happens”
• crimes
• lightning
• accidents
Observation datadefined
• An observation is a recording of a feature’s attribute values and location at a specific
moment in time.
- Observations are immutable, they happen and are typically not edited.
- Observations can be replayed over space & time.
- Moving observations are identifiable by a unique attribute, known as a TRACK_ID.
space & time
AM
Observations can be
shown at a specific time
PM
Moving observations can
be identified by a unique attribute
tracks
1
1
1
2
22
3
3
3
the tracks of moving observations
Moving observation illustration for two tracks
over space (X, Y) and time (T)
can be reconstructed
T
On-the-fly aggregationsof observations
• The spatiotemporal big data store enables you to aggregate data on-the-fly
- aggregating on-the-fly accommodates for real-time data
- you can perform exploratory analysis via any combination of space, time and attribute queries
- toggle from aggregation to raw feature view
- access feature level attributes in aggregate or feature level view
Demo
map & feature service
visualizing observation data
writing observation data
from GeoEvent Server3
Writing observation datafrom GeoEvent Server
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
IoT
map & feature service
data source
input output
GeoEvent Service
Demo
into a spatiotemporal big data store
from GeoEvent Server
Writing observation data
on-the-fly aggregations4
Preparing data for on-the-fly aggregationsspatial indexing: geohash
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for geohash aggregation is continuously updated
geohash aggregation (based on a geohash index)
Preparing data for on-the-fly aggregationsspatial indexing: geohash & square
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for geohash aggregation is continuously updated
- a spatial index for square aggregation is continuously updated
square aggregation (based on a square index)geohash aggregation (based on a geohash index)
Preparing data for on-the-fly aggregationsspatial indexing: triangle
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for ‘pointy’ triangle aggregation is continuously updated
- a spatial index for ‘flat’ triangle aggregation is continuously updated
flat triangle aggregation (based on a flat triangle index)pointy triangle aggregation (based on a pointy triangle index)
Preparing data for on-the-fly aggregationsspatial indexing: hexagon (same as triangle)
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for ‘pointy’ hexagon (pointy triangle) aggregation is continuously updated
- a spatial index for ‘flat’ hexagon (flat triangle) aggregation is continuously updated
flat hexagon aggregation (based on a flat triangle index)pointy hexagon aggregation (based on a pointy triangle index)
Preparing data for on-the-fly aggregationsspatial indexing
• as data is written to a data source in the spatiotemporal big data store:
- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle
- this is in addition to a temporal index on the time field
- and an inverted index on each of the attribute fields
geohash
square
trianglepointy
triangleflat
hexagonflat
hexagonpointy
Visualizing on-the-fly aggregationsgeohash
geohash aggregation response geohash aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
map servicefeature service
lodType=geohash&lod=2
feature servicelodType=geohash&lod=2&returnGeometry=false
geohash aggregation response
Visualizing on-the-fly aggregationssquare
map servicefeature service
lodType=square&lod=2
square aggregation response square aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
Visualizing on-the-fly aggregationshexagon aggregation
map service
flat hexagon aggregation response
feature servicelodType=flatHexagon&lod=2
flat hexagon aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
Visualizing on-the-fly aggregationsspatial projection
• some of the spatial indices support projection(s):
- the geohash spatial index only supports GCS WGS 1984, however you can project on-the-fly
- square, pointy & flat hexagon/triangle spatial indices are defined with a spatial reference
geohash aggregation in world mollweide projection (wkid = 54009) square aggregation in world mollweide projection (wkid = 54009)
Visualizing on-the-fly aggregationsdiscover which aggregations are enabled on a layer (data source)
Demo
geohash, square & hexagon
spatial projection
Visualizing on-the-fly aggregations
http://github.com/esri/aggregation-viewer-server-map-service http://github.com/esri/aggregation-viewer-client-feature-layer
writing analytic results
from GeoAnalytics Server5
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big Data
GeoAnalytics
Server
Writing analytic resultsfrom GeoAnalytics Server
IoT
GeoAnalytics Serveranalytic capabilities
Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksCreate Panel
Find LocationsFind Similar Locations
Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube
Use ProximityCreate Buffers
Manage DataCopy to Data StoreJoin Features
* time aware tools
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
IoT
GeoAnalytics
Server
Writing analytic resultsfrom GeoAnalytics Server
• GeoAnalytics Server has the ability to:
- perform analytics against data sources in the spatiotemporal big data store
- and write the analytic results to the spatiotemporal big data store
Web GIS layers
new Web GIS layers
use your Web GIS layers through Pro, Portal,Python notebooks or the REST API
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
GeoAnalytics
Server
big datafile shares
featureservice
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
Writing analytic resultsfrom GeoAnalytics Server
• GeoAnalytics Server has the ability to:
- perform analytics against files in an external big data file share
- and write the analytic results to the spatiotemporal big data store
new Web GIS layers
Web GIS layers
Loading datavia ‘Copy to Data Store’ GeoAnalytics tool
GeoAnalytics tool interface
output features
big datafile shares
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
featureservice
relationaldata store
spatiotemporalbig data store
Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool
GeoAnalytics tool interface
big datafile shares
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
featureservice
relationaldata store
spatiotemporalbig data store
Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool
output features
input features GeoAnalytics tool interface
R&D projects sneak-peek7
Polly Q*bert
project Pollyspatial indexing: polygon
• as data is written to the spatiotemporal big data store:
- a spatial index for polygon aggregation is continuously updated
polygonstate
polygoncounty
polygon aggregation (based on a county polygon index)polygon aggregation (based on a state polygon index)
projecton-the-fly & interactive space-time cubes
• leverage existing geohash, square & hexagon aggregation capabilities of
the spatiotemporal big data store to construct on-the-fly space-time cubes.
- enables exploratory & interactive analysis of real-time & historic observation data
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
ArcGIS API for
JavaScript
& WebGL
client-side rendering ofspace-time cubes via WebGL
lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour
on-the-fly & interactive space-time cubes feature service
space-time geohash (based on a geohash aggregation) response
project
feature servicelodType=geohash&lod=2
geohash aggregation response
lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour&f=json
also considering:f=protobuf
an array of time-slices of aggregated features
&returnGeometry=false &returnGeometry=false
Demo
on-the-fly & interactive space-time cubes
project
Q*bert
Summary
http://links.esri.com/geoevent-sbds
spatiotemporal big data store
• The spatiotemporal big data store enables:
- GeoEvent Server to write high velocity & volume observation data
- On-the-fly aggregations to be visualized using Map & Feature Services
- GeoAnalytics Server to read & write high volume analytic results
• To learn more:
- See the ‘Spatiotemporal Big Data Store’ tutorial
Real-Time & Big Data GIS
• Building Android Location Awareness with GeoEvent Server Thu, 6:00-6:30pm, Mesquite C
• GeoEvent Server: Making 3D Scenes Come Alive Fri, 8:30-9:30am, Primrose C-D
• GeoEvent Server: Internet of Things (IoT) Fri, 1:00-2:00pm, Primrose
other sessions
Questions / Feedback?
http://links.esri.com/geoevent
http://links.esri.com/geoevent-forum
To learn more:
Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri
@amollenkopf
Ricardo TrujilloReal-Time & Big Data GIS Developer, Esri
@rtrujill007