GeoEvent & GeoAnalytics:Leveraging the spatiotemporal big data store
Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri
@amollenkopf
Suzanne FossReal-Time & Big Data Product Engineer, Esri
@sfoss_esri
1
2
3
4
5
6
Agenda
why a new type of data store?
visualizing observation data
writing observation data from GeoEvent Server
on-the-fly aggregations
writing analytic results from GeoAnalytics Server
deeper dive
why a new type of data store?1
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
thousands3K e/s
3K e/s 200 e/s
BYO-RDBMS
X
X
200 e/s
GeoEvent
Server
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
GeoEvent
Server
thousands3K e/s
3K e/s 200 e/s
relational
data store
X
X
200 e/s
Web GISwith real-time & big data capabilities
Web GIS
spatiotemporal
big data store
ingest visualize
analyze store
real-time & big data
GeoEvent
Server
millions4K e/s
4K e/s 10Ks e/s
4K e/s
Web GISwith real-time & big data capabilities
Web GIS
spatiotemporal
big data store
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
GeoEvent
Server
4K e/s
4K e/s
4K e/s
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
GeoAnalytics
Server
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
1
2 43
MINIMUM environment
functional servers & spatiotemporal big data store
SHOULD BE on ISOLATED machines!!!
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
1
2 3 64 75 8
RECOMMENDED environment
functional servers & spatiotemporal big data store
SHOULD BE on ISOLATED machines!!!
ArcGIS Enterprisewith real-time & big data capabilities
DesktopWeb Device
visualization
live & historic
aggregates & features
map & feature service
• Ingest high velocity real-time
data into ArcGIS
• Perform continuous analytics
on events as they are received
• Store observations in a
spatiotemporal big data store
• Run batch analytics on
stored observations
• Visualize high velocity &
volume data:
- as an aggregation
- as discrete features
• Notify those who need to
know about patterns of
interest
stream service
live features
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
storage
ingestion
analytics
GeoAnalytics
Server
analytics
visualizing observation data2
Observation datatypes of observation data
movingsomething
that moves
• planes
• vehicles
• animals
• satellites
• storms
stationarystands still but
attributes change
• water gauges
• weather stations
• traffic sensors
• air quality
discretesomething that
“just happens”
• crimes
• lightning
• accidents
Observation datadefined
• An observation is a recording of a feature’s attribute values and location at a specific
moment in time.
- Observations are immutable, they happen and are typically not edited.
- Observations can be replayed over space & time.
- Moving observations are identifiable by a unique attribute, known as a TRACK_ID.
space & time
AM
Observations can be
shown at a specific time
PM
Moving observations can
be identified by a unique attribute
tracks
1
1
1
2
22
3
3
3
the tracks of moving observations
Moving observation illustration for two tracks
over space (X, Y) and time (T)
can be reconstructed
T
On-the-fly aggregationsof observations
• The spatiotemporal big data store enables you to aggregate data on-the-fly
- aggregating on-the-fly accommodates for real-time data
- you can perform exploratory analysis via any combination of space, time and attribute queries
- toggle from aggregation to raw feature view
- access feature level attributes in aggregate or feature level view
Demo
map & feature service
visualizing observation data
writing observation data
from GeoEvent Server3
Writing observation datafrom GeoEvent Server
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
IoT
map & feature service
data source
input output
GeoEvent Service
Demo
into a spatiotemporal big data store
from GeoEvent Server
Writing observation data
on-the-fly aggregations4
Preparing data for on-the-fly aggregationsspatial indexing: geohash
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for geohash aggregation is continuously updated
geohash aggregation (based on a geohash index)
Preparing data for on-the-fly aggregationsspatial indexing: geohash & square
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for geohash aggregation is continuously updated
- a spatial index for square aggregation is continuously updated
square aggregation (based on a square index)geohash aggregation (based on a geohash index)
Preparing data for on-the-fly aggregationsspatial indexing: triangle
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for ‘pointy’ triangle aggregation is continuously updated
- a spatial index for ‘flat’ triangle aggregation is continuously updated
flat triangle aggregation (based on a flat triangle index)pointy triangle aggregation (based on a pointy triangle index)
Preparing data for on-the-fly aggregationsspatial indexing: hexagon (same as triangle)
• as data is written to a data source in the spatiotemporal big data store:
- a spatial index for ‘pointy’ hexagon (pointy triangle) aggregation is continuously updated
- a spatial index for ‘flat’ hexagon (flat triangle) aggregation is continuously updated
flat hexagon aggregation (based on a flat triangle index)pointy hexagon aggregation (based on a pointy triangle index)
Preparing data for on-the-fly aggregationsspatial indexing
• as data is written to a data source in the spatiotemporal big data store:
- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle
- this is in addition to a temporal index on the time field
- and an inverted index on each of the attribute fields
geohash
square
trianglepointy
triangleflat
hexagonflat
hexagonpointy
Visualizing on-the-fly aggregationsgeohash
geohash aggregation response geohash aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
map servicefeature service
lodType=geohash&lod=2
feature servicelodType=geohash&lod=2&returnGeometry=false
geohash aggregation response
Visualizing on-the-fly aggregationssquare
map servicefeature service
lodType=square&lod=2
square aggregation response square aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
Visualizing on-the-fly aggregationshexagon aggregation
map service
flat hexagon aggregation response
feature servicelodType=flatHexagon&lod=2
flat hexagon aggregation response
• map & feature services can query a data source in the spatiotemporal big data store with results
aggregated on-the-fly.
Visualizing on-the-fly aggregationsspatial projection
• some of the spatial indices support projection(s):
- the geohash spatial index only supports GCS WGS 1984, however you can project on-the-fly
- square, pointy & flat hexagon/triangle spatial indices are defined with a spatial reference
geohash aggregation in world mollweide projection (wkid = 54009) square aggregation in world mollweide projection (wkid = 54009)
Demo
geohash, square & hexagon
spatial projection
Visualizing on-the-fly aggregations
http://github.com/esri/aggregation-viewer-server-map-service http://github.com/esri/aggregation-viewer-client-feature-layer
writing analytic results
from GeoAnalytics Server5
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big Data
GeoAnalytics
Server
Writing analytic resultsfrom GeoAnalytics Server
IoT
GeoAnalytics Serveranalytic capabilities
Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksCreate Panel
Find LocationsFind Existing LocationsFind Similar Locations
Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube
Use ProximityCreate Buffers
Manage DataCopy to Data StoreJoin Features
* time aware tools
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
IoT
GeoAnalytics
Server
Writing analytic resultsfrom GeoAnalytics Server
• GeoAnalytics Server has the ability to:
- perform analytics against data sources in the spatiotemporal big data store
- and write the analytic results to the spatiotemporal big data store
Web GIS layers
new Web GIS layers
use your Web GIS layers through Pro, Portal,Python notebooks or the REST API
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
GeoAnalytics
Server
big datafile shares
featureservice
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
Writing analytic resultsfrom GeoAnalytics Server
• GeoAnalytics Server has the ability to:
- perform analytics against files in an external big data file share
- and write the analytic results to the spatiotemporal big data store
new Web GIS layers
Web GIS layers
Loading datavia ‘Copy to Data Store’ GeoAnalytics tool
GeoAnalytics tool interface
output features
big datafile shares
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
featureservice
relationaldata store
spatiotemporalbig data store
Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool
GeoAnalytics tool interface
big datafile shares
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
featureservice
relationaldata store
spatiotemporalbig data store
Writing analytic resultsvia ‘Reconstruct Tracks’ GeoAnalytics tool
output features
input features GeoAnalytics tool interface
deeper dive6- operating system requirements
- resource requirements
- data reliability
- ObjectId
Operating system requirements64-bit Windows or Linux
resource minimum RECOMMENDED notes
64-bit
operating system
(Windows)
Windows Server:
2012 R2 Standard & Datacenter (Sep 2016)
2012 Standard & Datacenter (Sep 2016)
2008 R2 Standard, Enterprise & Datacenter (sp1)
2008 Standard, Enterprise & Datacenter (sp2)
Windows Server:
2012 R2 Standard & Datacenter (Sep 2016)
‘_’ characters are
not allowed in
machine names
64-bit
operating system
(Linux)
Red Hat Enterprise Linux Server 7 (update 8)
Red Hat Enterprise Linux Server 6 (update 2)
CentOS Linux 7 (7.2)
CentOS Linux 6 (6.8)
Scientific Linux 7 (7.2)
Scientific Linux 6 (6.8)
Ubuntu Server LTS (16.04.1)
SUSE Linux Enterprise Server 12 (sp1)
SUSE Linux Enterprise Server 11 (sp4)
Oracle Linux 7 (update 2)
Oracle Linux 6 (update 8)
Use your Linux flavor of choice with the
latest update version Esri has certified.
http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
Resource requirementsprocessors
http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
resource minimum RECOMMENDED notes
processors 8 cores 16 cores
More processors results in
faster writes (indexing time),
higher write throughput &
faster query response time.
Resource requirementsprocessors, memory
http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
resource minimum RECOMMENDED notes
processors 8 cores 16 cores
More processors results in
faster writes (indexing time),
higher write throughput &
faster query response time.
storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.
SSD results in much faster writes & queries.
Resource requirementsprocessors, memory, storage
http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
resource minimum RECOMMENDED notes
processors 8 cores 16 cores
More processors results in
faster writes (indexing time),
higher write throughput &
faster query response time.
storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.
SSD results in much faster writes & queries.
memory 16 GB 32 GBMore memory results in
higher likelihood of query cache hits.
32 GB is the maximum due to JVM limitations.
Resource requirementsprocessors, memory, storage & instances
resource minimum RECOMMENDED notes
processors 8 cores 16 cores
More processors results in
faster writes (indexing time),
higher write throughput &
faster query response time.
storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.
SSD results in much faster writes & queries.
memory 16 GB 32 GBMore memory results in
higher likelihood of query cache hits.
32 GB is the maximum due to JVM limitations
# of
ISOLATED
instances
1 3 or more
With one instance there is risk of data loss.
With three instances replicas are written
increasing data reliability.
The # of instances will vary based on the velocity of
incoming data as well as the data retention policy
configured on each data source.
http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm
http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm
spatiotemporal
big data store
GeoEvent
Server
Data reliabilityshards & replication factor
GeoAnalytics
Server
node 1
node 2
node 3node 4
node 5
T1
T3
T2
T2
r = 1
T1
T3
spatiotemporal
big data store
Data reliabilityauto-rebalancing of data upon node membership changes, + or -, in the big data store
GeoEvent
Server
GeoAnalytics
Server
node 2
node 3node 4
node 5
T1
T3
T2
T2
r = 1
x
T1
T3
T1
T1T3
purge based on
data retention
spatiotemporal
big data store
GeoEvent
Server
Data reliabilitydata retention policies, configured per data source
GeoAnalytics
Server
node 1
node 2
node 3node 4
node 5
r = 1
purge based on
data retention
spatiotemporal
big data store
GeoEvent
Server
Data reliabilityrolling index option, set appropriately to the velocity of your observation data
GeoAnalytics
Server
node 1
node 2
node 3node 4
node 5
r = 1
indices indices
indices indices
indices
purge based on
data retention
spatiotemporal
big data store
GeoEvent
Server
Data reliabilityautomatic data backups using periodic snapshots, including ability to restore from a snapshot
GeoAnalytics
Server
node 1
node 2
node 3node 4
node 5
r = 1
snapshot-2016-05-17-11-0-0.snapshot
snapshot-2016-05-17-12-0-0.snapshot
…
ObjectIdchoosing an Object Id option
ObjectIdchoosing an Object Id option
ObjectIdchoosing an Object Id option
Max Value # of IDs ArcGIS Clients
Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
events per day Int32
1,000 e/s 86,400,000 25 days
10,000 e/s 864,000,000 2.5 days
100,000 e/s 8,640,000,000 6 hours
1,000,000 e/s 86,400,000,000 36 minutes
10,000,000 e/s 864,000,000,000 4 minutes
ObjectIdchoosing an Object Id option
Max Value # of IDs ArcGIS Clients
Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps
events per day Int32
1,000 e/s 86,400,000 25 days
10,000 e/s 864,000,000 2.5 days
100,000 e/s 8,640,000,000 6 hours
1,000,000 e/s 86,400,000,000 36 minutes
10,000,000 e/s 864,000,000,000 4 minutes
ObjectIdchoosing an Object Id option
Max Value # of IDs ArcGIS Clients
Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps
events per day Int32 Int64 (signed)
1,000 e/s 86,400,000 25 days 292,472,000 years
10,000 e/s 864,000,000 2.5 days 29,247,200 years
100,000 e/s 8,640,000,000 6 hours 2,924,720 years
1,000,000 e/s 86,400,000,000 36 minutes 292,472 years
10,000,000 e/s 864,000,000,000 4 minutes 29,248 years
ObjectIdchoosing an Object Id option
Max Value # of IDs ArcGIS Clients
Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …
Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps
UniqueStringID n/a unlimited JavaScript, custom apps
events per day Int32 Int64 (signed)
1,000 e/s 86,400,000 25 days 292,472,000 years
10,000 e/s 864,000,000 2.5 days 29,247,200 years
100,000 e/s 8,640,000,000 6 hours 2,924,720 years
1,000,000 e/s 86,400,000,000 36 minutes 292,472 years
10,000,000 e/s 864,000,000,000 4 minutes 29,248 years
Summary
http://links.esri.com/geoevent-sbds
spatiotemporal big data store
• The spatiotemporal big data store enables:
- GeoEvent Server to write high velocity & volume observation data
- On-the-fly aggregations to be visualized using Map & Feature Services
- GeoAnalytics Server to read & write high volume analytic results
• To learn more:
- See the ‘Spatiotemporal Big Data Store’ tutorial
Real-Time & Big Data GIS
• Applying Real-Time Analytics Mon, 1:45pm-2:45pm, Room 147B
Tue, 11:00am-12:00pm, Room 151B
• Leveraging the Spatiotemporal Big Data Store Mon, 3:00pm-4:00pm, Room 147B
Tue, 1:30pm-2:30pm, Room 151B
• ArcGIS GeoAnalytics: Intro to Big Data Feature Analytics Mon, 4:15pm-5:15pm, Room 147B
Tue, 2:45pm-3:45pm, Room 147A
• Leveraging Stream Services Tue, 2:45pm-3:45pm, Room 151A
• ArcGIS GeoEvent Server: Best Practices Tue, 4:00pm-5:00pm, Room 151A
• Real-Time & Big Data GIS at a Massive Scale Tue, 5:15pm-6:15pm, Room 151A
other sessions
Please Take Our Survey on the Esri Events App!
Select the session
you attended
Scroll down to find
the survey
Complete Answers
and Select “Submit”
Download the Esri Events
app and find your event
Print Your Certificate of Attendance
Print stations located in the 140 Concourse
Monday
12:30 PM – 6:30 PM
GIS Solutions Expo,
Hall B
5:15 PM – 6:30 PM
Expo Social,
Hall B
Tuesday
10:45 AM – 5:15 PM
GIS Solutions Expo,
Hall B
6:30 PM – 9:30 PM
Networking Reception,
Smithsonian National Air
and Space Museum
Questions / Feedback?
http://links.esri.com/geoevent
http://links.esri.com/geoevent-forum
To learn more:
Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri
@amollenkopf
Suzanne FossReal-Time & Big Data Product Engineer, Esri
@sfoss_esri