Upload
others
View
12
Download
0
Embed Size (px)
Citation preview
Unstructured Data Analysis in ArcGIS
James Jones - Esri
Julia Bell - Esri
Scott Graff - Microsoft
What is Unstructured Data?
• Does not have a recognizable structure or is loosely structured
• Can be in a variety of formats and storage mechanisms
• Word Documents
• Social Media Posts
• PowerPoint
• Share drive
“Every two days we create as muchinformation as we did up to 2003”
Eric Schmidt, 2010
What does that look like?Every minute…
Twitter sees new 350,000 tweets Facebook has 510,000 comments posted, 293,000 statuses updated
15.2 million Text Messages are sent
954,000 new Microsoft Office documents are created
144 million e-mails are sent
How much spatial information are we missing out on?
How much spatial information are we missing out on?
How can we capture this information in ArcGIS?
CoordinatesCustom Locations
User defined keywords
Native Esri
Capability
What are you looking for?
What is the best tool?
How is it best used?
• Data is at least somewhat understood• Data benefits from identifiable and
repeating patterns• Little to no programming experience
available/needed
How to Integrate Unstructured Data into ArcGIS
ArcGIS Pro w/ LocateXT
ArcGIS Enterprise w/ LocateXT
ArcGIS Pro for Intelligence
Extract Locations from Unstructured Data
ArcGIS LocateXT
Extracting Locations with ArcGIS
• LocateXT Extension for ArcGIS Desktop and Enterprise
• Available for ArcMap 9.1 and later
• Available in ArcGIS Pro at 2.3
• 100% Feature function as ArcGIS Pro 2.4
• Uses pattern matching regular expressions (REGEX) to search for coordinates in a variety of formats
• Uses custom location list to match/extract other patterns (place names, codes, other terms)
Extracting Locations in ArcGIS Pro
• New option added to the “Add Data” button
• Allows for a user to drag and drop documents or copied text into a window
• Can create a new feature class or append it to an existing one
• Included with ArcGIS Pro for Intelligence
Extracting Locations in ArcGIS Pro
• Two Geoprocessing Tools added
• Located in the Conversion Tools –> To Geodatabase
• Extract Locations from Document
• Extract Locations from Text
Extracting Custom Attributes
• Ability to create custom attributes based on content within document or near a location
• Triggered by location extraction
• Based on keywords
• Tag locations based on keywords
• Scrape/harvest portions of document based on keywords
• Ability to extract based off of:
• Number of characters/words
• Number of lines/blank line
• Stop string
• Previously built in separate LocateXT desktop application (until Pro 2.4)
Extracting Addresses
• Ability to extract addresses from documents based on combination of:
• State
• Zipcode
• Ex. VA 22182
• The combination of extracted text and pre-text is geocoded
Explore Unstructured Data through LocateXT and Custom Attributes
How to Integrate Unstructured Data into ArcGIS
CoordinatesCustom Locations
User defined keywords
LocationsPeople/Organizations
EventsDates
Relationships
Native Esri
Capability
What are you looking for?
What is the best tool?
How is it best used?
• Data is at least somewhat understood• Data benefits from identifiable and
repeating patterns• Little to no programming experience
available/needed
• Data is not well understood• Data does not contain identifiable
and/or repeating patterns• Integration needed
Third Party
Integration
Natural Language Processing
ArcGIS Pro w/ LocateXT
ArcGIS Enterprise w/ LocateXT
ArcGIS Pro for Intelligence
with ArcGIS
Integrating NLP
Integrating NLP Capabilities with ArcGIS
• Many NLP offerings have Python APIs/SDKs or communicate over REST
• Integrates near seamlessly with ArcPy
• Create Python Toolboxes/Script Tools
• Allows to extract relevant data based on data local to their machine or as part of Enterprise Pipeline
• ArcGIS.Learn has incorporated support for Entity Recognition
APIs
Apps
Desktop
ArcGIS
NLTK
NLP Tools
Processing Unstructured Data Using ArcGIS and Microsoft Azure
Source Processing Storage Analysis Apps/Visualization
Key Take-aways:
1. Leverages modern, serverless processes and integration apps
2. Allows for a variety of NLP processes to be ran
3. Deep analytics with ArcGIS and Azure Cognitive Services/Machine Learning
Building an Unstructured Pipeline to Understand World Events
Source Processing Storage Analysis Apps/Visualization
Logic Apps / Power Automate watch RSS feeds and Website.Microsoft Cognitive Services extracts entities and analyze
sentiment.
Data is passed to ArcGIS GeoEvent Server for ingestion
into ArcGIS Platform. JSON files are stored in Azure Storage. Tabular data is
stored in a Azure SQL Data Warehouse.
ArcGIS Pro for Intelligence allows non-GIS Intel
professionals access to a custom experience of ArcGIS Pro to provide deep analysis. Microsoft Cognitive Services
further enriches data by running computer vision
against embedded images.
Operations Dashboard and ArcGIS Insights allows for very tailored views of the data to be
quickly analyzed and viewed by decision makers and non-
GIS Professionals.
ArcGIS Pro for Intelligence
Multi-int workstation for the intelligence professional
• Create and manage intelligence information
• Visualize and display your data in maps, charts, and timelines
• Perform spatial, temporal, relational, and predictive analysis
• Produce and disseminate intelligence products
NLP Integration with ArcGIS
Download the Esri
Events app and find your event
Select the session
you attended
Scroll down to
“Survey”
Log in to access the
survey
Complete the survey
and select “Submit”
Please Share Your Feedback in the App