Upload
phungdang
View
225
Download
2
Embed Size (px)
Citation preview
Cloud-based Geospatial Data services and
analysis Xuezhi Wang
Scientific Data Center
Computer Network Information Center Chinese Academy of Sciences
2014-08-25
Outlines
1 Introduction of Geospatial Data Cloud
2 Cloud-based Geospatial Data Analysis platform
3 Cloud-based Geospatial Data visualization platform
Geospatial Data Cloud
I. Platform overviews II. Geospatial data services III. Geospatial model services IV. Services achievements
Why do we do?
In 2008, USGS opened all geospatial datasets to the public for free.
• Landsat, MODIS is the most popular remote sensing dataset for scientists and researchers. Especially, the Landsat series.
• Due to the low international network speed, many Chinese scientists cannot take advantage of these resources easily.
• Based on the CNIC’s mass storage and high speed network( CSTNET: Chinese Science and Technology Network), we mirrored the geospatial datasets covered China and surrounding area, and provided online data services for
chinese scientists and researchers
Design a new cloud-based geospatial analysis and visualization platform
USGS opened its Landsat datasets. we contacted the USGS staffs through email.
We got an oral permit for Landsat redistribution from USGS. we download the Landsat datasets, and provide public
services for Chinese scientists
Cooperated with National Center for Atmospheric Research (NCAR)
Mirrored the NCAR datasets provide public services for Chinese
Scientists
We got a permit for SRTM dataset redistribution from International Centre for Tropical Agriculture(CIAT). We cache
SRTM global elevation data, and provide online data services.
Accroding to the user requirements, We make some remote sensing products from Landsat dataset and Modis dataset.
such as vegetation index, water cover dataset.
Dowload MODIS dataset covered CN Host the first user conference
Design a new cloud-based data platform
Geospatial data cloud opened for public services Host the second user conference with ESRI China
Platform History
Platform objectives Based on our cloud infrastructure, we hope to construct an 'one-stop'
geospatial data service platform for Chinese scientists and researchers.
An open cloud-based platform meeting the requirements of data search, data acquisition, online storage, scientific models, online calculation and map visualization.
Model
Online Calculation
Search Acquisition
Storage Visualization
http://www.gscloud.cn
General framework
Database cluster Online storage Near-line
storage
Data integration
Automatic Metadata crawling
Automatic data
entities cache
Mass Storage
Cloud environments
Distributed file system
Virtual Machines
Task scheduling
Data query
Data category
Integrated search
Complex search
One-key indexes
Data services
Smart data Order
Data application Favorite data Batch
download
Model services
Model package
Online execution
Web map services
Map publish Map visualization
Map style edit
Model management
User data spaces
Data management
Task management
Map management
Model flow
Map share
Service management
System managem
ent Users
Data
Models
Indexes
Tasks
Services
Resources monitor
Geospatial Data Cloud
I. Platform overviews II. Geospatial data services III. Geospatial model services IV. Services achievements
Online data resources Open geospatial datasets: LANDSAT、MODIS、EO-1、DEM、NCAR、
NOAA and LUCC
More than 350TB data entities, and over 7 million data records.
Covered China and surrounding area
Online services for Chinese scientists and researchers with free of charge
Landsat1-3 MSS Landsat 4-5 MSS Landsat 4-5 TM Landsat 7 SLC-on Landsat7 SLC-off Landsat 8 OLI
TERRA AQUA
TERRA AQUA
GDEM 30m SRTM 90m Ocean elevation
HYP_L1R HYP_L1G
AL1
LANDSAT MODIS MODIS_L1B DEM EO-1
Generated geospatial products Water-cover of China from Landsat TM/ETM in Summer
Slope, Aspect, Topographic position indexes et. of China
Land Surface Temperature, Vegetation indexes (5 days, 15 days, and 30 days) of China, from 2000 – now
ASPECT Water Cover LST
Geospatial data query
Massive geospatial metadata
I need Landsat data in
summer, 2000
I need DEM data of Beijing
Input: geo-location or address
PostgreSQL+PostGIS Cluster
Based on geospatial
storage cluster and search
engine, we provide three
ways for users to access
their preferred data:
1: by address name
2: by geo-location
3: by administrative unit
Geospatial Data Cloud
I. Platform overviews II. Geospatial data services III. Geospatial model services IV. Services achievements
Geospatial model services Why do we need this services?
For example: If we need Beijing DEM data, there have two ways. Traditional way: search DEM data, download DEM data,
mosaic DEM data with ArcGIS, and then clip DEM data with Beijing shape mask .
Convenience way: select Beijing administrative unit in the web, submit a clip task to system, and then download Beijing DEM data.
Which is the best way? I prefer the second way.
That’s why we provide online geospatial model services.
We developed an elastic online task processing system with cloud techniques.
Users fill model parameters on the web page and submit tasks to the system;
The tasks were run under cloud virtual machines;
The results were stored in cloud storage.
Task scheduling
Task
Task
Task
Computing resource pool Storage pool
VM VM
VM
virtual machines in Cloud
Overall frameworks
Model page Model page
Online models and tools DEM tools:
DEM clip and mosaic; Reprojection; Aspect, Slope, Topographical index calculation of
specified region LANDSAT tools:
Online gap-fill model for Landsat ETM7 SLC-off dataset
MODIS tools: Online Land Surface Temperature calculation of user
specified time range Online vegetation indexes calculation of user specified
time range
Private cloud storage We provide user with private cloud storage to store their own model inputs and outputs data.
User manage their own data here, including upload, download, rename, delete, create folder and move
Platform Screenshots Model parameters page of DEM clip and mosaic tool
User only need to fill task name, select a region in the web map, and choose output folder
Platform Screenshots Task tracking page list user’s submitted tasks
They can tracking task states, stop and start task, adjust task priority here.
Platform Screenshots Task result page
When the task was finished, the results can be download from here, or from user private cloud storage
Geospatial Data Cloud
I. Platform overviews II. Geospatial data services III. Geospatial model services IV. Services achievements
Our platform was wildly used in many research fields in China: earth science, environmental science, ecology, hydrology, meteorology, oceanography, natural disasters, city planning ...
Service target
Based on statistics in November 2013
User characteristics
Currently, over 70,000 users
Doctor
Master Bachelor
Others
University
CAS
Government
Company
Others
Service statistics
Avg. downloaded over: 200 GB/day; online downloaded over: 250 TB;
offline copied over: 137 TB;
Outlines
1 Introduction of Geospatial Data Cloud
2 Cloud-based Geospatial Data Analysis platform
3 Cloud-based Geospatial Data visualization platform
Cloud-based Geospatial Data Analysis Platform
Why do we need this platform? For example: If we want to calculate Chinese monthly
temperature variation from 2000 to 2014, how can I achieve this?
Traditional way: download all data, and calculate with workstations. It may spent several months to download data. This is a very bad thing.
Shortcuts: using cloud computing technology. Apply virtual machines in the cloud; Write code in the virtual machines; read all data from cloud storage; Run program in parallel
Which way is the best? I prefer the second way. That’s why we want to construct this platform.
Overall frameworks
...... Storage resource pool
Cloud manager with OpenStack
Distributed file system
Data pool
Computing resource pool
Public geospatial data
User private geospatial data
virtual machine pool
VM VM VM VM
Currently progress of the platform We have finished the prototype system. The system is
under inner tests for access pressure and stability. In the end of this year, it will be provided for the public access.
Outlines
1 Introduction of Geospatial Data Cloud
2 Cloud-based Geospatial Data Analysis platform
3 Cloud-based Geospatial Data visualization platform
Cloud-based Geospatial Data Visualization Platform
Why do we need this platform? In many cases, users want to publish their geospatial data,
model results, vector map to the web site; They also want to customize the map layer styles through
the web pages; They also want to customize the components and styles of
the webgis pages; To achieve this goal, we construct this platform.
Optimized Web Map Service
Computing resource pool Storage pool
VM VM
VM
Dynamical render map in Cloud HTML5 map client ...
Tiles Storage
Dynamical cache
Map styles editor
Map request
Overall frameworks
Currently progress of the platform Most of system functions were already finished,
under inner tests and bug fixed.
Published services provided:
OGC WMS(web map service), Google TMS , OGC WFS (web feature service)
GeoJson , WorldWind ...