Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
GISolve GISolve A GridA Grid--based problem solving environment for based problem solving environment for computationally intensive geographic information computationally intensive geographic information analysisanalysis
Shaowen WangShaowen Wang
Department of Geography & Department of Geography & Grid Research & educatiOn group @ IoWa (GROW)Grid Research & educatiOn group @ IoWa (GROW)
ITS Academic Technologies ITS Academic Technologies –– Research Services Research Services The University of IowaThe University of Iowa
Iowa City, IA 52242, USAIowa City, IA 52242, USA
July 24, 2005July 24, 2005
22
CollaboratorsCollaborators
Marc P. Armstrong, Professor, Department of Marc P. Armstrong, Professor, Department of Geography and Program in Applied Geography and Program in Applied Mathematical and Computational SciencesMathematical and Computational SciencesJun Ni, Ph.D., Associate Research Scientist, Jun Ni, Ph.D., Associate Research Scientist, ITS Academic Technologies ITS Academic Technologies –– Research Research ServicesServicesYan Liu, Graduate Research Assistant, ITS Yan Liu, Graduate Research Assistant, ITS Academic Technologies Academic Technologies –– Research Services Research Services
33
PurposePurpose
To address why the Grid is important to To address why the Grid is important to geographic information analysis geographic information analysis researchresearchTo demonstrate the To demonstrate the designdesign and and implementationimplementation of GISolve of GISolve –– a Grida Grid--based problem solving environment for based problem solving environment for computationally intensive geographic computationally intensive geographic information analysisinformation analysis
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
44
BackgroundBackground
The amount of geographic informationThe amount of geographic information–– Ever increasingEver increasing
Application drivenApplication driven–– GPS, LBS, RSGPS, LBS, RS
Computationally intensive geographic analysisComputationally intensive geographic analysis–– Heuristic searchHeuristic search–– SimulationSimulation–– OptimizationOptimization–– Statistical methods Statistical methods
Grid computingGrid computing–– CyberinfrastructureCyberinfrastructure–– Open Grid Service Architecture (OGSA)Open Grid Service Architecture (OGSA)
Web servicesWeb servicesProblem solving environmentsProblem solving environments–– Grid portalsGrid portals
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
55
GIScience Grid PortalGIScience Grid Portal
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
66
Grid Resources Grid Resources –– A User ViewA User View
R
GIPPL
MA
PeerDB
R: Resource (Running Sensor), e.g., GangliaGIP: Grid Information Provider, e.g., MDSDB: Database (MYSQL in this case)MA: Monitoring AgentPL: Peer Location ComponentVO: Virtual Organization
PeerDB
Peer
Peer
DB
DB
PeerDB
VO1
VO2
User Agent
Peer communicationServing User Requests
Fully ConnectedVO Networks
View 1View 2
User Agent
View 1
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
77
Grid ComplexityGrid Complexity
Grid middleware limitationGrid middleware limitation–– Provides a selected set of protocols and servicesProvides a selected set of protocols and services
Not directly userNot directly user-- or applicationor application-- orientedoriented
HeterogeneousHeterogeneousDynamicDynamicAdministrated in different security domainsAdministrated in different security domainsFaults Faults –– Usually unpredictableUsually unpredictable
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
88
GISolve Architecture
Computational domain modeling
Domain decomposition
Task scheduling
Information broker and resource
discoveryData access module
Problem solving environments implemented using Grid portal technologies
Grid information services
Protocols and services for data access on the Grid,
such as the Globus GridFTP
Grid Middleware such as Globus, Legion, and Condor
Resource allocation
99
GISolve ServicesGISolve Services
SecurityDecomposition and task schedulingGeographic data accessGeographic data accessResource information brokering Resource information brokering
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1010
Classification of GISolve Classification of GISolve ServicesServices
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1111
GISolve WorkflowGISolve Workflow
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
1212
A ThreeA Three--Layer Model of Layer Model of Grid Portal Technologies Grid Portal Technologies
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1313
Technology SpecificationTechnology Specification
JetspeedJetspeed portal serverportal server–– JetspeedJetspeed: portal engine : portal engine –– Turbine: MVC (ModelTurbine: MVC (Model--ViewView--Control) frameworkControl) framework–– Velocity: dynamic html generationVelocity: dynamic html generation–– Tomcat: portal containerTomcat: portal container
MultiMulti--user supportuser support–– User registration and managementUser registration and management–– UserUser--based state managementbased state management
Configurable portal interfaceConfigurable portal interface–– Customized layoutCustomized layout–– Display control of each Display control of each portletportlet interfaceinterface
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1414
APIs ImplementedAPIs Implemented
APIs developedAPIs developed–– [[org.gisolve.globusorg.gisolve.globus]: Globus Toolkit enhancement]: Globus Toolkit enhancement–– [[edu.uiowa.gisolve.ogsaedu.uiowa.gisolve.ogsa]: Grid service (interface ]: Grid service (interface
definition, userdefinition, user--level Grid service implementation)level Grid service implementation)–– [[org.apache.jetspeed.modules.actions.gisolveportlorg.apache.jetspeed.modules.actions.gisolveportl
etsets]: ]: JetspeedJetspeed portletportlet action implementationaction implementation–– [[gisolveportletsgisolveportlets]: portal support for Grid service ]: portal support for Grid service
APIs (state management, Grid service client APIs)APIs (state management, Grid service client APIs)APIs integratedAPIs integrated–– Java COG (GT2), GT3/GT4 Core, OGSA (WS Java COG (GT2), GT3/GT4 Core, OGSA (WS
GRAM, RFT), GRAM, RFT), MyProxyMyProxy, LDAP, JNI, JPL, LDAP, JNI, JPL
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1515
Defining GISolve ServicesDefining GISolve Services
••<definitions name="<definitions name="TSServiceTSService targetNamespacetargetNamespace==http://uiowa.edu/grow/namespaces/2004/04/gisolve/TShttp://uiowa.edu/grow/namespaces/2004/04/gisolve/TS …… />/>••<!<!---- import external types import external types ---->>••<import location="../<import location="../ogsi/ogsi.gwsdlogsi/ogsi.gwsdl““ ……/>/>••<!<!---- selfself--defined types defined types ---->>••<import location="<import location="gidTypes.xsdgidTypes.xsd““ namespace="http://uiowa.edu/grow/namespaces/2004/04/gisolve/gidnamespace="http://uiowa.edu/grow/namespaces/2004/04/gisolve/gidtypes"/>types"/>••<!<!---- local types schema local types schema ---->>••<types><<types><xsd:schemaxsd:schema ……/></types>/></types>••<!<!---- message definition message definition ---->>••<message name="<message name="SchedInputMessageSchedInputMessage">">……</message></message>••<message name="<message name="SchedOutputMessageSchedOutputMessage">">……</message></message>••<!<!---- service operation definition service operation definition ---->>••<<gwsdl:portTypegwsdl:portType name="name="TSPortTypeTSPortType" extends="" extends="ogsi:GridServiceogsi:GridService">">
••<operation name="schedule"><operation name="schedule">••<input message="<input message="tns:SchedInputMessagetns:SchedInputMessage"/>"/>••<output message="<output message="tns:SchedOutputMessagetns:SchedOutputMessage"/>"/>••<fault name="Fault" message="<fault name="Fault" message="ogsi:FaultMessageogsi:FaultMessage"/>"/>
••</operation> </operation> ••<<sd:serviceDatasd:serviceData name="name="TSStateTSState““ …… />/>
••</</gwsdl:portTypegwsdl:portType>>
A task scheduling service A task scheduling service gWSDLgWSDL::Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
1616
OGSI to WSRFOGSI to WSRFWSDL
<?xml version="1.0" encoding="UTF-8"?><definitions name="DCService" ... ...
xmlns:wsrp="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd"xmlns:wsrpw="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.wsdl"xmlns:wsdlpp="http://www.globus.org/namespaces/2004/10/WSDLPreprocessor" ><wsdl:import namespace="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties....wsdl"
location="../wsrf/properties/WS-ResourceProperties.wsdl"/><types>... ...<xsd:element name="DCResourceProperties">... ...</xsd:element></types><!-- WSRF Grid service PortType definition --><portType name="DCPortType" wsdlpp:extends="wsrpw:GetResourceProperty"
wsrp:ResourceProperties="tns:DCResourceProperties"><operation name="decompose">... ...</operation>
</portType>
gWSDL<?xml version="1.0" encoding="UTF-8"?><definitions name="DCService“ … …
xmlns:ogsi="http://www.gridforum.org/namespaces/2003/03/OGSI"xmlns:gwsdl="http://www.gridforum.org/namespaces/2003/03/gridWSDLExtensions"xmlns:sd="http://www.gridforum.org/namespaces/2003/03/serviceData"><!-- import ogsi Grid Service gwsdl. Location is under gt3/schema/ogsi/ --><import location="../ogsi/ogsi.gwsdl" namespace="http://www.gridforum.org/namespaces/2003/03/OGSI"/>... ...<!-- Grid service interface definition --><gwsdl:portType name="DCPortType" extends="ogsi:GridService">
<operation name="decompose">… …</operation> <sd:serviceData name="DCState" type="xsd:int" … … />
</gwsdl:portType>
1717
GISolve Services in ActionGISolve Services in Action
Deployed GISolve services:Deployed GISolve services:Purpose
Background
[globus4@rsgridportal globus4]$ globus-start-containerStarting SOAP server at: https://128.255.162.167:8443/wsrf/services/With the following services:... ...[12]: https://128.255.162.167:8443/wsrf/services/gisolve/TSService... ...[47]: https://128.255.162.167:8443/wsrf/services/gisolve/DCService
Design
Implementation
Case Study
Evaluation
Conclusions
1818
Case Study Case Study –– GGii*(d) *(d) StatisticStatistic
Purpose
)(* dGi = 2/12**1
*
)}1/(]){[(
)(
−−
−∑nWnSs
xWxdw
ii
j ijijBackground
Design
Implementation
Case Study ReferencesReferences–– GetisGetis, A., and , A., and OrdOrd, J.K., 1992, The analysis of spatial , J.K., 1992, The analysis of spatial
association by use of distance statistics. association by use of distance statistics. Geographical Geographical AnalysisAnalysis, , 2424(3): 189(3): 189--206.206.
–– OrdOrd, J. K., and , J. K., and GetisGetis, A., 1995, Local spatial autocorrelation , A., 1995, Local spatial autocorrelation statistics: distributional issues and an application. statistics: distributional issues and an application. Geographical AnalysisGeographical Analysis, , 2727(4): 286(4): 286--306.306.
Evaluation
Conclusions
1919
Synthetic DatasetsSynthetic Datasets
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
20000-point : two clusters, each of which has a normal distribution with a standard deviation of 2
20000-point a uniform random distribution
2020
User InterfaceUser Interface
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
2121
An Example of Spatial An Example of Spatial Computational Domain Computational Domain DecompositionDecomposition
PurposeY
Background
0, 0 1, 1 2, 4
4, 2 5, 3
11, 13
3, 5
6, 6 7, 7
8, 8 9, 9 10, 12
12, 10 13, 11 14, 14 15, 15Design
Implementation
Case Study
Evaluation
Conclusions
X
2222
A Task Scheduling ServiceA Task Scheduling Service
Decoupled from domain decomposition Decoupled from domain decomposition servicesservicesNPNP--complete problemcomplete problem–– MaxMax--min algorithm implementedmin algorithm implemented
Using the theoretical estimate to Using the theoretical estimate to compute the Expected Time to compute the Expected Time to Compute (ETC) matrixCompute (ETC) matrix
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
2323
A Diagrammatic Example A Diagrammatic Example of Static Task Schedulingof Static Task Scheduling
1, 10, 0 2, 4
4, 2 5, 3
11, 13
14, 14 15, 15
3, 5
6, 6 7, 7
8, 8 9, 9 10, 12
12, 10 13, 11
1, 10, 0 2, 4
4, 2 5, 3
11, 13
14, 14 15, 15
3, 5
6, 6 7, 7
8, 8 9, 9 10, 12
12, 10 13, 11
1, 10, 0 2, 4
4, 2 5, 3
11, 13
14, 14 15, 15
3, 5
6, 6 7, 7
8, 8 9, 9 10, 12
12, 10 13, 11
1, 10, 0 2, 4
4, 2 5, 3
11, 13
14, 14 15, 15
3, 5
6, 6 7, 7
8, 8 9, 9 10, 12
12, 10 13, 11
Resource1 Resource2 Resource3
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
2424
Grid Testbed Grid Testbed Implementation Implementation -- HawkGridHawkGrid
rtgrid1.its.uiowa.edu
The Open Science Grid
High energy physics VO
beowulf.stat.uiowa.edu rsgrid3.its.uiowa.edu
A server desktop at
ITS –rsgrid1
8.5Mbits/s
62.4Mbits/s
60.5Mbits/s
27.3Mbits/s 331Kbits/s 306Kbits/s
28.3Mbits/s
48.2Mbits/s73.7Mbits/s
ITS-Stat-Geo VO
Purpose
Background
Design
Implementation
Case Study
Evaluation
Conclusions
2525
Purpose
Background
DEMODesign
Implementation
Case Study
Evaluation
Conclusions
2626
Performance Performance
The problem cannot be solved by any The problem cannot be solved by any singlesingle computer in our Grid testbedcomputer in our Grid testbed–– 3.2G memory3.2G memory
Solved within 10 minutes through Solved within 10 minutes through GISolveGISolve
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
2727
ConclusionsConclusions
GISolve demonstrates how the Grid can GISolve demonstrates how the Grid can benefit research on computationally benefit research on computationally intensive geographic information intensive geographic information analysesanalysesGISolve integrates OGSAGISolve integrates OGSA--based Web based Web services to support the computational services to support the computational aspects of GIServicesaspects of GIServices
Purpose
Background
Design
Case Study
Implementation
Evaluation
Conclusions
2828
Ongoing Research Ongoing Research
Interoperability of GISolve servicesInteroperability of GISolve servicesAdaptive domain decomposition Adaptive domain decomposition servicesservicesEvaluation of GISolve performanceEvaluation of GISolve performanceExtension of the types of geographic Extension of the types of geographic information analyses GISolve supportsinformation analyses GISolve supports
2929
AcknowledgementAcknowledgement
This research is partially supported by This research is partially supported by the HawkGrid project funded by the the HawkGrid project funded by the Office of Vice President for Research at Office of Vice President for Research at The University of Iowa The University of Iowa Computational resources used for Computational resources used for experiments include a cluster supported experiments include a cluster supported by the U.S. NSF iVDGL (international by the U.S. NSF iVDGL (international Virtual Data Grid Laboratory) project Virtual Data Grid Laboratory) project