Advances in Science Technology and Engineering Systems JournalVol 3 No 3 XX-YY (2018)

wwwastesjcomASTES JournalISSN 2415-6698

Virtual Watershed System A Web-Service-Based SoftwarePackage For Environmental Modeling

Rui Wu1 Connor Scully-Allison2 Moinul Hossain Rifat2 Jose Painumkal2 Sergiu Dascalu2 Frederick CHarris Jr2

1Department of Computer Science East Carolina University 27858 USA2Computer Science amp Engineering Department University of Nevada Reno 89557 USA

A R T I C L E I N F O A B S T R A C T

Article historyReceivedAcceptedOnline

KeywordsWeb-based Virtual WatershedSystemModel as serviceCloud-based applicationHydrologic model

The physically-based environmental model is a crucial tool used in manyscientific inquiries With physical modeling different models are used tosimulate real world phenomena and most environmental scientists usetheir own devices to execute the models A complex simulation can betime-consuming with limited computing power Also sharing a scientificmodel with other researchers can difficult which means the same modelis rebuilt multiple times for similar problems A web-service-based frame-work to expose models as services is proposed in this paper to addressthese problems The main functions of the framework include model exe-cutions in cloud environments NetCDF file format transmission modelresource management with various web services As proof of concepta prototype is introducedimplemented and compared against existingsimilar software packages Through a feature comparison with equiva-lent software we demonstrate that the Virtual Watershed System (VWS)provides superior customization through its APIs We also indicate thatthe VWS uniquely provides friendly usable UIs enabling researchers toexecute models

1 Introduction

Modeling has become an indispensable tool in grow-ing environmental scientistsrsquo understanding of hownatural systems react to changing conditions It shedslight on complex environmental mysteries and helpsresearchers in formulating policies and decisions onfuture scenarios Environmental modeling is highlychallenging as it involves complex mathematical com-putations rigorous data processing and convolutedcorrelations between numerous parameters Threecommonly-used and important scientific software qual-ity measures are maintainability quality and scalabil-ity Issues like data storage coupling models retrievaland running are hard problems and need to be ad-dressed with extra efforts from software engineeringperspective It is a challenging job to design integratedsystems that can address all these issues

It is essential to build high-quality software toolsand design efficient frameworks for scientific researchAbundant scientific model data are generated and col-

lected in recent years Software engineering can as-sist this emergence through the creation of distributedsoftware systems and frameworks enabling scientificcollaboration with previously disparate data and mod-els It is challenging to implement software tools forinterdisciplinary research because of the problem ofcommunication and team building among different sci-entific communities For example the same terminol-ogy can have different meanings in different domainsThis increases difficulties in comprehending softwarerequirements when a project involves stakeholders (egresearchers) from different fields

Most work described in this paper is for WatershedAnalysis Visualization and Exploration (WC-WAVE)which is a NSF EPSCoR-supported project and initi-ated by jurisdictions of EPSCoR of Nevada Idaho andNew Mexico The WC-WAVE project includes threeprincipal components watershed science data cyber-infrastructure and data visualization [1] The projectmain goal is to implement VWS with the collaborationsbetween cyberinfrastructure team members and hy-

Connor Scully-Allison 1664 N Virginia Street CSE (0171) Reno NV 89557 (775) 771-1469 amp cscully-allisonnevadaunredu

wwwastesjcom 1

drologists The platform is able to store share modeland visualize data on-demand through an integratedsystem These features are crucial for hydrologic re-search

Different hydrologic models such as ISNOBAL andPRMS are commonly used by WC-WAVE hydrologistsThese models are leveraged to predict or examine hy-drologic processes of Lehman Creek in Nevada DryCreek and Reynolds Creek in Idaho and Jemez Creekin New Mexico We propose a framework for repre-senting model data in a standardized format calledthe ldquoNetwork Common Data Formatrdquo (NetCDF) andexposing these hydrologic models through web ser-vices This framework is based on our previous workintroduced in [2] To improve on our prior work wehave implemented some new Docker APIs to controlsystem components wrapped in docker containers Tosimplify data extraction modification and storageNetCDF data format [3] is used in the system for grid-organized scientific data Most parts of the systemframework can be reused for data-intensive purposebecause it is designed with the blueprint and templateconcepts ISNOBAL and PRMS are physically-basedmodels which can produce very accurate results How-ever these two models require abundant computingpower To solve the challenge we leverage a clusterto execute models in parallel ISNOBAL and PRMSare used in this paper to demonstrate the ideas andfunctionality of the proposed framework Throughoutthe remainder of this paper we refer to this prototypesystem as the VWS

ISNOBAL is a grid-based DEM (Digital ElevationModel) and created to model the seasonal snow covermelting and development The model author is Markset al [4] and it is initially developed for Utah Califor-nia and Idaho mountain basins The model determinesrunoff and snowmelt based on terrain precipitationregion characteristics climate and snow properties[4]

PRMS is short for Precipitation-Runoff ModelingSystem and is initially written with FORTRAN in 1983PRMS is prevalent physical process based distributed-parameter hydrologic model and the main functionof a PRMS model is to evaluate a watershed responseto different climate and land usage cases [5 6 7] Itcomposed of algorithms describing various physicalprocesses as subroutines The model now in its fourthversion has become more mature over the years ofdevelopment Different hydrology applications suchas measurement of groundwater and surface water in-teraction the interaction of climate and atmospherewith surface water water and natural resource man-agement have been done with the PRMS model [5 67]

This paper is organized as follows in its remainingsections Section 2 introduces background and relatedwork Section 3 describes the system design Section 4describes the prototype system and how the softwarewas built using RESTful APIs Section 5 compares ourwork with related tools and Section 6 contains thepapers conclusions and outlines planned future work

2 Background and Related Work

rdquoHow to implement software for interdisciplinary re-searchrdquo is an interesting question and there existssome successful work on environments and frame-works which seek to answet this question In this sec-tion relevant popular earth science applications andframeworks are introduced

Community Surface Dynamics Modeling System(CSDMS) was a project started in 1999 to conduct expe-ditious research of earth surface modelers by creating acommunity driven software platform CSDMS appliesa component-based software engineering approach inthe integration of plug-and-play components as thedevelopment of complex scientific modeling systemrequires the coupling of multiple independently de-veloped models [8] CSDMS allows users to write theircomponents in any popular language Also they canuse components created by others in the communityfor their simulations CSDMS treats components aspre-compiled units which can be replaced added toor deleted from an application at runtime via dynamiclinking Many key requirements drove the design ofCSDMS including the support for multiple operat-ing systems language interoperability across both pro-cedural and object-oriented programming languagesplatform independent graphical user interfaces use ofestablished software standards interoperability withother coupling frameworks and use of HPC tools tointegrate parallel tools and models into the ecosystem

A leading hydrologic research organization isCUAHSI rdquoCUAHSIrdquo and acronym for rdquoConsortiumof Universities for the Advancement of Hydrologic Sci-ence Incrdquo represents universities and internationalwater science-related organizations One of mosthighly esteemed products is HydroShare which isa hydrologic data and model sharing web applica-tion Hydrologists can easily access different modeldatasets and share their own data Besides this thisplatform offers many distributed data analysis toolsA model instance can be deployed in a grid cloud orhigh-performance computing cluster with HydroShareAlso a hydrologist is able to publish outcomes of theirresearch such as a dataset or a model In this way sci-entists use the system as a collaboration platform forsharing information HydroShare exposes its function-ality with Application Programming Interfaces (APIs)which means its web application interface layer andservice layer are separated This enables interoperabil-ity with other systems and direct client access [9]

Model as a Service is proposed by Li et al [10] forGeoscience Modeling It is a cloud-based solution andLi et al has implemented a prototype system to exe-cute high CPU and memory usage models remotelyas a service with third party platform such as AWS(Amazon Web Service) and Microsoft Azure

The key idea of MaaS is that model executions canbe done through a web interface with user inputs Com-puter resources are provisioned with a cloud providersuch as Microsoft Azure The model registration isdone in the framework with a virtual machine image

wwwastesjcom 2

repository If a model is registered and placed in therepository it can be shared by other users and multiplemodel instances can be executed in parallel based ondemand

McGuire and Roberge designed a social network topromote collaboration between watershed scientistsDespite being highly available the collaboration be-tween the general public scientists and citizen has notbeen leveraged Also hydrologic data is not integratedin any system The main goal of this work is to designa collaborative social network for multiple watershedscientific and hydrologic user groups [11] Howevermore efforts need to be done

The Demeter Framework by Fritzinger et al [12]represents another attempt to utilize software frame-works as a scientific aid in the area of climate changeresearch A software framework named ldquoThe DemeterFrameworkrdquo is introduced in the paper and one of thekey ideas is a component-based approach to integratedifferent components into the system for the ldquomodelcoupling problemrdquo ldquoThe model coupling problemrdquorefers to using a modelrsquos outputs as another modelrsquosinputs to solve a problem

Walker and Chapra proposed a web-based client-server approach for solving the problem of environ-mental modeling compared to the traditional desktop-based approach The authors assert that with the im-provement in modern day web browsers client-sideapproaches offer improved user interfaces comparedto traditional desktop software In addition power-ful servers enable users to perform simulations andvisualizations within the browser [13]

The University of New Mexico has implementeda data engine named GSToRE (Geographic StorageTransformation and Retrieval Engine) The engine isdesigned for earth scientific research and the mainfunctions of the engine are data delivery documenta-tion and discovery It follows the combination of com-munity and open standards and implemented basedon service oriented architecture [14]

21 Service Oriented Architecture

Industry has shown more and more interests on ServiceOriented Architecture (SOA) to implement softwaresystems [15] The main idea of SOA is to have busi-ness logic decomposed into different units (or services)These units are self-contained and can be easily de-ployed with container techniques such as Docker

Representational State Transfer Protocol (REST) isprimarily an architectural style for distributed hyper-media systems introduced by Fielding Roy Thomas inhis PhD dissertation [16] REST defines a way for aclient-server architecture on how a client and a servershould interact by using a set of principles REST hasbeen adopted for building the main architecture ofthe proposed system Statelessness uniform Interfaceand cache are the main characteristics of a REST client-server architecture [16] It is for these characteristicsthat REST is leveraged in our proposed system

Statelessness Statelessness is the most importantproperty or constraint for a client-server architectureto be RESTful The communication between the clientand the server must be stateless which means theserver is not responsible for keeping the state of thecommunication It is the clientrsquos responsibility A re-quest from the client must contain all the necessaryinformation for the server to understand the request[16 17] Two subsequent requests to the server will nothave any interdependence between each other Intro-ducing this property on the client-server architecturepresents several benefits regarding visibility reliabilityand scalability [16 17] For example as the server isnot responsible for keeping the state and two subse-quent requests are not interrelated multiple serverscan be distributed across a load-balanced system wheredifferent servers can be responsible for responding todifferent requests by a client

Uniform Interface Another important property of aRESTful architecture is it provides a uniform interfacefor the client to interact with a server Instead of anapplicationrsquos particular implementation it forces thesystem to follow a standardized form For exampleHTTP 11 which is a RESTful protocol provides a setof verbs (eg GET POST PUT DELETE etc) for theclient to communicate with the server The verbs suchas ldquoGETrdquo and ldquoPOSTrdquo work as an interface makingthe client-server communication generic[16]

Cache REST architecture introduces cache constraintto improve network efficiency [16] A server can allowa client to reuse data by enabling explicitly for label-ing some data cacheable or non-cacheable A servercan serve data that will not change in the future ascached content allowing the client to eliminate partialinteraction with that data in a series of requests

22 REST Components

The main components of REST architecture includeresources representations and resource identifiers

Resources The resource is the main abstract repre-sentation of data in REST architecture [16] Any pieceof data in a server can be represented as a resource to aclient A document an image data on todayrsquos weathera social profile everything is considered a resource inthe server Formally a resource is a temporarily vary-ing function of MR(t) that maps to a set of entities fortime t [16]

Representations A resource is the abstract buildingblock of the data in a web server For a client to con-sume the resource it needs to be presented in a way theclient can understand This is called representationA representation is a presentation format for repre-senting the current state of a resource to a consumerSome commonly used resource representation formatin the current standard are HTML (Hypertext Markup

wwwastesjcom 3

Language) XML (Extensible Markup Language) JSON(JavaScript Object Notation) etc A server can exposedata content in different representations so that con-sumer can access the resources through resource iden-tifiers (discussed in Section 22) in the desired format

Resource Identifiers A resource is uniquely identi-fied through a resource identifier in a RESTful archi-tecture For example in HTTP Uniform Resource Iden-tifier (URI) is used to identify a resource in a server AURI can be thought as the address of a resource in theserver [18] A resource identifier is the key for a clientto access and manipulate a resource in the server

23 Microservice Architecture

Software as a Service (SaaS) as a new software deliveryarchitecture has emerged to leverage the widely-usedREST standard for processing in addition to data trans-fer The main advantage of SaaS is that it does notrequire local installation and this is a significant ITtrend based on industry analysis [19]

Similarly ldquomicroservicerdquo decomposes an applica-tion into small components (or services) and these com-ponents communicate with each other through APIsldquoMicroservicerdquo is a solution to monolithic architecturerelevant problems [20] Because of the ldquomicroservicerdquocharacteristics an application can be easily scaled andthe deployment risks have been reduced without inter-rupting other services

Traditional monolithic applications are built as asingle unit using a single language stack and oftencomposed of three parts a front end client a back-end database and an application server sitting in themiddle that contains the business logic Here the ap-plication server is a monolith that serves as a singleexecutable A monolithic application can be scaledhorizontally by replicating the application server be-hind a load balancer to serve the clients at scale Thebiggest issue with monolithic architecture is that asthe application grows the deployment cycle becomeslonger as a small change in the codebase requires thewhole monolith to be rebuilt and deployed [20] Itleads to higher risk for maintenance as the applicationgrows These pitfalls have lead to the idea of decom-posing the business capabilities of an application intoself-contained services

Being a relatively new idea researchers have at-tempted to formalize a definition and characteristicsof Microservices Lewis and Fowler [20] has put to-gether a few essential characteristics of a microservicearchitecture that are described in brief in the followingsections

Figure 1 High-level diagram of VWS clients and VWS commu-nicate through REST Web Services Possible clients include a scriptand applications VWP provide data service auth service and modelexecution service

Componentization via Services The most impor-tant characteristic of a microservice architecture is thatservice functionalities need to be componentized In-stead of thinking of components as libraries that usein-memory function calls for inter-component commu-nication we can think of components in terms out-of-process services that communicate over the networkquite often through a web service or a remote proce-dure call A service has to be atomic doing one thingand doing one thing well [21]

Organized Around Business Capabilities Mono-lithic applications are typically organized around tech-nology layers For example a typical multi-tiered ap-plication might be split logically into persistence layerapplication layer and UI layer and teams are also orga-nized around the technologies This logical separationcreates the need for inter-team communication evenfor a simple change In microservices the product isorganized around business capabilities where a serviceis concentrated on one single business need and ownedby a small team of cross-domain members

Decentralized Governance In a monolithic applica-tion the product is governed in a centralized mannermeaning it restricts the product to a specific platformor language stack But in microservices as each serviceis responsible implementing an independent businesscapability it allows for building different services withdifferent technologies As a result the team gets to

wwwastesjcom 4

choose the tools that are best suited for each of theservices

Newman in his book Building Microservices [21]has discussed several concrete benefits of using mi-croservices over a monolith Several important benefitsare discussed below in brief

Figure 2 System-level diagram of VWS two main system compo-nents are VW-MODEL and VW VW-MODEL contains componentsrelative to model execution and VW contains components relativeto the platform

Technology Heterogeneity When independent ser-vices are built separately it allows adoption of differ-ent technology stacks for different services This givesa team multiple advantages liberty to choose a tech-nology that best suits the need and adopt technologyquickly for the needs

Scaling Monolithic applications are hard to dealwith when it comes to scaling One big problem withmonolithic applications is that everything needs to bescaled together as a piece But microservice allows hav-ing control over the scaling application by allowing toscale the services independently

Ease of Deployment Applying changes for a mono-lithic application requires the whole application to bere-deployed even for a minor change in the codebaseIt poses a high risk as unsuccessful re-deploymentseven with minor changes can take down the softwareMicroservices on the other hand allow low-risk de-ployments without interrupting the rest of the servicesThey also allow having faster development processwith small and incremental re-deployments

3 Proposed Method

Here we introduce detailed design documentation forthe VMS We present the design using many commondiagrams used in software engineering including sys-tem diagrams and workflow diagrams The VWS aims

to create a software ecosystem named the Virtual Water-shed by integrating cyberinfrastructure and visualiza-tion tools to advance watershed science research Fig 1shows a general high-level diagram of componentsof the ecosystem The envisioned system is centeredaround services comprised of data modeling and vi-sualization components A high-level description ofeach depicted component is provided in Fig 1

31 Data Service and Modeling Service

To create a scalable and maintainable ecosystem ofservices a robust data backend is crucial The en-visioned data service exposes a RESTful web serviceto allow easy storage and management of watershedmodeling data It allows retrieval of data in variousOGC (Open Geospatial Consortium) standards likeWCS WMS to allow OGC compliant clients to retrievedata automatically This feature is very important fordata-intensive hydrologic research and is essential fora scientific modeling tool

Hydrologists often use different modeling tools tosimulate and investigate the change of different hydro-logic variables around watersheds The modeling toolsare often complex to setup in local environments andtakes up a good amount of time setting up [10] Besidesthese modeling tools may require high computationaland storage resources that make them hard to run inlocal environments The proposed modeling serviceaims to solve these issues by allowing users to submitmodel execution tasks through simple RESTful webservice API This approach of allowing model-runsthrough a generic API solves multiple problems

bull It allows the users to run models on demandwithout having to worry about setting up envi-ronments

bull It allows modelers to accelerate the process ofrunning models with different input parametersby submitting multiple model-runs to be runin parallel which might not be feasible in localenvironment due to lack of computational andstorage resources

bull It opens the door for other services and clientsto take advantage of the API to automate theprocess of running models in their workflow

The primary goal of this Virtual Watershed systemis to allow watershed modelers to share their data re-sources and execute relevant models through web ser-vices without having to install the models locally Thesystem is designed as a collection of RESTful microser-vices that communicate internally The web servicesits on top of an extensible backend that allows easyintegration of models and scalability over model runsand number of users connected to the service

The system provides a mechanism for integratingnew models by conforming with a simple event drivenarchitecture that allows registering a model in the sys-tem by wrapping it with a schema driven adaptor

wwwastesjcom 5

As model execution is a CPU intensive process scal-ing the server with growing number of parallel modelexecution is an important issue The proposed architec-ture aims to solve this problem by introducing a simpledatabase-oriented job queue that provides options foradding more machines as the system grows

32 System Level Design

Several different submodules comprise The VirtualWatershed system Each module provides differentfunctionalities to the entire framework This relation-ship between modules is described with Figure 2 Thefollowing paragraphs will explain each submodule indetail

VW-PY Using the VW-PY module users can de-fine adapters to configure compatibility between differ-ent models and the VWS An adaptor is python codewhich encapsulates a model and that allows for it to berun programmatically VW-PY handles many aspectsof model rdquowrappingrdquo with this python code throughan interface Through this API an user can definethe code which handles converting between data for-mats executing models and model-progress-basedevent triggering This event triggering system pro-vides model-wrapper developers with the tools to emitmodel execution progress

VW-MODEL The VW-MODEL submodulethrough a web service exposes a RESTful API tothe userclient Users can upload query and retrievemodel run packages using this API

VW-WORKER The VW-WORKER is a servicewhich encapsulates model adaptors in a worker ser-vice that is organized in a queue data structure Thiscomponent communicates with the VW-MODEL com-ponent using a redis data-backend

VW-STORAGE The VW-STORAGE module pro-vides an interface for object storage for the VW-MODEL component Developed as a generic wrappersysadmins can configure this interface to work withlocal or cloud-hosted storage providers

VW-AUTH The VW-AUTH module provides au-thentication services for usersclients connecting tothe VWS framework This service provides clientswith a JWT token enabling them to securely utilize theother services described in this section

VW-SESSION The VW-SESSION sub module coor-dinates different components in the VWS to provide acommon session backend for a single user Data abouteach session is maintained and managed with a Redisdata store shared between services

VW-WEB This is the web-application front-endwhich provides users with APIs to interact with themodel processing modules described above In a stan-dard use case of this module users will get access toa session after logging into this system with the VW-AUTH component Once given a session users can usethis interface to access resources run models trackprogress and uploaddownload model run resources

Figure 3 Workflow for accessing secure REST endpoint with JWTtoken

33 Detailed Design

The VWS is comprised of many distinct servicesand web applications which communicate with one-another to facilitate model runs We are able to providesecure and centralized communications with the aid ofa common authentication gateway VW-AUTH was de-veloped as a micro service to provide this functionalityIt aggregates functionality for registration authentica-tion and authorization for system users

The Authentication component itself exposes REST-ful endpoints that provide user access to different ser-vices in addition to endpoints for authentication andregistration To enable the verification of a clientrsquosauthentication by other services a JSON Web Token(JWT) based authorization scheme is utilized As anRFC standard for exchanging information securely be-tween a client and a server the using JWT ensures ahigh level of security in this system A standard work-flow for user authorization is depicted in Figure 3

Users can manage uploaded models via a RESTfulAPI endpoint provided by the Model Web Service Us-ing this endpoint a properly authenticated user is ableto request a model run upload necessary input filesneeded by the model and start model execution Modelrun data is stored by the VWS in a dedicated databaselocated on a server operating out of the University ofNew Mexico [1]

Available models in the system are self-describingwith schema that indicates necessary input files for-mats and execution policies The schema also showsthe mapping between user facing and model adaptorparameters A userclient can use this schema to under-stand the necessary resources required to run a givenmodel The a typical model run from the perspectiveof the clinent side application involves six steps 1)retrieve the server-side model schema 2) instantiatea model run session on the server 3) upload modelinputs 4) run the model 5) track model run progressand finally 6) download outputs

Figure 4 shows a simple workflow chart from a useror clientrsquos perspective This workflow offers many ad-vantages this workflow and sever-client architecturecompared to traditional approaches to model running

wwwastesjcom 6

Figure 4 Workflow from userrsquos perspective to run a model

bull Users do not need to know internal specifics ofthe model Setup details and dependencies arehidden from the user which provides a moreseamless experience

bull Users donrsquot need to worry about installing modeldependencies

bull Users can easily initiate multiple parallel modelruns on a server expediting research

bull Users have ubiquitous access to server-side datavia provided RESTful APIs

The first step in creating the architecture for expos-ing ldquomodels as servicesrdquo is enabling the programmaticrunning of models A hurdle to accomplishing this isthat a model can have many different dependencies re-quired to run Programmatic running is accomplishedin this program by providing developers the tools tocreate python wrappers around models These wrap-pers expose dependencies to model through containerimages

In creating this system we had to strongly considerproblems of data format heterogeneity with model in-puts and outputs Different models commonly acceptand output data in different file formats To achievefacilitate greater data interoperability between thesemodels we provided an option for developers to writeNetCDF adapters for each models A model adaptor isa Python program that handles data format conversionmodel execution and progress notification Adaptordevelopers must provide converters which define themethod of conversion and deconversion between thenative formats used by the model and netCDF An ex-ecution function must also be implemented for themodel In this method resource conversion and modelexecution occurs The wrapper reports on model exe-cution events via a provided rdquoevent emitterrdquo An eventlistener can catch these events as they are reported bythe emitter This listener provides a bridge betweenthe internal progress of the model and the user withthe aid of a REST endpoint

Through an adaptor models can be encapsulatedfor programmatic execution However to bridge fron-tends with actual model execution a process is requiredfrom the model worker module Utilizing a messagingqueue we create a bridge between the client frontendand worker backend When a run task is submittedthrough from the client side it is placed into the queueand assigned a unique id The consumerworker pro-cess listens to the queue through a common protocolfor new jobs

The worker service is a server-side python mod-ule that has access to a modelrsquos executable code andinstalled dependencies The worker resides in an iso-lated server instance that contains the dependenciesand libraries of the model installed This module usesLinux containerization to facilitate easy deployment ofmodel workers34 Deployment Workflow

A Linux-container-based deployment workflow hasbeen devised for the for many different componentsof VWS We utilized the containerization softwareDocker to enable our implementation of this work-flow [22] Docker containers have advantages overtraditional virtual machines because they use fewerresources This workflow allows for iterative deploy-ment simple scaling of the containerized componentsand provides a strategy to register new models in thesystem

wwwastesjcom 7

Figure 5 Registration page of Virtual Watershed System

Every VWS component is containerized withDocker We have a set up a central repository of dockerimages which contain images for each component inthis system A docker rdquoimagerdquo describes a templatethat provides Docker with the OS dependencies of anapplication and the application itself Docker uses thistemplate to build a working container Each repos-itory in the Virtual Watershed has a Dockerfile thatdescribes how the component should be built and de-ployed into a query-able container The repositoriesuse webhooks to automate the building of docker con-tainers

It is easy to register new models in this system us-ing this containerization workflow Using our systema user may register a model with the creation of dockerimage describing the model With this image users candeclare the OS libraries and other dependencies for amodel A typical registration of a model requires thefollowing steps 1) create a repository for the model 2)develop the wrapper 3) specify dependencies withinthe dockerfile and finally 5) create a docker image inthe image repository

4 System Prototype and Testing

The project re-used some code and similar structuresto those introduced in [23 24] The web service fron-tends were built using a Python micro-service frame-work called Flask with various extensions [25] Some

key libraries used were 1) Flask-Restless used to im-plement the RESTful API endpoints 2) SQLAlchemyto map the python data objects with the databaseschema [26] 3) PostgreSQL as the database [27] and4) Flask-Security with Flask-JWT which provides theutilities used for security and authentication Celerywas used to implement the task queue for model work-ers [28] Redis worked as a repository for model resultsoutput by Celery A REST specification library calledSwagger was used to create the specification for theREST APIs For the web front end HTML5 CSS Boot-strap and Javascript (with ReactsJS) were used

Figure 6 Dynamically generated upload form

To efficiently develop the VWS an MVC design pat-tern was used to structure the code The primary repos-itory used to manage and distribute code was GithubSource code for this project is made publicly availablethrough the Virtual Watersheds GitHub repository [28]Dockerhub [29] is used for image management whichcan automatically update images when Github code ischanged

The prototype system is comprised of two maincomponents 1) the authentication module and 2) themodeling module First activities related to the VWSare handled by the authentication module All stan-dard user management functionality is handled bythis module registration logins verification pass-word management and authentication token genera-tion Figure 5 shows the registration page of VirtualWatershed system In addition to this interface theVWS also provides endpoints for registration and au-thentication from user constructed scripts

wwwastesjcom 8

Figure 7 Scenario creation interface for PRMS model that usesModeling REST API to execute model

Modeling is necessary and commonly used in hy-drologic research Modification of the existing modelsimulations is a complex activity that hydrologists of-ten have to deal with while analyzing complicated en-vironmental scenarios Modelers must make frequentmodifications to underlying input files followed bylengthy re-runs Programming languages could helpsignificantly with this easily automated and repetitioustask However it is complex and time consuming forhydrologists to write their own programs to handlefile modifications for scenario based studies To solvethis challenge the modeling module provides an in-tuitive user interface where users can create uploadrun and delete models The UI informs users about theprogress of the model run with a bar that displays apercentage of completion The module also includes adashboard where users can view the models being runthe finished model runs and also download the modelrun files A user can execute multiple PRMS models in

parallel by uploading the three input resources neededfor PRMS model The upload interface as shown in Fig-ure 6 is generated dynamically from the model schemadefined for each of the models

The modeling interface displays peak utility whenusers use it to programmatically run a bundle of mod-els in parallel without having to worry about resourcemanagement In the prototype system the client isused to adjust input parameters for models and formodel execution

Model calibration can be a time consuming processfor many hydrologists It often requires that the mod-eler re-run and re-run a model many time with slightlyvaried input parameters For many modelers this is amanual process They will edit input files with a texteditor and execute the model from a command line ontheir local machine

Understanding that it was crucial to address thisissue the virtual watershed protoype system was de-veloped with a web-based scenario design tool Thistool provides a handy interface which enables for therapid adjustment and calibration of PRMS model in-put variables It also provides the tools to execute thetweaked models via the modeling web service andvisually compare outputs from differently calibratedruns Figure 7 shows the interface developed whichprovides visually oriented tools for data manipulationThis interface abstracts away technical model data rep-resentations and allows users to model in intuitiveways Figure 8 shows the output visualization after athe modeling web service has finished executing themodel run

Figure 8 Comparing the results of the scenarios created using themodeling service

We developed multiple IPython [30] notebooks todemonstrate the programmatic approach to runningan ensemble of models in parallel by using the mod-eling API by Matthew Tuner who is a PhD Student

wwwastesjcom 9

in Cognitive and Information Sciences at Universityof California Merced Figure 9 shows an importantstep of modeling with the PRMS model By using themodeling API an user can execute multiple modelsin parallel programmatically Users can use serverresources to run a model many many times withoutconcerning themselves about limited time and CPUresources

Figure 9 Example of tuning a hundred model through an IPythonnotebook with different parameters

5 Comparison with Related Tools

A feature based comparison is made between similarsoftware tools focused on hydrologic and environmen-tal modeling in Table 1 The tools discussed in Sec-tion 2 are CSDMS [8] Hydroshare [9] and MAAS [10]Though each of these tools were designed and devel-oped with different goals in mind all of them are ademonstration of work to assist environmental model-ing in general Each of these tools has their pros andcons from different users perspectives

CSDMS is a community driver system where model-ers can submit their models to CSDMS by implement-ing model interfaces provided by them The modelgets evaluated and added into the system by the CS-DMS committee CSDMS uses a language interoper-

ability tool called Babel to handle language heterogene-ity From the userrsquos perspective CSDMS provides aweb and desktop based client where users access mod-els and run them using different configurations fromwithin the client On the server side CSDMS main-tains an HPC cluster where they have all the modelsand relevant dependencies installed

Hydroshare [9] on the other hand is a projectstarted to accommodate data sharing and modeling forhydrologists The platform is in active developmentand frequently adds new features The current ver-sion of Hydroshare is more concentrated towards datasharing and discovery for hydrology researchers [31]Hydroshare also provides programmatic access to itsREST API which makes it a good candidate for a datadiscovery backend for any other similar system

The MaaS [10] is the most similar project to theVirtual Watershed modeling tool The main advantageour work has over MaaS is resource management DrLi et al proposed MaaS but they use Virtual Machinetechniques in their framework In our system Dockercontainers are used which require fewer resourcesAlso we implemented APIs to check and stop a dockercontainer based on model execution status Existingcontainer orchestration tools like Docker Swarm arelimited in thier ability to manage containers in thisway Though the approach and implementation of thiswork differ in various ways the end goal is similarto what we are trying to achieve MaaS introducedmodels as services by creating an infrastructure usingcloud platforms The framework provides users with aweb application to submit jobs The backend that takescare of on-demand provisioning of virtual machinesthat containing model execution environment setupMaaS achieves model registration by encapsulating amodel as a virtual machine image in an image hub Itprovides an FTP based database backend to store themodel results

6 Conclusion and Future Work

A hydrologic model execution web service platformnamed VWS is described in this paper The key idea ofthe proposed platform is to expose model relevant ser-vices through python wrapper This wrapper enablesrepresentation of model resources with a common dataformat (eg NetCDF) increasing inseparability Alsoit requires NetCDF format resources which simpli-fies model execution on a server node A user canlogin once and access all services of the VWS withthe authenticationauthorization component EachVWS Component is wrapped in a docker containerwhich requires less resources than a traditional Vir-tual Machine Docker container APIs including thecontainer termination API are also implemented toautonomously manage the dynamic resource needs ofthis system

This software has significant room to be expandedupon with furutre work For example the systemcurrently only provides the option to integrate with

wwwastesjcom 10

Table 1 Feature Comparison with Related Hydrologic and Environmental Modeling tools

FeatureTool

CSDMS HYDRO-SHARE MAAS VW-MODEL

Open source Yes Yes No YesProvides REST API No Yes No YesProvides interface for model implementation Yes No No YesAllows model coupling Yes No No NoAllows model registration Yes No Yes YesData storage No Yes Yes YesData sharing and discovery No Yes No No

GSToRE data backend This can be improved throughimplementation of a generic data backend which en-ables integration with other existing data providerslike DataONE Hydroshare and CUHASI This in turncould provide better access to data for modeling au-tomation Developing a pricing component for serviceusage would be considered for future developmentThis aspect can help system manager to sustainablymaintain the server and hire software developers toimplement other useful features

Conflict of Interest The authors declare no conflictof interest

Acknowledgment We would like to thank MatthewTurner for his sample python scripts described in Sec-tion 4 and all the reviewers for their suggestions

This material is based upon work supported bythe National Science Foundation under grant numbersIIA-1301726 and IIA-1329469

Any opinions findings and conclusions or recom-mendations expressed in this material are those of theauthors and do not necessarily reflect the views of theNational Science Foundation

References

[1] Nmepscororg The Western Consortium for Wa-ter Analysis Visualization and Exploration mdashNew Mexico EPSCoR [Accessed on 10 June2018] 2018 url https www nmepscor

org science 5C 5Cwestern - consortium -

water-analysis5C5C-visualization-and-

exploration

[2] Md Moinul Hossain Rui Wu Jose T PainumkalMohamed Kettouch Cristina Luca Sergiu MDascalu and Frederick C Harris ldquoWeb-serviceframework for environmental modelsrdquo In In-ternet Technologies and Applications (ITA) 2017IEEE 2017 pp 104ndash109

[3] Russ Rew and Glenn Davis ldquoNetCDF an inter-face for scientific data accessrdquo In IEEE computergraphics and applications 104 (1990) pp 76ndash82

[4] Danny Marks James Domingo Dave SusongTim Link and David Garen ldquoA spatially dis-tributed energy balance snowmelt model for ap-plication in mountain basinsrdquo In Hydrologicalprocesses 1312-13 (1999) pp 1935ndash1959

[5] Steven L Markstrom Robert S Regan Lauren EHay Roland J Viger Richard M Webb RobertA Payn and Jacob H LaFontaine PRMS-IV theprecipitation-runoff modeling system version 4Tech rep US Geological Survey 2015

[6] Chao Chen Ajay Kalra and Sajjad AhmadldquoHydrologic responses to climate change usingdownscaled GCM data on a watershed scalerdquo InJournal of Water and Climate Change (2018)

[7] Chao Chen Lynn Fenstermaker HaroonStephen and Sajjad Ahmad ldquoDistributed Hy-drological Modeling for a Snow Dominant Water-shed Using a Precipitation and Runoff ModelingSystemrdquo In World Environmental and WaterResources Congress 2015 2015 pp 2527ndash2536

[8] Scott D Peckham Eric WH Hutton and Boy-ana Norris ldquoA component-based approach tointegrated modeling in the geosciences The de-sign of CSDMSrdquo In Computers amp Geosciences 53(2013) pp 3ndash12

[9] David G Tarboton R Idaszak JS HorsburghD Ames JL Goodall LE Band V Merwade ACouch J Arrigo RP Hooper et al ldquoHydroSharean online collaborative environment for thesharing of hydrologic data and modelsrdquo In AGUFall Meeting Abstracts 2013

[10] Zhenlong Li Chaowei Yang Qunying HuangKai Liu Min Sun and Jizhe Xia ldquoBuildingModel as a Service to support geosciencesrdquo InComputers Environment and Urban Systems 61(2017) pp 141ndash152

[11] Michael P McGuire and Martin C Roberge ldquoThedesign of a collaborative social network for wa-tershed sciencerdquo In Geo-Informatics in ResourceManagement and Sustainable Ecosystem Springer2015 pp 95ndash106

[12] Eric Fritzinger Sergiu M Dascalu Daniel PAmes Karl Benedict Ivan Gibbs Michael JMcMahon and Frederick C Harris ldquoThe Deme-ter framework for model and data interoperabil-

wwwastesjcom 11

ityrdquo PhD thesis International EnvironmentalModelling and Software Society (iEMSs) 2012

[13] Jeffrey D Walker and Steven C Chapra ldquoA client-side web application for interactive environ-mental simulation modelingrdquo In EnvironmentalModelling amp Software 55 (2014) pp 49ndash60

[14] Jon Wheeler ldquoExtending Data Curation ServiceModels for Academic Library and InstitutionalRepositoriesrdquo In Association of College and Re-search Libraries (2017)

[15] Thomas Erl Service-oriented architecture (SOA)concepts technology and design Prentice Hall2005

[16] Roy T Fielding Architectural styles and the de-sign of network-based software architectures Vol 7University of California Irvine Doctoral disser-tation 2000

[17] Alex Rodriguez ldquoRestful web services The ba-sicsrdquo In IBM developerWorks 33 (2008)

[18] Leonard Richardson and Sam Ruby RESTful webservices OrsquoReilly Media Inc 2008

[19] Peter Buxmann Thomas Hess and SonjaLehmann ldquoSoftware as a Servicerdquo InWirtschaftsinformatik 506 (2008) pp 500ndash503

[20] James Lewis and Martin Fowler ldquoMicroservicesa definition of this new architectural termrdquo InMartinFowler com 25 (2014)

[21] Sam Newman Building Microservices OrsquoReillyMedia Inc 2015

[22] Dirk Merkel ldquoDocker lightweight linux contain-ers for consistent development and deploymentrdquoIn Linux Journal 2014239 (2014) p 2

[23] Md Moinul Hossain ldquoA Software Environmentfor Watershed Modelingrdquo PhD thesis Universityof Nevada Reno 2016

[24] Rui Wu ldquoEnvironment for Large Data Process-ing and Visualization Using MongoDBrdquo MA the-sis University of Nevada Reno 2015

[25] Miguel Grinberg Flask web development develop-ing web applications with python OrsquoReilly MediaInc 2018

[26] Rick Copeland Essential sqlalchemy rdquo OrsquoReillyMedia Incrdquo 2008

[27] Bruce Momjian PostgreSQL introduction andconcepts Vol 192 Addison-Wesley New York2001

[28] Ask Solem Celery Distributed Task Queue [Ac-cessedon 10 June 2018] 2018 url httpwww20celeryproject20org

[29] Docker Docker Hub [Accessed on 10 June 2018]2018 url httpshubdockercom

[30] Fernando Perez and Brian E Granger ldquoIPythona system for interactive scientific computingrdquo InComputing in Science amp Engineering 93 (2007)

[31] Daniel P Ames Jeffery S Horsburgh Yang CaoJirı Kadlec Timothy Whiteaker and DavidValentine ldquoHydroDesktop Web services-basedsoftware for hydrologic data discovery down-load visualization and analysisrdquo In Environ-mental Modelling amp Software 37 (2012) pp 146ndash156

wwwastesjcom 12