Akdeniz ve Karadenizin Trkiye iklim sistemine etkisinin birleik bir atmosfer-okyanus modeli ile incelenmesi

Overview of Model Coupling in Earth System Sciences and Recent Developments

Ufuk Turuncoglu (1,2)

(1) Informatics Institute, Computational Science and Engineering, ITU, Turkey(2) ESP Section, ICTP, ItalyDeveloper School for HPC applications in Earth Sciences, 10-12 Nov. 2014

OutlineEarth System Modeling and EvolutionModel CouplingTechniques and ToolsBasic Definitions and Design StrategiesInterpolationLand-sea Mask IssuesAutomatization via WFRegESMComponentsDesignRun SequenceConcurrent vs. SequentialProject Home and DocumentationEarth SystemCombination of different components and each one of them interact with both positive and negative feedback mechanisms

http://paos.colorado.edu

MetOfficeEarth SystemTime Scales

Time scales and response times are different !

Complex interaction and feedbackmechanismsamongcomponentsIPCC Report

Climate ModelingTimeline

by Steve Easterbrook First ESMCoordinated Experiments CMIPsRapid developmentafter 80sThe complexity of the models are getting increasehttp://prezi.com/pakaaiek3nol/timeline-of-climate-modeling/NWF byRichardsonAutomatized NWF by Von NeumannHigh Performance ComputingTimeline

There is a close relationshipbetween development inHPC and climatemodeling

The developmentcurve (or slope) ismore steep after80s!

Complexity of the Climate ModelsGrowth of Climate Modeling

by UCAR

Model complexity is increasing: in terms ofPhysicsComponentsmain breaking pointAlong with the development in HPCsIncreased Horizontal and Vertical Resolution Climate AssessmentsEvolution of Models

IPCC

More processes (i.e. chemistry, interactivevegetation etc.) are represented bymodelsIn higher temporal andspatial scaleClimate AssessmentsType of ModelsSimple energy balance models very sophisticates ESMs

Atmosphere-Ocean General Circulation Models (AOGCMs)Earth System Models (ESMs) AOGCM + Biogeochemical Cycles (Carbon, Sulphur, Ozone)Earth System Models in Intermediate Complexity (EMICs)Idealized or lower resolution. Applied to answer specific questions. Needs less computational powerRegional Climate Models (RCMs)Limited area models to downscale results of the lower resolution models. It might also have multiple components!

http://www.climatechange2013.org/images/report/WG1AR5_Chapter09_FINAL.pdf

Climate AssessmentsEvolution of IPCC Reports

http://www.scientificamerican.com/article/climate-assessments-25-years-of-the-ipcc/No decrease in uncertainty!but models are more accurate in 20CEvolution of Published PapersCoupled Modeling

Keyword: Coupled climate model (in Topic)

Keyword: Earth System Model (in Topic)Source: Web of ScienceWhat is Model Coupling?It is a sophisticated way of simulate climate system and complex interactions between its components.The individual components are not perfect !Our theoretical understanding of climate is still incomplete, and certain simplifying assumptions are unavoidable when building these models (Reichler and Kim, 2008) ! Coupled model does not always gives best results but help to understand interactions and processes between components !

Difficulties in Model Coupling?Example from Atmosphere + Ocean Coupling

Initial state of the ocean is not precisely knownLack of continuous and high resolution observations. Remote sensing might help but covers only ocean surface and recent decades. The quality also depends on many issues (atmospheric correction and used algorithm, coastal regions etc.)

An imbalance in the surface flux (heat, momentum and freshwater) much smaller than the observational accuracy is enough to cause a drifting of coupled GCM simulations into unrealistic statesIt is not easy tightly couple models that are written by different groupsHigh accuracy of conservation is needed!http://www.ipcc.ch/publications_and_data/ar4/wg1/en/ch1s1-5-3.htmlModel CouplingBut still, common trend is to create multi-component and multi-scale model applications and simulationsIt needs close collaboration between different disciplines and research groupsCESS: Computational Earth System ScienceComputer ScienceScience &EngineeringEarth SystemScienceMeteorologyOceanographyLand Surface + HydrologyChemistryDesigning efficient and scalable modelsHandling high volume of data (data compression, parallel I/O)Automatization of model handling, reproducibility and metadata/provenance information collectionIS-ENES2, METAPHOR+CIMCuratorCMIPs (CMIP3/5)ESGF

Types of Model CouplingThere are two types of model coupling

OfflineModels run sequentially and the interaction between them is in only one way. The simplest example is dynamical downscaling (GCM to RCM) or one-way nesting (RCM to RCM)OnlineIn this case, models interact in both way and feedback mechanisms among components represented (two way nesting, fully coupled atmosphere-ocean models or ESMs.)

Techniques for Model CouplingThe ESM components might be coupled using different programming approachesmore information in Valcke, 2009 EGUMerge individual model codessingle executable!

program ocn! read initial and boundary datacall read()! run modelcall run()! write results to fileend program ocnOCN easy to code, portablesubroutine ocn (exchange vars)! read initial and boundary datacall read()! run modelcall run()! write results to fileend subroutine ocnprogram atm! read initial and boundary datacall read()! run modelcall run()! write results to fileend program atmhost / parentATMprogram atm! read initial and boundary datacall read()! run modelcall run()! run ocn modelcall ocn(exchange vars)! write results to fileend program atm not flexible andgeneric http://www.cerfacs.fr/~coquart/pagecerfacs/seminars/201405_OASIS3MCT.pdfTechniques for Model CouplingThe ESM components might be coupled using different programming approachesmore information in Valcke, 2009 EGUUse existing communication protocols: MPI, InterComm program ocn! read initial and boundary datacall read()! run modelcall run()! write results to fileend program ocnOCNprogram atm! read initial and boundary datacall read()! run modelcall run()! write results to fileend program atmhost / parentATMprogram atm! receive exchange fieldcall recv()! run modelcall run()! send exchange fieldcall send()! write results to fileend program atmprogram ocn! receive exchange fieldcall recv()! run modelcall run()! send exchange fieldcall send()! write results to fileend program ocnmultiple executable!ATMOCN existing code and concurrent components not easy to implement, not flexible andgeneric Techniques for Model CouplingThe ESM components might be coupled using different programming approachesmore information in Valcke, 2009 EGUUse coupling framework & libraries: FMS, ESMF, MCT

Code divided into units and calling interface redesigned (i.e. initialization, run and finalize)Hierarchical model structures are supported

Existing codes must be restructured

flexible, efficient, portable, support to use generic utilities for interpolation, time management and unit conversions), support both sequential and concurrent components

Techniques for Model CouplingThe ESM components might be coupled using different programming approachesmore information in Valcke, 2009 EGUUse a coupler: OASIS, OASIS-MCT, MPCCI, PALM

Coupler link the model components and handle coupling processes such as interpolation, synchronization etc.Components models might be separate or single executable

flexible, efficient, portable, support to use generic utilities for interpolation, time management and unit conversions), support concurrent components

multi-executable: more difficult to debug; harder to manage for the OS and possible waste of resource if seq. execution enforced

What is driver / coupler ?The driver can be defined as a glueIt merges the different components of the modeling systemIn general, the modeling system might be designed to haveSingle executableMultiple executable (each component has an executable)In the multiple executable case, the usage of the modeling system might be difficultThe driver is generally responsible forInteraction among the components: exchange boundary data between the components, regridding etc.Synchronization: to coordinate the time evolution of the physical models (coupling time step)20Coupled Model DesignGlobal Models (GCMs and ESMs)There is no single design and ultimate solution!Kaitlin Alexander and Steve Easterbrook

http://climatesight.files.wordpress.com/2011/08/poster.pdf

Coupled Model DesignExamples for Regional Modelshttp://www.remo-rcm.de/Atmosphere-Ocean-coupled.1269.0.htmlMORCE, IPSL

REMO-RCM, Max PlanckRCA4-NEMO, SMHIhttp://www.smhi.se/polopoly_fs/1.28917!/RO_47.pdfData exchange and interpolation is one of the most important component of the model design!Common Problems in ES ModelingThe models are developed by different groups / organizationsVarious types of grids (curvilinear, regular, triangular etc.)Written in various programming languages (Fortran, C, C++)Different I/O strategies (ASCII, binary and netCDF)Different configuration mechanisms (simple ASCII files or well structured XML files - CESM)Each component (ATM, OCN, etc.) has its own learning curveGenerally, does not follow common programming standards (CF conventions, GridSpec etc.)Requirements:Need to know the way of real programmingNeed close collaboration between scientists and programmersThe standardization is very crucial. We need to think more generic solutions: standardized codes = better interoperability

23Common Problems in Coupled Model DesignNeed more attentionData exchange and interpolation between model gridsSupport for different types of gridsConservation of flux fields (high order conservative regridding?)Extrapolation might be needed for unmatched land-sea masks. Exchange grid might help!Self describing modelsNeed to document a simulation or runIt is important for reproducibilityBoth metadata and provenanceinformation needed for trackingIs it possible to collect automatically?Synchronization of multi-componentmodelsTools like OASIS and ESMF help!adapted from Steve EasterbrookRe-run and get same resultsRe-run the code and get different resultsConfirm results using a different methodRepeatability = Reproducibility24Interpolation / RegriddingThe problem arises when numerical grid of the model components are not same (i.e. atmosphere and ocean) ?There are two main types:ConservativeNon-conservative

https://wiki.cc.gatech.edu/CW2013/images/8/89/201302_OASIS3MCT_CW2013.pdf

25Interpolation / RegriddingConservative type regridding is used to preserve flux fields (i.e. moisture and heat fluxes)It can be implemented to conserve field in Local: integral of the local patches are equalGlobal: integral of the source and destination grid is equalThe current tools uses 1st order conservative regriddingThe resource field might have artifacts if the ration between source and destination grid resolution is high (i.e. atmosphere model in 50 km but ocean model 10 km)http://www.earthsystemmodeling.org/presentations/pres_1002_siam_ryan.pdf

26Interpolation / RegriddingIn general, coupled models follows SCRIP conventionSteps involved regridding Generate interpolation weightsApply weights to interpolate field using a sparse matrix multiply

The weight generation might be online (via subroutine calls): ESMFoffline (external applications such as, ESMF_RegridWeightGen and SCRIP): OASIS, MCT, ESMFhttp://oceans11.lanl.gov/trac/SCRIP

27Peggy Li, 2010, ESMFvsSCRIP-v2.dochttp://pantar.cerfacs.fr/globc/publication/technicalreport/2012/SCRIP_ESMF_LiYan.pdf SCRIP vs. ESMF_RegridWeightGenSCRIP does calculations in (lat,lon) space but ESMF does it in (x,y,z) spaceDifferent area calculation for conservative remappingSCRIP is serial but ESMF provide also parallel execution Interpolation / Regridding

ESMF Interpolationis currently used by NCL and ESMPy28Different model components with different land-sea maskPlay with land-sea mask of each components to match themIt is hard and very time consumingMust be repeated for each individual applications. It is not a generic solutionUse exchange grid (needs conservative type regridding)

Two step interpolation Interpolation + ExtrapolationUnmatched Land-sea Masks

OCNATMMismatch

orphan grid cellassigned to land (arbitrarily)clipped29Two step interpolation (i.e. interpolation over ocean)

Unmatched Land-sea Masks

STEP 2Use result of previous step, interpolate data from OCN to OCN from mapped grid points to unmapped ones usingnearest-neighbor type regriddingMerge results of 1 and 2 tocreate filled field

+STEP 3=Still has problem in some applications (sharp gradient in some cases) but used in RegESMOther extrapolation techniques?Thanks to ESMF Group (especially to Bob Oehmke) for their support and help

STEP 1Interpolate from ATM to OCNusing bilinear interpolation. Use only sea grid pointsmappedunmapped30Automatization The models are getting more complex (lots of configuration file and parameters, huge output) and it is not easy to run and keep track of simulations. It is very time consuming.

ConceptualCCSM / CESMWorkflow31Automatization Scientific workflow tools might help to simplify the regular tasks!Teragrid:

creates WS-GRAM XML job description files automaticallybuild modelcollectprovenance and metadata inforun modelTuruncoglu, CG: 2011, EMS: 2013 and ESM Book (Vol. 5): 201232Automatization Scientific workflow tools might help to simplify the regular tasks!Modified workflow for conventional cluster

workflow wide globalparametersbuild modelcollectprovenance andmetadata inforun modelTuruncoglu, CG: 2011, EMS: 2013 and ESM Book (Vol. 5): 201233Modeling Experiences in ICTPRegCMRegional Climate Model (hydrostatic)Sophisticated sub-grid parameterizationCoupled with Atm. Chemistry, 1D Lake and Slab Ocean ModelsAGCM (Speedy) Intermediate Complexity ModelSimplified GCM (Molteni and Kucharski)Spectral Model (hydrostatic)Slab models for land and sea-iceICTP-NEMO Coupled ModelSpeedy + NEMO + OASIS3T30/L8 and 2 deg (+ equatorial refinement 1/3 deg)Supports one and two way couplinghttp://indico.ictp.it/event/a10154/session/6/contribution/4/material/0/0.pdfNew regional modeling system (RegESM)? To design easy to use and extend modeling systemENEAs PROTHEUS system uses MITgcm+RegCM3+OASISIt is not easy to use, extend and upgrade (It uses multiple executable, no online interpolation support, hard to add new component such as wave)To support Med-CORDEXSet of coordinated experiments to better understand Mediterranean climate: standalone and coupled RCMsIt would be good to have another regional coupled modeling system that uses different model componentsTo gain experience about design and use of coupled modeling systemsThe applications of coupled modeling system getting increased in last decade along with the rapid development in HPCEvolution of RegESMThe first prototype version is created in 2012There were no driverRegCM was host the whole coupled modeling systemSingle ocean component was supported (ROMS)Poor energy conservation (for exchange fields)Applied to Caspian Sea (Turuncoglu et al., 2013, GMD)

Then, more generic version is designed and released in 2013Centralized driver using ESMFs NUOPC layer (via connectors)All the components plugged into driver (less dependence to the model component code)Support for multiple ocean model (ROMS and MITgcm)Energy conservation is improved (global conservation)Support to extrapolation (land-sea mask mismatch)Applications: Mediterranean, Black Sea and Caspian Sea

RegESM Design & ComponentsCurrently, RegESM supports three different component:

ATM:RegCM

OCN:ROMS / MITgcm

RTM:HD (special thanks to Prof. Stefan Hagemann)

DRIVER:RegESM

atmosphericchemistryTwo differentland surface modelDriver Design in ESMF/NUOPCThe current driver uses only connector component (NUOPC)

The driver defines;The model components (i.e. RegCM, ROMS) grid, fields, states etc.Coupler component (connector)Global clock and calendarand synchronizes the sequence of actions

https://www.earthsystemcog.org/projects/nuopc/38Driver Designs in ESMF/NUOPCDifferent designs are possible with NUOPC layer

The mediator approach is much more efficient when the number of components increaseSimple Coupling with ConnectorsATM+OCNCoupling through a Mediator

Connectorshttps://www.earthsystemcog.org/projects/nuopc/39Run Sequences in ESMF/NUOPCThe NUOPC Layer is capable of implementing many different coupling schemesExplicit: Exchange data in same time. Two connector (i.e. backward and forward)Explicit with slow and fast clock support:Different interaction time step among the componentsSemi-implicit (leap-frog style interaction)Fully-implicit

Run Sequence in RegESMThe RegESM might use both explicit and semi-implicit coupling schemes along with the support of fast and slow time steps.Fast interaction between ATM and OCNSlow interaction between ATM and RTM, RTM-OCNHere, RTM is optional component

Example for 3 component and explicit coupling:Sequential vs. Concurrent ExecutionThe modeling system supports two different approach to run the model components.

In sequential mode: model components are run in orderIn concurrent mode: all models are active at same time (it does not allow overlapping of the used PETs cores / CPUs)

Repository and Project HomeThe RegESM project is hosted by GitHubhttps://github.com/uturuncoglu/RegESMThe project home page

Implemented by using CoG1

http://cess.be.itu.edu.tr/projects/regesm/I) https://www.earthsystemcog.org/projects/cog/DocumentationInitial version of user guide is readyBasic model designInstallation of libraries, components and RegESM modelDefinition of configuration files(exchange fields and model)Known bugs and limitationsSend any comments by e-mailExperience about installation in different computing systemBugsUnfortunately, there is no support ticket handling system yet!Results of your applications (tested domains, detail about configuration etc.)

Questions!

Contact:u.utku.turuncoglu@be.itu.edu.tr

Dont miss Lab-Session related with ESMF at 16:30 !!!