O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY 1 Global (and Regional) Climate Modeling Overview What is the climate system? How is it modeled?

1

OAK RIDGE NATIONAL LABORATORYU.S. DEPARTMENT OF ENERGY

Global (and Regional) Climate Modeling

Overview• What is the climate system?

• How is it modeled?

• What are the computational issues?

John Drake Computational Climate Dynamics Group Computer Science and Mathematics DivisionOak Ridge National Laboratory

For Fall Creek Falls Workshop, October 27, 2003


Why is DOE Interested in Climate?



Climate Change Doesn’t Just Happen

• A Key figure


PCM Ensembles of 21st Century Business as Usual

• Results show that the global warming trend since the late 1970’s is likely to continue through much of the twenty-first century.

• Model includes a full range of greenhouse gas changes together with sulfate aerosol effects and no artificial flux adjustments.

• The pattern of surface warming from 1961-90 to 2070-99 under BAU scenario shows that warming ranges between 1 and 2 C over oceans and is above 2 C over many land areas, especially in northern high latitudes during winter where the warming is above 5 C.

• The meridional overturning in the North Atlantic is reduced by 20% from 1961-90 to 2070-99… reduced local vertical mixing causes cooling over the mid-latitude North Atlantic.

• Ensemble-averaged precipitation in BAU shows a 20%-40% increase at high latitudes during winter and a 10%-30% decrease over subtropical dry areas.


Simulation of Future Climates

Computing for this simulation was done at DOE's National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory, NCAR and Oak Ridge National Laboratory Center for Computational Sciences (CCS). Credits Animation and data management:Michael Wehner/Lawrence Berkeley National Laboratory Parallel Climate Model:Warren Washington, Jerry Meehl, Julie Arblaster, Tom Bettge and Gary Strand/National Center for Atmospheric Research Visualization softwareDean Williams/Lawrence Livermore National Laboratory


PCM Business as Usual and Stabilization Scenarios

• The ensemble-mean temperatures under the BAU and STA550 scenarios start to diverge in 2040 but become significantly different only after the mid 2060’s.

• Stabilized atmospheric CO2 at 550ppm by 2150 will only slow down the warming moderately during the 21st century but could be large (1.5 C globally and 12 C in DJF at northern high-latitudes) by the later part of the 22nd century.

Dai, Meehl, Washington, Wigley, Arblaster, Ensemble Simulation of Twenty-First Century Climate Changes, BAMS Vol 82, No. 11, November 2001

Dai, Wigley, Meehl and Washington, Effects of Stabilizing Atmospheric CO2 on Global Climate in the Next Two Centuries, GRL Vol. 28, No. 23, December 2001



Why a Community Model?

• NCAR adopted to encourage climate research in universities using state of the art models. –1980

• New components of a coupled system. - 1990• Individual projects unable to adapt to parallel architectures

-2000• Pool resources and complete mission critical simulations

for NSF, DOE and NASA• CCSM has become the “national model”• Annual Breckenridge CCSM Workshop grown to over 270

researchers


DOE SciDAC Workshop on Porting CCSM to the CRAY X1

• Goals Identify individuals and organizations engaged in

porting one or more of the CCSM component models

Report progress and problems in current CCSM vectorization activities.

Identify gaps or issues in the current efforts. Establish lines of communication between the

different efforts and NCAR software engineers to encourage sharing of results and code.

Begin defining requirements and procedures for the adoption ofvector-friendly code in future released versions of CCSM.

• Represented: NCAR, NASA-Goddard, ORNL, LANL, LBNL, Cray, NEC, Fujitsu, CRIEPI

• Held Feb. 2003 in Boulder, also June in Breckenridge 2003


Coupler ArchitectureTony Craig, Rob Jacob, Brian Kaufman, Jay Larson, E. Ong

Figure 6: Coupler structure

Issues:Issues: •sequencingsequencing•frequencyfrequency•distributiondistribution•parallelism parallelism •single or multiple single or multiple executablesexecutables•stand alone executionstand alone execution

Version 1.0 ReleasedNovember 2002

• MPH3 (multi-processor handshaking) library for coupling component models

• CPL6 -- Implemented, Tested, Deployed


Evolution of Performance of the Community Atmospheric ModelPat Worley, John Drake

•Hybrid MPI/OpenMP programming paradigm•Cache friendly chunks, load balance, improved algorithms


Land Surface Model and River Transport Model Forrest Hoffman, Mariana Verenstein, Marcia Branstetter

Community Land Model SciDAC software engineering is focused on the interface and reduction of

gather/scatters; communications bottleneck removed Rewrite for vectorization and CLM2.2 now complete RTM is currently single processor -- designing parallel implementation and data

structures Analysis of runoff in CCSM control simulation. Effect on July ocean salinity.

Amazon

0

50,000

100,000

150,000

200,000

250,000

300,000

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Dis

ch

arg

e (

m3 /s

)

CCSM2

Observed

CCM3


POP Ocean ModelP. Jones, J. Dukowicz, J. Baumgardner, W. Lipscomb

• Software Engineering for POP and CICE Design and implementation for the new ocean model (HYPOP) and

CICE in progress

• Ocean Model Performance POP2: new design involves a decomposition of the computational

domain into blocks that can be sized to fit into cache On 1/10 degree, SGI (2x), IBM (1.25x), long vector gets 50% peak on

Fujitsu

• HYPOP Model Development Treat purely Lagrangian dynamics of constant-mass layers as they

inflate and deflate in regions intersecting bottom topography Pressure gradient is split into a 'baroclinic' part that vanishes and a

'barotropic' part that does not vanish when the density is uniform Comparison of surface height in Lagrangian and

Eulerian vertical after 400 baroclinic steps

old new

Total 115s 55s

Baroclinic 93s 38s

Barotropic 9s 7s


Sea Ice ModelJ. Schramm, P. Jones, W. Lipscomb

• Incremental Remapping for Sea Ice and Ocean Transport Incremental remapping scheme that proved to be three times faster than MPDATA,

total model speedup of about 30% --added to CCSM/CSIM Cache and vector optimizations CICE3.0 restructered for vector Community Sea Ice Model

• Sensitivity analysis and parameter tuning test of the CICE code Automatic Differentiation (AD)-generated derivative code Major modeling parameters that control the sea ice thickness computation were the

ice-albedo constants, densities and emissivities of ice and snow, and salinity constant Parameter tuning experiment with gradient information


Atmospheric ChemistryP. Cameron-Smith, J. Taylor, D. Erickson, J.F. Lamarque, S. Walters, D. Rotman

• Gas-phase chemistry with emissions, deposition, transport and photo-chemical reactions for 89 species.

• Experiments performed with 4x5 degree Fvcore – ozone concentration at 800hPa for selected stations (ppmv)

• Mechanism development with IMPACT A) Small mechanism (TS4), using the ozone field it generates for photolysis rates. B) Small mechanism (TS4), using an ozone climatology for photolysis rates. C) Full mechanism (TS2), using the ozone field it generates for photolysis rates.

Zonal mean Ozone, Ratio A/C

Zonal mean Ozone, Ratio B/C


Ocean BiogeochemistryS. Elliot, S. Chu, M. Maltrud

•Iron Enrichment in the Parallel Ocean Program•Surface chlorophyll distributions in POPfor 1996 La Niña and 1997 El Niño


Global DMS Flux from the Ocean using POPD. Erickson, J. Hernandez, M. Maltrud, S. Chu

The global flux of DMS from the ocean to the atmosphere is shown as an annual mean. The globally integrated flux of DMS from the ocean to the atmosphere is 23.8 Tg S yr-1 .


Toward Regional Impacts

•Snow pack in Western U.S. reduced by greater than 50% by mid-century•Higher likelihood of wintertime flooding along Cascades and Sierras•Increasing model resolution•Cluster analysis of PCM results


Resolution and PrecipitationPhil Duffy

CCM3 extreme precipitation events depend on model resolution. Here we are using as a measure of extreme precipitation events the 99th percentile daily precipitation amount. Increasing resolution helps the CCM3 reproduce this measure of extreme daily precipitation events.

(DJF) precipitation in the California region in 5 simulations, plus observations. The 5 simulations are: CCM3 at T42 (300 km), CCM3 at T85 (150 km) , CCM3 at T170 (75 km), CCM3 at T239 (50 km), and CAM2 with FV dycore at 0.4 x 0.5 deg.


Subgrid Orography SchemeSteve Ghan, Tim Shippert

• Reproduces orographic signature without increasing dynamic resolution

• Realisitic precipitation, snowcover, runoff

• Month of March simulated with CCSM


Why Climate Prediction is Compute Limited

• Long time integrations: Historical validation 1870-2000 Future scenarios 2000-2200

• Comprehensive, coupled processes Models still under development Nonlinear feedbacks and

sensitivities

• Multi-scale interactions• Need for ensemble forecasts• Decision support scenarios


Leadership Class Computing

• Will enable Additional atmospheric chemistry

Tropospheric Stratospheric

Interactive land and biogeochemistry

Comprehensive carbon cycle models

Increased resolution Atm 30 km Ocn 1/10 degree Lnd 1 km

Better throughput for coupled models


Effect of Earth Simulator onHardware and Software Issues

• Challenges assumptions Capability computing versus capacity computing

“software is the issue” Any code can be made to run

fast on any machine. If not, change the algorithm.

Special purpose processors and vector supercomputers have run out of steam

Price - performance ratio Mass market business model.

• Assertions– Vector versus cache is not

the issue

– Effective bandwidth and latency of memory subsystem and interconnect are key

– Performance portability among platforms is possible

– High percentage of peak indicates a balanced system


Modeling Resource Requirements

2003

Global Coupled Models Current

Model years/day 8

Storage (TB/century) 1

At current scientific complexity a century requires 12.5 days to simulate.

Single researcher transfers 80Gb/day. Requires 30TB storage for year.

Machine and Data Requirements

3

6.6

14.5

31.9

70.3

154

340.1

750

1

2.2

4.8

10.6

23.4

51.5

113.3

250

1

10

100

1000

dyn

veg

trop

chem

istry

biogeo

chem

inter

active

strat c

hem

eddy

reso

lv

cloud

reso

lv

Years

Tflops

Tbytes


How Many Terabytes of Output? • PCMDI Archive contains >3600

years of coupled simulation • IPCC simulations beginning


Infrastructure Challenges• Integration with existing tools• Extension of existing tools for multiple component

climate system models• Speed of transfers• Parallelism: I/O(NetCDF) and analysis• Metadata and Grid databases• Analysis of ensembles• Advanced statistical methods for analysis of

coupled systems.

CCC(2100)Current (VEMAP)

UKMO Hadley(2100)


Teams Using Multiple Centers

• DOE Supports Analysis and Archiving of Climate Data at PCMDI.

NASA DAO

PCMDI NCAR

GFDLLANL

ORNL-CCS

LANL

NERSC

NCDC

CoreUniversities



The End


Goals of the Consortium

• Performance portability for the CCSM

• Open software design process

• Layered software architecture to insulate modeling

• Readiness for global to regional climate change simulations

• High fidelity ocean and ice models

• Extension of atmospheric chemistry capability

• Development of biogeochemical, terrestrial and hydrological aspects of the CCSM

• Towards comprehensive coupled climate simulations for study of decadal to century climate change.


Tools

Documents

O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY 1 Global (and Regional) Climate Modeling Overview What is the climate system? How is it modeled?