The 2010 Workshop on the Solution of Partial Differential Equations on theSphereThe 2010 Workshop on the Solution of Partial Differential Equations on theSphereAugust 24-27, 2010August 24-27, 2010

Alteration of vertical grid in NICAMAlteration of vertical grid in NICAMtowards the super-high resolution towards the super-high resolution

simulationssimulations

Hirofumi TOMITAHirofumi TOMITARIGC / JAMSTECRIGC / JAMSTEC

NICAM : DX=3.5kmMTSAT-1R

OutlineOutline

Introduction of NICAMIntroduction of NICAM Summary of current version of DCSummary of current version of DC

Vertical problemVertical problem Horizontal high resolution runHorizontal high resolution run

• PGF errorPGF error• Land surface model couplingLand surface model coupling

Future research plans using the next generation Future research plans using the next generation supercomputer in JAPANsupercomputer in JAPAN 10PFLOPS machine in Kobe10PFLOPS machine in Kobe

• Named “K-computer” Named “K-computer”

SummarySummary

Summary of Summary of NICAM dynamical coreNICAM dynamical core

NICAM dynamical core (1)NICAM dynamical core (1)

Horizontal gridHorizontal grid Icosahedral gridIcosahedral grid

– modification by spring modification by spring dynamicsdynamics

– Reallocation of grid to Reallocation of grid to gravitational center of control gravitational center of control volume volume

Tomita et al. 2001,2002, JCP

NICAM dynamical core (2)NICAM dynamical core (2)

Vertical gridVertical grid Terrain-following coordinateTerrain-following coordinate

• with Lorentz gridwith Lorentz grid

Governing equationGoverning equation Non-hydrostatic equationNon-hydrostatic equation

• with deep atmospherewith deep atmosphere

032/1

hh

hG

WRt

VG

V

CoriolishhhPP

tFADVGV

3

Coriollisz FADVRgG

PWt

22/12

heathh

hhh

QWgG

P

R

WPP

R

G

WhhE

t

22/123

32/1

GV

VG

V

Continuity eq.

Horiz. ,mom. eq.

Vert. ,mom. eq.

Energy. eq.

Tomita & Satoh 2004, Fluid.Dyn.Res

Satoh et al.2008 JCP

NICAM dynamical core (3)NICAM dynamical core (3)

SolverSolver Split explicit methodSplit explicit method

• Fast mode ( e.g. acoustic wave, gravity wave )Fast mode ( e.g. acoustic wave, gravity wave )

– Small time step ( forward-backward )Small time step ( forward-backward )

• Slow mode ( e.g. advection term )Slow mode ( e.g. advection term )

– Large time step ( by RK2 or RK3 )Large time step ( by RK2 or RK3 )

In slow modeIn slow mode Horizontal explicit / vertical implicit schemeHorizontal explicit / vertical implicit scheme

NICAM dynamical core (3)NICAM dynamical core (3)

Horizontal advection schemeHorizontal advection scheme Miura ( 2007, MWF ) scheme with Thurburn(1996) Miura ( 2007, MWF ) scheme with Thurburn(1996)

flux limiterflux limiter• Simple way of flux estimationSimple way of flux estimation• 22ndnd order and monotonicity. order and monotonicity.

Consistency With Continuity in Consistency With Continuity in tracer advection.tracer advection. Niwa et al. (2010)Niwa et al. (2010)

• Operator splitting into horizontal and vertical Operator splitting into horizontal and vertical directionsdirections

• Use of intermediate density (Easter(1993))Use of intermediate density (Easter(1993))

Option of other grid configuration in NICAMOption of other grid configuration in NICAM

icosahedral grid different topology

Non-uniform resolution(Iga 2010, MWR submitted )

Stretched (Tomita2008)

Vertical problem of Vertical problem of dynamical coredynamical core

at the high-resolution runsat the high-resolution runs

Numerical problemNumerical problem

Owing to steep mountainOwing to steep mountain & terrain following coordinate& terrain following coordinate In TFC, one reference state should be set in all the In TFC, one reference state should be set in all the

region.region. Limited area modelLimited area model

• No problemNo problem Global modelGlobal model

• Difference from reference state is large.Difference from reference state is large.• > 50K between tropics and polar region.> 50K between tropics and polar region.• Pressure gradient force error!Pressure gradient force error!• Remedy : setting the mid-latitude profile as the reference Remedy : setting the mid-latitude profile as the reference

state.state.– Tibetan Platau, Rocky mountain, Andes mountain are Tibetan Platau, Rocky mountain, Andes mountain are

in the mid-latitude.in the mid-latitude.

The simulation result depends on the reference state!

Sometimes, the model blow up at the high Sometimes, the model blow up at the high resolution simulation!resolution simulation! PGF error acts violently!PGF error acts violently!

• So-called “hydrostatic inconsistencySo-called “hydrostatic inconsistency

Reconsideration of vertical discretization from the Reconsideration of vertical discretization from the terrain-following coordinate to height basis terrain-following coordinate to height basis coordinate.coordinate.

– Vertical adaptive mesh for the PBL scheme.Vertical adaptive mesh for the PBL scheme.

The problem becomes severe when the resolution increases!

Problem about coupling with land model(1)Problem about coupling with land model(1)

Energy balance equation at the interfaceEnergy balance equation at the interface

Usual way:Usual way: explicit estimation of bulk coef in the surface flux : explicit estimation of bulk coef in the surface flux :

using the previous skin temperature. using the previous skin temperature. oscillatory of skin temp. oscillatory of skin temp.

LTFTFFTFTF SQSRSSRLSG )()()()(0

Ground flux

Long wave

Short wave

Sensible heat

latent heat

Typical diurnal variation of skin temperature

At dawn & suset,

bulk coef. is oscillatory!

Problem about coupling with land model(2)Problem about coupling with land model(2)

Best SolutionBest Solution Atmosphere vertical turbulence & surface energy Atmosphere vertical turbulence & surface energy

balance is implicitly solved at the same time.balance is implicitly solved at the same time. However,,,,,However,,,,,

• Modulization?Modulization?

We want to pursue the separate modules for We want to pursue the separate modules for atmosphere turbulence and land surface model.atmosphere turbulence and land surface model.

Another solutionAnother solution The bulk coefficient at The bulk coefficient at

the surface is implicitly solved!the surface is implicitly solved!

Tomita (2009, JHM)

Research plans toward the next-Research plans toward the next-generation supercomputer generation supercomputer

The next-generation supercomputer in KobeThe next-generation supercomputer in Kobe

The next-generation supercomputer system (~2012)The next-generation supercomputer system (~2012)

SystemSystem One nodes:One nodes:

• 128GFLOPS128GFLOPS• Memory band width : Memory band width :

64GB/s 64GB/s• B/F : 0.5B/F : 0.5

80k nodes/640k core80k nodes/640k core Peak performacne : Peak performacne :

~10PFLOPS ~10PFLOPS Total memory : ~1PBTotal memory : ~1PB Network 3D torusNetwork 3D torus

bi-direction 5GB/s X6 bi-direction 5GB/s X6

What can we expect on it?What can we expect on it?

Next-generation supercomputer strategic programNext-generation supercomputer strategic program--- Field 3--- Field 3 ：： Research of prediction of weather and climate Research of prediction of weather and climate contributing to disaster preventioncontributing to disaster prevention

Change of tropical cycloneChange of tropical cyclone in the future climate in the future climate• NICAM3.5km-modelNICAM3.5km-model

(7km model also) (7km model also)– IntensityIntensity– FrequencyFrequency– Local informationLocal information

• Integration time : 10 years orderIntegration time : 10 years order• More reliable statistics More reliable statistics

Predictability of tropical weatherPredictability of tropical weather• NICAM3.5km, 1.7kmNICAM3.5km, 1.7km• MJO, tropical cyclone, monsoonMJO, tropical cyclone, monsoon

– How long can we extend How long can we extend the tropical prediction by GCRM?the tropical prediction by GCRM?

Courtesy of Prof. H.Tanaka(Tsukuba Univ.)

Miura et al. 2007

Grand challenge!Grand challenge!

Global 400m mesh runGlobal 400m mesh run Collaboration project with RIKENCollaboration project with RIKEN and JAMSTECand JAMSTEC

• Horizontally 400m with 100 vertical levelsHorizontally 400m with 100 vertical levels – Truly, global cloud-resolving!Truly, global cloud-resolving!

• Use of the full computational resourceUse of the full computational resource– 1PB1PB memorymemory– All computational nodesAll computational nodes

• Integration time Integration time ：： 1 week?1 week? PurposePurpose ：：

• Improve PBL cloud?Improve PBL cloud?• Improve detail structure of cumulus?Improve detail structure of cumulus?• Diurnal cycle?Diurnal cycle?• Computationally,Computationally,

– Well work in the case of full use?Well work in the case of full use?– Suggestion to the next-next computer systems planSuggestion to the next-next computer systems plan

野田暁氏提供野田暁氏提供

How about the sustained performance?How about the sustained performance?

Vector machineVector machine Problem size:Problem size: 　　 7km/L407km/L40 ESES ：３０～４０：３０～４０ %% ES2:ES2: １０～１５１０～１５ %%

Scalar machine ( NO TUNING! )Scalar machine ( NO TUNING! ) Cray XT4 (AMD Opteron ) : Cray XT4 (AMD Opteron ) : 2.9%2.9% 　　 (@ Oct. 2009(@ Oct. 2009 ））

• collabolation with COLA & ECMWFcollabolation with COLA & ECMWF T2K tsukuba(AMD Opteron) : T2K tsukuba(AMD Opteron) : 3.3%3.3% IBM :BlueGene/P : IBM :BlueGene/P : 6%6%

• Good balance between memory access speed and CPU Good balance between memory access speed and CPU speedspeed

Recent PC : Core i7(nehalem core)Recent PC : Core i7(nehalem core) 6%6%• Problem size : 240km/L54Problem size : 240km/L54

– (owing to memory band widith:6.4GB/sX3=19.2GB/s)(owing to memory band widith:6.4GB/sX3=19.2GB/s)– B/FratioB/Fratio ：： 0.450.45

The Next Generation Machine(BF : 0.5)The Next Generation Machine(BF : 0.5) Default : 5%Default : 5% Medium tuning : 10%Medium tuning : 10% Highly tuining : 20%???Highly tuining : 20%???

SummarySummary

NICAM:NICAM: Global non-hydrostatic atmospheric modelGlobal non-hydrostatic atmospheric model Icosahedral grid Icosahedral grid

Vertical discretization:Vertical discretization: plan to change from terrain following coordinate plan to change from terrain following coordinate

to height based coordinate.to height based coordinate. Future plans:Future plans:

On the next-generation supercomputer systemOn the next-generation supercomputer system• Change of tropical cyclone in the future climateChange of tropical cyclone in the future climate• Predictability of tropical weatherPredictability of tropical weather• Grand Challenge! ( 400m global mesh with 100 levels. )Grand Challenge! ( 400m global mesh with 100 levels. )