Technical details on how to use NICAMTechnical details on how to use NICAM

Chihiro KodamaResearch Institute for Global Change (RIGC)

Japan Agency for Marine-Earth Science and Technology (JAMSTEC)

22Oct2009 COLA seminar

2

Contents of my talkContents of my talk

• How to use NICAM (technical details)– compile

– pre-processes (horizontal grid, external condition, initial condition)

– run (configuration)

– post-processes (lat-lon data)

– trouble shooting (runtime error)

• Following tutorial package (updated 2009.10.16) – http://nicam.jp/tutorial/ (password needed)

• Please feel free to ask questions!

3

Compile NICAMCompile NICAM

% ln –s Mkinclude.ES2 Mkinclude <- link Mkinclude.[arch] % emacs Mkinclude <- change $TOPPATH to NICAM-dir. % make

NOTE: Please mind that unit length of record must be 1 byte. In addition, input/output files are big-endian.

Intel ifort: -assume byterecl (compiler option) setenv F_UFMTENDIAN big

Earth Simulator 2: setenv F_RECLUNIT BYTE

TOPPATH: NICAM source code path

4

Managing Information Files (mnginfo)Managing Information Files (mnginfo)

• They contain:– number of regions

– linkages between one and other regions

– number of processes

– process-region mapping (Which process control a region?)

• mkmnginfo.cnf&MKMNGINFO rlevel = 5, prc_num = 320, output_fname = ‘rl5-prc320.info’,/

% ${TOPPATH}/bin/mkmnginfo

necessary to run MPI-based program with 320 processes, rlevel=5.

Concept of rlevel will be explained later.

5

Concept of Horizontal GridConcept of Horizontal Grid• What is “glevel” ?

– Total Grid Division Level

Figures from http://www.ccsr.u-tokyo.ac.jp/~satoh/nicam/ico.html

g: glevelnumber of rhomboses = 10×4g

S: Earth Surface [m2]resolution ~ sqrt [S / (10×4g)]

glevel 07141 km

glevel 13570 km

glevel 21785 km

glevel 3893 km

glevel 4446 km

glevel 5223 km

glevel resolution 6 112 km 7 56 km 8 28 km 9 14 km 10 7 km 11 3.5 km

6

Concept of Horizontal GridConcept of Horizontal Grid• What is “rlevel” ?

– Region Division Level– Designed for MPI program

– File name: var-name.rgn????? e.g. sst.rgn00000 ~ sst.rgn10239– Note: As rlevel increases, total memory consumption is increased, especially when

glevel-rlevel is small. • due to overlapped grid along edge of the region.

rlevel 010 regions / globe

rlevel # of regions 0 10 1 40 2 160 3 640 4 2560 5 10240

If using 20 processes,128 regions per process

rlevel 140 regions / globe

regionregion

7

Flow of Horizontal Grid GenerationFlow of Horizontal Grid Generation

mkgrid

mkgcgrid

splitgrid

_grid.rgn????? (for rlevel=0)

_grid.rgn????? (for rlevel > 0)

Horizontal Grid Filesgrid.rgn?????

provided in the package

increase rlevel

modify grid (gravitational-centered)

if rlevel=0

generate grid

8

Generating Horizontal Grid (1/2) splitgridGenerating Horizontal Grid (1/2) splitgrid

• splitgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 0, vlayer = 1, rgnmngfname ='../../mnginfo/data/rl00-prc05.info', /&SPLITGRIDPARAM input_base = '../../../NICAM_DATABASE/_hgrid/gl10/rl00/grid', output_base = 'data/_grid', alt_rlevel = 5,/

% mpirun -np 5 ${TOPPATH}/bin/splitgrid

Output: _grid.rgn00000, _grid.rgn00001, ..., _grid.rgn10239

9

Generating Horizontal Grid (2/2) mkgcgridGenerating Horizontal Grid (2/2) mkgcgrid

• mkgcgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&MKGCGRIDPARAM input_base = '../splitgrid/data/_grid', output_base = 'data/grid',/

% mpirun -np 5 ${TOPPATH}/bin/mkgcgrid

Output: grid.rgn00000, grid.rgn00001, ..., grid.rgn10239

10

External Condition Files (external_data/)External Condition Files (external_data/)• MATSIRO/

– topog: topography

– topog_std: standard deviation of topography

– albedo_*: surface albedo

– gradz: tangent of mean surface slope

– lai: leaf area index

– slidx: soil index

– veget_mat: vegetation index

• O3/ : ozone• OCN/ (for nudging slab ocean)

– sst: sea surface temperature [K]

– ice: sea ice mass [kg/m2]• ice is NEITHER sea ice concentration NOR sea ice fraction.

MATSIRO : Minimal Advanced Treatments of Surface Interaction and RunOff

11

Topography Files (MATSIRO/)Topography Files (MATSIRO/)

• To avoid numerical instability, use topography files included in the tutorial package.– For higher rlevel run, split the above files using cnvrlev,

which increases rlevel of general data

– cnvrlev.cnf&ADMPARAM ... /&SPLITGRIDPARAM num_data = 1, input_base = '../../NICAM_DATABASE/sfcdata/gl10/rl04/topog', input_direct = .false., output_base = 'data/topog', output_direct = .false., output_rlevel = 5,/

% mpirun -np 5 ${TOPPATH}/bin/cnvrlev

Output: topog.rgn00000, topog.rgn00001, ..., topog.rgn10239

12

Other MATSIRO Files / O3 FilesOther MATSIRO Files / O3 Files

• See tutorial package for details. Probably, all you have to do is modify glevel & rlevel.– mklanddata2.cnf

– mko3.cnf

13

OCN Files (1/2)OCN Files (1/2)

• SST: sea surface temperature [K]• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../dataset/oisst_v2/sst.grd', output_sst_base = './data/sst', num_data = 8, data_periodic = .false., imax = 360, jmax = 180, slat = 89.5, slon = 0.5, dlat = 1.0, dlon = 1.0, opt_southward = .false., direct_access = .true.,/

% mpirun -np 5 ${TOPPATH}/bin/mkgisst

14

OCN Files (2/2)OCN Files (2/2)

• ICE: sea ice mass [kg/m2]– In our previous package, sea ice fraction is used. It’s

wrong!

• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../NICAM_DATABASE/rawdata/hadsst1979-1999_clm.dat', ice_lldata_fname = '../../../NICAM_DATABASE/rawdata/cmip3sit1979-1999_clm.dat', output_sst_base = './data/hadsst1979-1999_clm', output_ice_base = './data/cmip3ice1979-1999_clm', num_data = 12, data_periodic = .false., direct_access = .true., offset = 0.0, ice_fact = 1000.0,/

NOTE: SST above (HadSST) is used only for modifying ice.

15

Initial Condition Files (1/2)Initial Condition Files (1/2)

• Atmosphere– 3D velocity– temperature– water vapor mixing ratio– cloud water mixing ratio

• Ocean (for slab ocean)– SST– sea ice fraction & mass & surface temperature– snow on sea ice

• Land (for MATSIRO)– soil moisture & temperature– surface accumulated snow

Most variables can be obtained from NCEP Final Analysis

16

Initial Condition Files (2/2)Initial Condition Files (2/2)• Atmosphere

– mkinit_atm.cnf&MKINIT_ATM_PARAM glevel = 10, rlevel = 5, rain_type = 'COLD', plus_tke = .true., hgrid_basename = '../../../NICAM_DATABASE/hgrid/gl10/rl05/grid.', vgrid_filename = '../../grid/vgrid/data/vgrid40.dat', topo_basename = '../../../NICAM_DATABASE/sfcdata/gl10/rl05/topog', ncep_atm_filename = '../atm.dat', init_atm_basename = 'init_atm',/

After you prepare atm.dat (see tutorial for data format),

% ${TOPPATH}/bin/mkinit_atm

Preparing ocean and land conditions is similar.

17

Prep. for Icosahedral Data Prep. for Icosahedral Data Lat-Lon Data Lat-Lon Data• Lat-Lon mapping files (llmap) are necessary

– mkllmap.cnf&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&GRDPARAM hgrid_fname = '../../../NICAM_DATABASE/hgrid/gl10/rl04/grid‘,/&LATLONPARAM imax = 5120, jmax = 2560,/&MKLLMAP_PARAM output_dir = './data/‘,/

% mpirun -np 5 ${TOPPATH}/bin/mkllmap

Output: llmap.rgn00000, llmap.rgn00001, ..., llmap.rgn10239

18

Final Step for NICAM simulationFinal Step for NICAM simulation

• You have to prepare nhm_driver.cnf.• See tutorial package for more detailed descriptions of

nhm_driver.cnf

% mpirun -np 320 ${TOPPATH}/bin/nhm_driver

19

nhm_driver.cnf (Run Type Settings)nhm_driver.cnf (Run Type Settings)&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'COLD', NDIFF_LOCATION = 'IN_LARGE_STEP2' RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'NSW6', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'NONE', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 1, SFC_DIV_NUM = 1,/

For high resolution run For low resolution run

&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'CLOUD_PARAM', NDIFF_LOCATION = 'IN_LARGE_STEP2', RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'LSC', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'PAS', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 24, SFC_DIV_NUM = 12,/

20

nhm_driver.cnf (Time Step Control)nhm_driver.cnf (Time Step Control)&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.

start data/time

Δt [s]

number of time step

21

nhm_driver.cnf (Initial Conditions 1/2)nhm_driver.cnf (Initial Conditions 1/2)&RESTARTPARAM input_basename = '../../initial_data/ATM/init_atm' input_direct_access = .true., output_direct_access = .true., trc_vmax_input = 3,/##### <--- when restart, delete “TRC_VMAX_INPUT=3”,&NM_OCEAN_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/OCN/init_ocn', merged_output_restfname = 'restart_ocn',/&NM_LAND_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/LND/init_lnd', merged_output_restfname = 'restart_lnd', opt_albsfc_in = .false., opt_albsfc_out = .true.,/###### <--- When restart, opt_albsfc_in = .true.

Atmosphere

Ocean

Land

22

nhm_driver.cnf (Initial Conditions 2/2)nhm_driver.cnf (Initial Conditions 2/2)

• When you restart run,– &RESTARTPARAM

• change input_basename• delete trc_vmax_input

– &NM_OCEAN_INIT• change merged_input_restfname

– &NM_LAND_INIT• change merged_input_restfname• let opt_albsfc_in = .true.

– &DIAGVARPARAM• set tb_fname

&DIAGVARPARAM tb_fname = 'NONE', input_direct_access = .true., output_direct_access = .true., /###### <--- When restart, set TB_FNAME (restart_diagvar)

Normally, no initial conditions for turbulence

Turbulence

23

nhm_driver.cnf (External Conditions 1/2)nhm_driver.cnf (External Conditions 1/2)• Case 1: Time invariant field

– e.g. ‘SLIDX’, ‘GRTANS’, ‘GRALB’, ‘GRALBN’

• Case 2: Monthly-constant field– e.g. ‘OZONE’

&NM_EXTDATA dataname ='OZONE', fname = '../../external_data/O3/data/amip2o3', num_of_data = 12, opt_monthly_cnst = .true., layer_type = 'ATM',/

&NM_EXTDATA dataname = 'GRTANS', fname = '../../external_data/MATSIRO/data/gradz/gradz', num_of_data = 1, layer_type = 'SFC',/

24

nhm_driver.cnf (External Conditions 2/2)nhm_driver.cnf (External Conditions 2/2)• Case 3: Step-by-step field (with time interpolation)

– e.g. ‘SST’, ‘ICE’, ‘GRLAI’

&NM_EXTDATA dataname = 'SST', fname = '../../external_data/OCN/data/sst', num_of_data = 8, data_date(:,1) = 2004, 5,26,0,0,0, data_date(:,2) = 2004, 6,02,0,0,0, data_date(:,3) = 2004, 6,09,0,0,0, data_date(:,4) = 2004, 6,16,0,0,0, data_date(:,5) = 2004, 6,23,0,0,0, data_date(:,6) = 2004, 6,30,0,0,0, data_date(:,7) = 2004, 7,07,0,0,0, data_date(:,8) = 2004, 7,14,0,0,0, layer_type = 'SFC', opt_periodic_year = .false.,/###### <--- PLEASE CHECK THE DATES BEFORE RUNNING!!!

25

nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)

• ktype = ‘3D’, ‘2D’, ‘GO’ (ocean), ‘GL’(land), ‘ISCCP’

&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360,/&NMHIST ktype = ‘2D’, item = ‘sl_slp’, file = ‘sa_slp’, output_type = ‘AVERAGE’, step = 120,/

variable namefile name ( ms_tem.rgn?????)

‘SNAPSHOT’or

‘AVERAGE’

timestep interval of outputif DTL=30s, step = 120: 1 hourly step = 360: 3 hourly

26

nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)

• &NMHISD: default setting of &NMHIST

• Output with pressure levels

&NMHISD direct_access = .TRUE., no_vintrpl = .false.,/

.true. for z* coordinate

.false. for z coordinate

&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360, out_prelev = .true.,/

&NMHISD... npreslev = 23,   pres_levs = 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 10, 7, 5, 3, 2, 1, 0.4,/

27

Run NICAM !Run NICAM !

• See tutorial package for more detailed descriptions of nhm_driver.cnf

% mpirun -np 320 ${TOPPATH}/bin/nhm_driver

Output: ms_tem.rgn?????, sa_slp.rgn?????, .....(huge number of files are created)

28

Icosahedral Data Icosahedral Data Lat-Lon Data Lat-Lon Data

• ico2ll.cnf&ICO2LL_PARAM glevel = 10, rlevel = 5, input_dir = './icodata/', output_dir = './lldata/' info_fname = 'history.info', llmap_dir = '../../grid/llmap/data/' llmap_base = 'llmap', lon_swap = .true., input_size = 4, init_year = 2004, init_month = 6, init_day = 1,/

% ${TOPPATH}/bin/ico2ll

automatically generated by nhm_driver

Output: ms_tem.ctl, ms_tem.grd, sa_slp.ctl, sa_slp.grd, .....

29

Restarting RunRestarting Run

• Followings should be changed in nhm_driver.cnf:– &TIMEPARAM: start_*

– External condition• &RESTARTPARAM

• &NM_OCEAN_INIT

• &NM_LANDINIT

• &DIAGVARPARAM

30

Changing GlevelChanging Glevel

• Followings should be changed in nhm_driver.cnf:– &RUNCONFPARAM (e.g. cloud resolving / cloud param.)

– &ADMPARAM• glevel, rlevel, rgnmngfname

– &NUMFILTERPARAM (as glevel increase by 1,)• Kh_coef_maxlim , Kh_coef_minlim, alpha_d : divided by 8

• gamma_h_lap1 : divided by 2

– &TIMEPARAM• dtl, lstep_max (if necessary)

– &NM_RD_TIME• TINTV= 10 mins for gl9, but 5 mins for gl10 and 11

– File names

31

Handling Runtime ErrorHandling Runtime Error

• Model bug? Configuration mistake? Numerical instability?• Model run diagnosis

– msg.????? : message from NICAM– standard out/err

• Check List– Where is the run fail? What module/subroutine/line?– Are the prescribed files such as external_data and initial_data correct?

• check them using ico2ll• unit of variables• glevel/rlevel• file name, little/big endian

– compiler option (such as “-assume byterecl” in intel ifort)– memory enough for the run?– parameters in nhm_driver.cnf correct?– Does the error also occur in the lower glevel/rlevel?

If the runtime error seems to occur due to numerical instability, ...

32

Numerical Filter in nhm_driver.cnfNumerical Filter in nhm_driver.cnf&NUMFILTERPARAM dep_hgrid = .false. DEEP_EFFECT = .false., hdiff_fact_rho = 0.01D0, hdiff_fact_q = 0.0d0,

hdiff_type = 'NONLINEAR1', cfact = 2.0D0, Kh_coef_maxlim = 3.125D+12, Kh_coef_minlim = 3.125D+11, ZD_hdiff_nl = 20000D0,

gamma_h_lap1 = 6.25D+5, ZD_hdiff_lap1 = 20000.0D0,

divdamp_type = 'DIRECT', lap_order_divdamp = 2, alpha_d = 3.125D+11, alpha_dv = 0.00D0,

gamma_v = 0.0D0,

ZD = 25000.0d0, alpha_r = 0.0D0,/

horizontal diffusion

Laplacian-type horizontal diffusion

3D divergence damping

vertical diffusion

Rayleigh friction

In addition, 2D divergence damping

33

Other Configurations in nhm_driver.cnfOther Configurations in nhm_driver.cnf

&nm_tb_my2moist ICE = .true., LENGTH_TYPE = '3KM', NU_MIN = 0.1D0, KH_MIN = 0.1D0, N2max = 0.0016, /##### <--- If unstable, reduce the N2max.

&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.

34

Adjusting Initial ConditionsAdjusting Initial Conditions

• Tool from Tatsuya Seiki (CCSR, Univ. Tokyo)– modify pressure to achieve hydrostatic balance

– modify vertical velocity using continuity equation

35

SummarySummary

• Please inform us if you find bugs, misleading points, suggestions, etc in NICAM tutorial package.

Thank you

36

37

##### Appendix / Memo ########## Appendix / Memo #####

38

0. Who am I ?0. Who am I ?

• Research interest– storm track

– stratosphere dynamics

– global warming

– role of cloud in climate system

• Misc– GrADS script library

39

Memory Consumption as rlevel increaseMemory Consumption as rlevel increase

40

Generating Vertical GridGenerating Vertical Grid

• mkvlayer.cnf&MKVLAYER_CNF num_of_layer = 40, layer_type = 'GIVEN', infname = './vgrid40.txt', outfname = './data/vgrid40.dat',/

0.000000000000000E+000161.683000000000335.958000000000523.806000000000726.285000000000944.5340000000001179.781000000001433.349000000001706.667000000002001.272000000002318.822000000002661.104000000003030.045000000003427.720000000003856.368000000004318.400000000004816.418000000005353.223000000005931.837000000006555.516000000007227.769000000007952.380000000008733.428000000009575.3060000000010482.753000000011460.876000000012515.179000000013651.596000000014876.521000000016196.850000000017620.010000000019154.012000000020807.488000000022589.744000000024510.811000000026581.496000000028813.457000000031219.252000000033812.418000000036607.555000000039620.3830000000

TODO: how to create refstateorNo explanation of vertical grid

41

refstaterefstate

42

nhm_driver.cnfnhm_driver.cnf&nm_rd_time

TINTV=10.0D0,TUNIT='MIN'

/###### <--- TINTV = 10min for gl9, but 5min for gl10 and 11.

43

nhm_driver.cnf (Resolution, Grid)nhm_driver.cnf (Resolution, Grid)

&ADMPARAM glevel = 10, rlevel = 5, vlayer = 40, rgnmngfname = '../../mnginfo/data/rl05-prc320.info',/&GRDPARAM hgrid_fname = '../../grid/hgrid/data/grid', vgrid_fname = '../../grid/vgrid/data/vgrid40.dat', topo_fname = '../../external_data/MATSIRO/data/topog/topog', toposd_fname = '../../external_data/MATSIRO/data/topog_std/topog_std', vegeindex_fname = '../../external_data/MATSIRO/data/veget_mat/veget_mat'/

44

nhm_driver.cnf (Ocean)nhm_driver.cnf (Ocean) &nm_ocean_mixedlayer osstfx = .false.,

oicefx = .false.,oicrfx = .false.,TNUGST = 432000.0D0,TNUGSI = 432000.0D0,

/###### <--- The relaxation times of SST and Sea ICE are 5 days.