GFESuite Configuration Mark Mathewson. Sept. 24-27, 2002GFESuite Configuration2 Topics Documentation Directory Structure Base/Site Configuration Base/Site/User

GFESuite ConfigurationGFESuite Configuration

Mark Mathewson

Sept. 24-27, 2002 GFESuite Configuration 2

TopicsTopics

DocumentationDirectory StructureBase/Site ConfigurationBase/Site/User configurationProblem SolvingMisc. Items


DocumentationDocumentation


Where is it?Where is it?

GFE HELP menuFSL AWIPS doc web site

http://www-md.fsl.noaa.gov/eft/AWIPSdoc.html

Many other regional/national web sites contain information

listservers another source ndfd_team, ifpsrppinfo, ifps


What is available?What is available?

User Training GuidesReference ManualsConfiguration GuidesProduct Generation GuidesREADME file


How good is it?How good is it?Issues:

Little indexing available…. High priority for accurate info. Lower priority for indexing than s/w

enhancementsGenerally fairly accurate – especially the

configuration guidesSpend a few minutes with it – it will save

you time.


Directory StructuresDirectory Structures


Top-LevelTop-Level

Serverconfiguration

documentation

examples

Run scripts


databases Structuredatabases Structure

KEEP OUT!


databases Structuredatabases Structure


Database grids StructureDatabase grids Structure


GFE files on HPUXGFE files on HPUX


Directory TipsDirectory Tips

Do not modify ANY files in etc/BASE Use etc/SITE instead

Do not modify ANY files under data/databases These are controlled by the ifpServer

README file contains list of directories that will be “purged” during installation


BASE/SITE ConceptsBASE/SITE Concepts

ifpServer Configurability


Base/Site conceptBase/Site concept

Applies to /etc/BASE, /etc/SITE Used to preserve YOUR changes from upgrade to

upgrade.

BASE file SITE file Purpose

serverConfig.py localConfig.py General server overrides

serverConfig.py localWxConfig.py Wx configuration override

Maps.py, MapFiles.py

localMaps.py, localMapFiles.py

Map background override

dfcConfig.py localDfcConfig.py Daily Forecast Critique override

MapFiles – new in IFPS12



Use favorite text editor to change files.Normally have to restart ifpServer after

changing files.Use runIFPServer –n to check for errors.If server doesn’t come up, check for errors.

Look through log files.



Configuration files are Python-executable code

Very particular to syntax.Local (e.g., SITE) overrides using Python

importing Can be tricky


Server Configuration Server Configuration OverridesOverrides

serverConfig.py

Maps.py

localConfig.py

Maps.py

SITEBASE

Python Path:SITE, then BASE

CopiedThenmodified

importPartialoverrides

CompleteOverride (don’t do)


Python “dual” ImportingPython “dual” ImportingserverConfig

TC6NG = (0*HOUR,6*HOUR,6*HOUR)

DFC_UPDATE_INTERVAL = 15

Dbs = DBS + localConfig.parms

Eta = (ETA, TC6NG)

import localConfig

localConfig

serverConfig.TC6NG = (1*HOUR,…)TC6NG = …

parms = [([MaxT], TC6NG)]

parms = [([QPF], TC6NG)]


import serverConfigfrom serverConfig import *




import localConfig

localConfig



Result?DFC_UPDATE_INTERVAL = ???

start


Python “dual” ImportingPython “dual” Importing

import serverConfig Allows you to access variables/methods from the

module you import, via: serverConfig.varName from serverConfig import *

Makes a copy of the variables, you can access the copies of these variables directly, via: varName, but these are COPIES. If you change them, the originals aren’t changed.

Exceptions: dictionaries, arrays Don’t require serverConfig. in front of variable.



TC6NG = (0,6,6)MaxT = (“MaxT”, SCALAR, "F", …)

Dbs = DBS + localConfig.parms

MaxT_PG = (MaxT, TC6NG)

import localConfig

localConfig

serverConfig.TC6NG = (1,6,6)TC6NG = (2,6,6)

parms = [([MaxT], TC6NG)]

parms = [([QPF], TC6NG)]


What is TC6NG at this point?What is the TC for MaxT at this point? Two parm

lines



GOTCHA – Python files .py .pyc .pyo

If you delete the .py file, such as localConfig.py, you may still have *pyc or *pyo files. The system will use those and will behave as if you never deleted the .py file.


localConfig capabilitieslocalConfig capabilities

Adding new weather elementsAdding new modelsChanging characteristics (# of versions)Changing time constraints for elementsChanging domain/resolution


localMaps/localMapFileslocalMaps/localMapFiles

Overrides Maps.py and MapFiles.py Add new maps Remove existing maps Modify characteristics of maps

FiltersAutomatic edit areas

MapFiles/localMapFiles (new IFPS12) Defines the names of the shapefiles


Changing map backgrounds – Changing map backgrounds – filteringfiltering

Can filter by shapefile attributes. Make a new map just for ifpIMAGE generation.

CWAzones.filter(lambda x : x['CWA'] == CWA)

def cwaZoneFilt(x): return x['CWA'] == CWA CWAzones.filter(cwaZoneFilt)

ShapefileAttribute

Identical Filter


Changing map backgrounds - Changing map backgrounds - filteringfiltering

def exampleFilt(x): myCounties = ['Summit', 'Sandusky', 'Huron', 'Medina'] return x['COUNTYNAME'] in myCounties and \ x['STATE'] == "OH" and x['CWA'] = 'CLE'

excludedCities = [‘Clinton’, ‘Pearl’, ‘Brandon’]cities.filter(lambda x,y=excludedCities: x[‘POP_1990’] > 10000 \ and x[‘NAME’] not in y

Note:


Updating Map ShapefilesUpdating Map Shapefiles

Download new shapefiles from NWSHgzip each component:

*.dbf, *.shp, *.shxPlace into /awips/GFESuite/data/mapsIf the name changed, add/edit

localMapFiles.py [IFPS12] to point to the new file

Caution: shapefiles replaced during upgrade


localWxConfiglocalWxConfig

Similar to localConfig, except for the Weather Definition

Uses: Adding additional attributes Adding additional weather types Adding additional coverage/prob, intensities

May make incompatible grids with other NWS offices.


Tips (and Rules)Tips (and Rules)

DON’T edit files in etc/BASE They are wiped clean each upgrade

Check your changes immediately: runIFPServer –n (even with server running)

Some changes may not be compatible with IFPS Time constraints, domains/resolution


BASE/SITE/USER ConceptsBASE/SITE/USER Concepts

GFE / ifpIMAGE Configurability


Base/Site/User conceptBase/Site/User concept

Applies to: gfe and ifpIMAGE configuration files Sample sets, edit areas Color tables Smart Tools/Procedures User-defined time ranges Weather Element Groups


Tips (and Rules)Tips (and Rules) Just like etc/BASE, do not edit files in

data/databases/BASE since they are wiped clean during the next upgrade.

Do not edit files directly in the databases directory. Some files require associated metadata files that can’t

be created with a text editor Changing files can confuse the ifpServer and GFEs if

they are running at the time. Use the GFE to edit files, OR use the ifpServerText

program.


““Special” UsersSpecial” Users

SITE – site administrator login. Files in this category are seen by all users.

BASE – brings up GFE using BASE configuration only --- as supplied by FSL. Good way to verify a problem is a local configuration problem.

“BASE” and “SITE” will not appear in the GFE startup dialog.


Python Errors…GFE is hung!Python Errors…GFE is hung!

FSL

Not any more…. Starting GFE witha bad configuration file will cause adialog to popup showing you theerror location.


Why Can’t I find my config Why Can’t I find my config filesfiles

Are your config files lost?Perhaps you created them under a different

user?Use ifpServerStats to help you find them.

ifpIMAGE uses the user SITE for execution – you need to create files in correct place.


I changed my config file, but I changed my config file, but the change isn’t working…the change isn’t working…

gfeConfig BASE

3. Later, log onto awipsusr,the BOUsiteConfig “seen” doesn’thave the latest changes.

BOUsiteConfig awipsusr (or other user)

1. User awipsusr createsBOUsiteConfig, to test out thenew configuration.

BOUsiteConfig SITE

2. Later, the same user logsonto user SITE and copiesthe awipsusr BOUsiteConfig using ifpServerText. Makes another change to it.

import


The “READ-ONLY” flagThe “READ-ONLY” flag

Setting the Read-Only flag on the various configuration files will stop the user from overriding them.

No GFE GUI to set the read-only flag.Must do it directly on the file system.

chmod 444 filename

Fully implemented with IFPS12


Some sites have…Some sites have…

Made copies of files in BASE and placed them in SITE Copied lots of files from SITE to “user” This practice is NOT recommended. You

might get different behavior based on login name.

BASE SITE user

gfeConfig gfeConfig

50 tools 50 tools

copied

copied


Concepts to understandConcepts to understand

Python files (gfeConfig) Can use Python importer for partial overrides

Non-Python files (sample sets) Must use complete overrides, no importer capability


Order of PrecedenceOrder of Precedence

ERROR

no SITE File

Exists?

yes

no

Return SITE File

BASE File

Exists?

no

yes

Return BASE File

USER File

Exists?

Request File

yes

Return USER File


Retrieval of Color Table Retrieval of Color Table ExampleExample

QPFColorTable

BASE

QPFColorTable

SITE

QPFColorTable

USER

X X


Some files are PythonSome files are Python

You can use the Python “import” mechanism and you should.

Advantage: You can override small pieces of files. You don’t need to redefine everything

Examples: Smart Tools, gfe/ifpIMAGE configuration files


Python importingPython importing

A B C

import B

codecodecode

import A

codecodecode

codecodecode


Python importingPython importing

C C

import C

codecodecode

codecodecode

XCan’t import from “yourself”, even if at different BASE/SITE/USER levels


Python-file RetrievalPython-file Retrieval


GFE Config file exampleGFE Config file example

…T_defaultColorTable=‘midRange’ImageOnActive=yes…

USER (joeConfig)

import gfeConfigT_defaultColorTable=‘lowRange’…

SITE (siteGfeConfig)

Site could also have redefined the color table

import siteGfeConfigT_defaultColorTable=‘myOwn’ImageOnActive=noContour_font=3T_spatialImageType = [‘Image’, ‘Contour’]…

User could also have imported directly from the base.

BASE (gfeConfig)


TipsTips

Software doesn’t seem to be working. Start GFE as user “BASE”, config “gfeConfig”

Run ifpServerStats and look at your environment.

Move files around if needed using moveGFEData.


Misc. TopicsMisc. Topics


Installs/UpgradesInstalls/Upgrades

SITE and USER directories not touched by installs.

BASE files overwritten. Recommend you do partial overrides, not

complete files copies in order to ensure operability of software and to catch latest changes in the upgrade.

Watch out for directories that are overwritten! README file.


Product Generation Product Generation

Site/base/user concept applies to all programs using the server.

By Default, many product generation programs become user “SITE”.

Prevents user overrides from affecting site’s products

Can start programs with specific user switch (-u username).


Watch your localConfig Watch your localConfig configuration changes….configuration changes….

Some changes may “drop” your database. Changing domains, resolutions, data type

Some changes only “drop” a single weather element. Changing the Weather definition

Some changes may shift around the grids in time. Changing time constraints


Problems…and how to Problems…and how to diagnose themdiagnose them

Each program uses logfiles. Located in /awips/GFESuite/data/logfiles

Search logfiles for “PROBLEM”, “BUG”

Many problems can be diagnosed simply by examining the logfiles.

But some entries in logfiles are cryptic to forecasters.

Documents

GFESuite Configuration Mark Mathewson. Sept. 24-27, 2002GFESuite Configuration2 Topics Documentation Directory Structure Base/Site Configuration Base/Site/User