Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C –...

Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal Tel.: (351) 21 844 7000 Fax: (351) 21 840 2370 e-mail: informacoes@meteo.pt URL: http://www.meteo.pt

MONITORING

WEATHER

CLIMATE

SEISMIC ACTIVITY

A SUSTAINABLE DEVELOPMENT AND A SAFER WORLD

EUMeTrain LSA SAF Week

Ricardo Silva, IM

Session 2 – Messing around with the data

Who am I?

Ricardo Silva

Where do you come from?

How often do you use LSA SAF products?

Occasionally

Never used, but plan to start

Regularly

Session 2 Summary

Obtaining LSA SAF products General Product Information

Documentation File structure

Quick Visualization using HDFView GIS tools for visualization and analysis

Quantum GIS GRASS GIS ILWIS

Tool for georeferencing LSA SAF products Using programming tools – Python

Obtaining LSA SAF products

Near Real Time Dissemination via EUMETCast Dissemination on-demand

http:landsaf.meteo.pt Products are available for registered users

Registration is free of charge, but may take up to 3 working days for approval

Downloads use the FTP protocol Several download methods according to desired temporal

extent and download settings

How do you obtain LSA SAF products?

LSA SAF (website, FTP, helpdesk)

Other?

EUMETCast

Obtaining LSA SAF products

Dissemination options Ordering methods

Random slot ordering The user can select individual time slots for several products Automatic dissemination

Long series ordering Ordering of long time series of products. Requires human intervention and may take up to one week

Dissemination methods LSA SAF FTP server

The order is placed in the LandSAF server for the user to download User's personal FTP server

LSA SAF's system transfers the order directly to the user's FTP server

Product Documentation

Product User Manual (PUM) Main documentation source

Physical phenomena Algorithm basis Output data specifications Etc.

Product Output Format (POF) Details concerning description of metadata, file naming

formats, etc. Validation Report (VR) Algorithm Theoretical Basis Document (ATBD)

Product file structure

LSA SAF products use the HDF5 format http://www.hdfgroup.org/

Hierarchical Data Format (HDF) Groups – Containers for datasets and other groups Datasets – Multidimensional arrays of a homogeneous type Attributes – Metadata on groups and datasets

HDF files: Are self-describing. Complex data relationships and dependencies Can contain binary data Support n-dimensional datasets

Each element in the dataset may itself be a complex object

Product file structure

LSA SAF HDF5 files: A common set of attributes on the root group

general information A dataset for the main product (LST, DSLF, etc) Additional datasets for cell metadata (Quality flags, etc.) Each dataset may have:

Specific attributes (Scaling factor, missing value, etc.) Different dimensions (Most datasets are 2D) Different data types (Most datasets are 16bit signed integers)

Quick visualization of products

HDFView software

http://www.hdfgroup.org/hdf-java-html/hdfview/index.html#download_hdfview Java-based HDF Viewer Inspection of a HDF5's structure and data Visualize data in spreadsheet or image mode Basic statistics, line plotting and histogram generation Export to image formats

Have you used HDFView before?

HDFWhat?

HDFView

1. Datasets

2.Attributes

3.Table View

4.Table options

5.Image view

6.Image options

7.Pixel values

HDFView

Inspecting file and dataset attributes

HDFView

When visualizing products take note:

1. Cell values are unscaled

2. The 'missing value' values have no physical meaning

3. Product is not georeferenced (line, column coordinates only)

GIS tools for visualization and analysis

Quantum GIS User friendly interface Many plugins created by the community

GRASS GIS Many raster analysis modules Python scripting

ILWIS Direct import of LSASAF products through the

GEONETCast toolbox Other tools (ArcGIS, Mapinfo, SAGA, etc.)

Quantum GIS

http://www.qgis.org/ Open source multi-platform GIS software User-friendly interface Reads many input formats

Raster files, Vector files, Web services, etc On-the-fly reprojection for vector and raster data Large and friendly community of users and

developers

Quantum GIS

Integration of different data sources

Quantum GIS

Quick inspection of relationships between data

GRASS GIS

Geographic Resources Analysis Support System http://grass.fbk.eu

Open source multi-platform GIS software Hundreds of modules for raster and vector

processing Reads many input formats

Can read HDF5 directly (won't be georeferenced) It is preferable to georeference the HDF5 file first

GRASS GIS

Modular and flexible user interface

Raster calculator Allows for computing a combination of logical and algebraic

expressions Example: Calculate the arithmetic mean LST values for a specific day (using

hourly sample)

24 raster files each with the LSA SAF LST product for the hourly timeslots

GRASS GIS

( a+ b+ c3

(if (isnull (a) ,0,a)+ if (isnull (b ) ,0,b)+ if (isnull (c ) ,0, c )if (isnull (a) ,0,1)+ if (isnull (b) ,0,1)+ if ( isnull (c ) ,0,1)

Approach #1: Using only cells where there is enough data for the whole day(In GRASS, any math performed with NULL cells always results in a NULL value for these cells

Approach #2: Using all the cells where there is at least one value in the day

GRASS GIS

Approach #1

GRASS GIS

Approach #2

GRASS GIS

Most commands are available in two interfaces: Graphical – Good for visualization and exploratory analysis

Command-line – Nice for automating repetitive tasks

Are you used to creating small scripts for automating tasks?

Not really

I might start using themIn the future...

Python scripting example: Automatically import and rename 191 LST files for a

given day

GRASS GIS

GRASS 6.4.1 (world4326):~ > python

>>> # example importing all the LST geotiffs into GRASS>>> import grass.script as grass>>> from glob import glob>>> import os>>> import datetime as dt>>>>>> for path in glob('/home/geo2/test_data/lsasaf/modificados/georefs/*LST*.tif'):... fName = os.path.basename(path)... nameFields=fName.split('_')... product, area = nameFields[3:5]... timeslot = dt.datetime.strptime(nameFields[5], '%Y%m%d%H%M')... newName = '_'.join((product, area, timeslot.strftime('%Y_%m_%d_%H_%M')))... grass.run_command('r.in.gdal', flags='oe', overwrite=True, input=path, output=newName)

Integrated Land and Water Information System http://52north.org/communities/ilwis/

Open Source GIS software Available only for MS Windows (as of now) Imports LSA SAF products, using the GEONETCast

Toolbox Several modules for raster and vector processing

GEONETCast Toolbox

GEONETCast Toolbox http://52north.org/communities/earth-observation/geonetcast-toolbox-software

Available as a toolbox for ILWIS Developed by ITC – University of Twente Enables easy access to and management of

GEONETCast data various satellite and environmental data and/or resulting

products Tailored for users running GEONETCast reception

stations

ILWIS/GEONETCast

Installation Follow the GEONETCast Toolbox's user manual Read section 1.1 and 1.2 CAREFULLY

Importing LSA SAF products Beware of file names:

Files obtained through GEONETCast (EUMETCast) S-LSA_-HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2 Everything will work OK

Files obtained through LSA SAF's on-demand dissemination HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2 GEONETCast Toolbox expects the 'S-LSA_-' prefix and will throw errors Solutions:

Uncompress → rename → recompress (time consuming and boring) Modify the GEONETCast's batch import scripts Contact LSA SAF's helpdesk and they'll send you the modified scripts

ILWIS/GEONETCast

Modular user interface, integrating the GEONETCast Toolbox

ILWIS/GEONETCast

Use native NGP projection of the LSA SAF products

Have you used these GIS tools before?

I might start using themIn the future...

Yes, some of them

LSA SAF Product Coordinate System

Normalized Geostationary Projection (NGP)

Projection of the underlying MSG LRIT source data

Important parameters for MSG satellite:

Satellite height above ellipsoid: 35785831 m

Central meridian: 0º

Documented on EUMETSAT's LRIT and HRIT Global Specification

http://www.eumetsat.int/groups/cps/documents/document/pdf_cgms_03.pdf

LSA SAF Product Coordinate System

LSA SAF products hold coordinate system information in the attributes of the root group

But HDF5 files have no standard way of defining coordinate systems It's hard to automatically import LSA SAF products into

a Geographical Information System (GIS) software AND have the data correctly georeferenced

Visualization of LSA SAF products

HDFView is nicefor a quick inspection

Some GIS toolscan open HDF5 directly BUT ...

GIS tools are nice forvisualization, analysis

map making, etc.

Most GIS tools cannotrecognize LSA SAF

coordinate information

The files will not beprojected

It won't be possible tooverlay different data UNLESS …

LSASAF Georeferencer Tool

A graphical tool for georeferencing LSA SAF products

Still in-development, but already usable Open development

Source code hosted on a public repository https://github.com/ricardogsilva/LSASAF-georeferencer

Uses open source tools Python, GDAL, Proj, Qt4

Multi-platform

Convert LSASAF HDF5 files to GeoTiff GeoTiff is a Tiff based interchange format for georeferenced raster

imagery

Reproject from the NGP projection to other coordinate systems

Apply scaling factor and missing value information to the output file

Output GeoTiff files are suitable for further processing in GIS software

Process multiple files simultaneously Also has a CLI interface, useful for scripting

Graphical Window usage example

1. Select the file(s) to process

2. Load file information

3. Choose the dataset(s) to be processed

4. Choose the output projection

5. Choose the output directory

6. Process the files

Command line usage example

# display usage informationricardo@geo2:~$ ./georef_hdf5.py -h

# process all the LST files in ~/input_dirricardo@geo2:~$ ./georef_hdf5.py -x -o ~/my_output_dir ~/input_dir/*LST*

Guess who is developing the LSA SAF Georeferencer Tool?

Some other dude...

Ricardo Silva(me) :)

Contributions, bug reports, etc. are very much welcome!

Programming tools – Python

Python(x, y) http://code.google.com/p/pythonxy/ A (free) scientific and engineering development software

for numerical computations, data analysis data visualization

Bundles Python and a big set of extra modules in a single package Includes modules for manipulating HDF5 files

Some useful python modules for working with LSA SAF products:

The fundamental package needed for scientific computing with Python

Provides an N-dimensional array object and specialized data types and functions

A library built to work with NumPy arrays

Provides efficient numerical routines for interpolation, statistics, linear algebra, etc.

Matplotlib

2D plotting library which produces publication quality figures

PyTables

Package for managing HDF5 files

Designed to efficiently and easily cope with extremely large amounts of data

A translator library for raster geospatial data formats.

Has read/write functionality for many raster formats (HDF4/5, GeoTiff, GRIB, netCDF, etc.)

Allows assigning georeferencing information

ricardo@geo2:~$ python >>> import tables >>> import numpy as np>>> import matplotlib.pyplot as plt >>> product = tables.openFile('path_to_HDF5_file')# list attributes of the root group>>> product.root._v_attrs # list attributes of the LST dataset>>> product.root.LST.attrs # import the LST dataset into a 32bit float numpy array>>> lst = np.array(product.root.LST, dtype=np.float32)# convert missing values to nan and apply scaling factor>>> lst[lst == product.root.LST.attrs['MISSING_VALUE']] = np.nan>>> lst = lst / product.root.LST.attrs['SCALING_FACTOR']# plot the data >>> plt.imshow(lst, interpolation='nearest')>>> plt.colorbar()>>> plt.show()

Example interactive session for manipulating a LST product

The script's output:

END of session 2

Thank you for attending!

Enjoy the rest of the LSA SAF week

Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C –...

Documents

Meteorologia aeronautica

Meteorologia Aeronáutica

WV images, Potential Vorticity and Conceptual Models Nuno Moreira (nuno.moreira@meteo.pt) Instituto de Meteorologia, Lisboa, Portugal 2003

WV images and Jet Streams Cases Nuno Moreira (nuno.moreira@meteo.pt) Instituto de Meteorologia, Lisboa, Portugal 2003

CÓDIGO DE ÉTICA E CONDUTA - ipma.pt · CÓDIGO DE ÉTICA E CONDUTA Instituto Português do Mar e da Atmosfera, I.P. Rua C - de Lisboa 1749-077 Lisboa, Portugal 3 | 8

4.º SUP 1145 (Magalhães) · 2016-03-11 · boletimmunicipal cÂmara municipal de lisboa resoluÇÕes dos ÓrgÃos do municÍpio sumÁrio sede: campo grande, 25, 1.o-b 1749-099 lisboa

Meteorologia - IAG - Instituto de Astronomia, …agg_1400200/moddata/METEOROLOGIA/2_Tempera… · Meteorologia Profa.: Rita Ynoue Disciplina: ... •Experimentos/Previsão ... propagação

RELATÓRIO TÉCNICO€¦ · RELATÓRIO JUNHO 2018 Instituto Português do Mar e da Atmosfera, I.P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal 1|35 Tel: (+351) 21 844

Resumo Climatológico Maio e Primavera de 2020...2020/06/05 · Rua C - Aeroporto de Lisboa 1749-077 Lisboa, Portugal 1 | 10 info@ipma.pt Tel: (+351) 21 844 70 00 Fax: (+351) 21 840

RELATÓRIO AGOSTO 2017...RELATÓRIO AGOSTO 2017 Instituto Português do Mar e da Atmosfera, I.P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal 2|34 Tel: (+351) 21 844

METEOROLOGIA - RESUMOaeroclubedebrasilia.org.br/aulas/PPMET_-_METEOROLOGIA_RESUMO.pdf · METEOROLOGIA - RESUMO Cap. 1: Introdução a meteorologia Meteorologia pura: E o estudo dirigido

Resumo Climatológico Outubro de 2019 · Rua C - Aeroporto de Lisboa 1749-077 Lisboa, Portugal 5 | 5 info@ipma.pt Tel: (+351) 21 844 70 00 Fax: (+351) 21 840 23 70 Figura 7 – Distribuição

APPUNTI DI METEOROLOGIA - la-meteo.it · METEOROLOGIA (9): Masse d’ aria e fronti METEOROLOGIA (10): Circolazione generale dell’ atmosfera. 2 METEOROLOGIA (1) Composizione chimica

DRAUGI 1 Campo Grande, 25-3º Bloco E – 1749-099 …...Campo Grande, 25-3º Bloco E – 1749-099 Lisboa – T 21 798 87 00 | 911 032 254 – E-mail: dmpu.dp.draugi@cm-lisboa.pt Ponto

Boletim climatológico mensal – novembro 2012 · Boletim climatológico mensal Instituto Português do Mar e da Atmosfera, I.P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa –

meteorologia 1

BOLETIM SISMOLÓGICO PRELIMINAR DO CONTINENTE E … · INSTITUTO DE METEOROLOGIA, I.P. Ministério da Ciência, Tecnologia e Ensino Superior Rua C - Aeroporto – 1749-077 Lisboa

Relatório de Projecto - di.fc.ul.ptpaa/reports/T17.pdf · Universidade de Lisboa Faculdade de Ciências Bloco C6 - Piso 3 - Campo Grande, 1749-016 Lisboa Tel & Fax: 351.217500084

WV images and Rapid cyclogenesis CASES Nuno Moreira (nuno.moreira@meteo.pt) Instituto de Meteorologia, Lisboa, Portugal 2003

meteorologia 2