Metview in a Pythonic World - KNMI · Metview in a Pythonic World EGOWS 2019, KNMI, De Bilt, The Netherlands Iain Russell Development Section, ECMWF Thanks to Linus Magnusson and

© ECMWF November 13, 2019

Metview in a Pythonic World

EGOWS 2019, KNMI, De Bilt, The Netherlands

Iain Russell

Development Section, ECMWF

Thanks to Linus Magnusson and

Martin Janousek

What is Metview?

• Workstation software for researchers and operational analysts

– Runs on UNIX, from laptops to supercomputers

• Retrieve/manipulate/visualise/examine meteorological data

• Batch mode or graphical user interface

• Can access MARS, either locally or through the Web API

• Open Source under Apache Licence 2.0

• Metview is a co-operation project with INPE (Brazil)

2EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

Built on top of ECMWF software packages


Metview

Data decoding

ecCodes

(GRIB, BUFR)

ODB_API

Other (NetCDF,

Geopoints, CSV)

Regridding

MIR

Plotting

Magics

Data Access

MARS CDS Files WMS URL

Using Metview


• Icon-based user interface

– interactive investigation of data

– icons represent data, settings and processes

– icons can be chained together –output from one is input to another

• e.g. filter fields from a certain date, then

pass that to the Cross Section icon

• Powerful Python/Macro scripting language

– more serious computations

– batch or interactive usage

Generating Python code from the GUI


Generating Python code from the GUI


Data Formats

• GRIB, BUFR, NetCDF, ODB, Geopoints, CSV

• Plot

• Examine

• Filter, regrid, masking

• Maths, Boolean

• Specialised:

– Cross section

– Thermodynamics

– Gradient

– Vertical integration

– Model to pressure lev

– Etc


Contouring schemes


• Plenty of options for complete customisation of palettes

Contouring schemes

• A set of pre-defined palettes is also available

– But you still have to supply the mapping between values and colours


Contouring schemes

• Can select from pre-defined styles

– the styles come from ecCharts

– everything is done for you

– or choose “Contour Automatic Setting = ECMWF” – style will be chosen based on meta-data


Complete ecCharts Layers

• The ecCharts icon goes further – retrieves data from MARS and styles it


Macro language

• Powerful, high-level scripting language

• Native handling of major data types

– e.g. fieldsets (GRIB)

• Some nice stuff, e.g. model-obsdifferences (gridded minus scattered data)


Goals of a Python interface

• Combine the power of Macro and the Python ecosystem

– High-level meteorological aware functions and objects

– Thin Python bindings on top of the core C++ Metview code

– Easy translation from Macro to Python

– With help from B-Open


Goals of a Python interface (2)

• Opens up new development environments

• E.g. Jupyter notebooks

– Great teaching tool

– Mentioned quite a lot in ECMWF’s recent workshop “Building reproducible workflows for earth sciences”


Creating a Python interface

• Python provides all of the language features that are available in Macro (e.g. loops, conditionals, functions), so no need to try to ‘extend’ it

• Python provides the main ‘primitives’ that Metview uses, e.g. numbers, strings, datetimes, lists, dictionaries

• So the only things we need to provide are:

– All the Macro functions

– Correct handling of those ‘primitive’ types

– Correct handling of indexing (Python uses 0-base, Macro uses 1-base)

– Specific classes, e.g. Fieldset, Geopoints, Odb

– Make them Pythonic, e.g. iterable containers

– Overload operators, e.g. diffs = fieldset1 – fieldset2


Implementation

• First: split Metview’s Macro module into a shared library plus a small executable that links to it

• Use cffi in Python to load the library

– Need a header file with C-style function prototypes for all API calls

• Obtain list of all functions

– Dynamically create a new Python function to call each one and add to namespace


Implementation

• Add classes that know how to be passed between C++ and Python

– Allow object-oriented, e.g. mv.interpolate(precip, 20.1, 19.3) == precip.interpolate(20.1, 19.3)

• Add tests ☺


Python bindings

• Can extract data as numPy arrays from most data types

– See Jupyter example “Principal component analysis of ensemble forecast fields”


Python bindings

• Can obtain a pandas dataframe from Geopoints, Table and ODB

– See Jupyter example “Difference between gridded field (GRIB) and scattered observations (BUFR)”


Python bindings

• Can obtain an xarray dataset from GRIB

– see Jupyter example “Computing ensemble mean and spread with xarray and plotting the results with Metview”


Internally uses the Python

module cfgrib, developed with

B-Open

Metview now used in Verification

• “The availability of Metview's classes and functions in Python stimulated a significant overhaul of a major operational verification software at ECMWF

• Metview now takes care of all meteorological data decoding, filtering and post-processing, including geography-aware calculations, extending the applicability of the verification software to a wider range of data formats, grids and parameters

• Thanks to utilising Metview the verification application significantly reduced its code base and opened up the path to a more modular and fast-to-develop software architecture.”

• Work done with help from B-Open


Verification project – next steps

• Development will continue into 2020 with a focus on enhancing its modularity, building a 'verification toolbox' with an extended scope to cover the wide range of recent and future verification demands in research and operations


Metview in Diagnostics

• “For model evaluation and diagnostics, Metview's Python interface is a clear step forward

• Here the combination in Python of using Metview for the calculations on the GRIB files and the Pandas module for making the time-series analysis has proven to be very powerful

• One example application is a simple cyclone tracker aimed for advanced diagnostics”


Metview in Diagnostics (2)

• “The Python script uses the Metview module to first retrieve data from the MARS archive

• Next, a Python function performs the cyclone tracking by using several Metview functions on the fieldset containing the mean-sea-level-pressure

• It returns a new position of the cyclone, which is used together with the distance(), mask() and integrate() functions in Metview to obtain various diagnostic quantities

• These are collected in a Pandas array, which can then for example be used for plotting with the matplotlib library”


Future

• Ability to add new Metview modules written in Python, not C++

– Should increase community contributions

• Build on this work to create something that benefits from all the hindsight that we have, and utilises more existing Python modules to provide a richer data processing / visualisation experience

• Already in prototype phase: a new plotting module written in Python and using the SkinnyWMS


Examples

• See the Gallery for Macro and Python examples


Examples (2)


Examples

• See the Jupyter Notebooks for more Python


Tutorials

• Plenty of material online including tutorials, but most of it is currently Macro-based (very easy to convert though)


Metview Availability

• Available on ECMWF systems:

– Versioned using the ‘module’ system

– [module swap metview/new]

– metview

• On other systems:

– Install from RPM (Linux)

– Install from conda (Linux and macOS)

– Ubuntu package

– Build from source

– Build from bundle

– pip install metview


For more information…

• Email the developers:

– [email protected]

• Or ask User Support

– [email protected]

• Visit our web pages:

– http://confluence.ecmwf.int/metview

• Documentation and tutorials available

• Gallery of examples


Questions?

Documents

Metview in a Pythonic World - KNMI · Metview in a Pythonic World EGOWS 2019, KNMI, De Bilt, The Netherlands Iain Russell Development Section, ECMWF Thanks to Linus Magnusson and