QUALITY INFORMATION DOCUMENT For Arctic Ocean Physical ... · 1.4 02 Sep 2013 All New document format. Added reference to Altika data. G.A. Waagbø, B. Hackett, A. Melsom Laurent

© EU Copernicus Marine Service – Public Page 1/ 39

QUALITY INFORMATION DOCUMENT

For Arctic Ocean Physical Analysis and Forecast Product

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001_A

Issue: 6.0

Contributors: Arne Melsom, Magne Simonsen, Laurent Bertino, Bruce Hackett, Geir Arne Waagbø, Roshin Raj

Approval Date by Quality Assurance Review Group : 19 june 2017

QUID for Arctic Ocean Physical Analysis and Forecasting Product


Ref : CMEMS-ARC-QUID-002-001a

Date : 22 May 2017

Issue : 6.0


CHANGE RECORD

Issue Date § Description of Change Author Checked By

1.0 08 Jun 2011 All First version of document, With copy-paste from ScVR

L. Bertino

1.1 30 Sep 2011 Removal of links to ScVR L. Bertino Bruce Hackett

1.2 05 Jul 2012 II.1 Revise schedule G.A.. Waagbø

1.3 08 Apr 2013 Revise AATSR validation information

B. Hackett

1.4 02 Sep 2013 All New document format. Added reference to Altika data.

G.A. Waagbø, B. Hackett, A. Melsom

Laurent Bertino

1.5 02 Dec 2013 I.1 Information regarding atmospheric forcing data added.

G.A. Waagbø L. Bertino

1.6 15 Jan 2014 III.1, IV.1 Modified for 10-members ensemble mean forecast

L. Bertino

4.0 13 Mar 2014 All Updated document format. Revisions due to QuARG review.

A. Melsom, G. A. Waagbø, B. Hackett

L. Bertino

4.1 3 Jun 2014 I.3, III.1, V.1-V.5

V.6

Updated description of assimilated SLA-products, as well as updated validation figures.

Removed temporarily section on surface buoy drift until “undrogued”-QC is implemented

G.A. Waagbø L. Bertino

4.2 12 Jun 2014 III.2 Change related to assimilation of Hy2a SLA-products.

G.A. Waagbø, H. Engedahl

5.0 01 Jan 2015 Updated version numbers L. Bertino

5.1 06 Mar 2015 All Revision after V5 acceptance

5.2 1 May 2015 All Change format to fit CMEMS graphical rules

L. Crosnier




Date : 22 May 2017

Issue : 6.0


5.3 11 Jan 2016 All Change text to reflect organizational change, updated metrics values and graphics, added results for sea level anomaly correlation

A. Melsom

5.4 04 Apr 2016 Note on Jason 2, updated links L. Bertino

5.5 25 Apr 2016 All Metrics quantities now reflect results from operational phase; a minor adjustment in definition of sea ice classes has been implemented as suggested by CMEMS OSI experts

A. Melsom

5.6 29 Aug 2016 I.3, IV, V.6 Added validation results for bottom temperature. Also, v5.5 edited in response to comments from CMEMS product quality team

A. Melsom, R. Raj

5.7 09 Oct 2016 I.3, IV, V.6 Added more details regarding results for bottom temperature

A. Melsom

5.8 18 Jan 2017 V.5 Added note on the “buffering” of SLA

L. Bertino

6.0 22 May 2017 Metrics now includes results for two years in operational phase; also included are results for sea ice type, mixed layer depth

A. Melsom




Date : 22 May 2017

Issue : 6.0


TABLE OF CONTENTS

I EXECUTIVE SUMMARY............................................................................................................................... 5 I.1 Products covered by this document ........................................................................................................... 5 I.2 Summary of the results ............................................................................................................................... 5 I.3 Estimated Accuracy Numbers .................................................................................................................... 6

I.3.1 BIAS ................................................................................................................................................... 7 I.3.2 RMS differences ................................................................................................................................. 9 I.3.3 Correlation.......................................................................................................................................... 9

II External Operational Products .................................................................................................................... 10

III Production Subsystem description ............................................................................................................. 11 III.1 TOPAZ4 Production Cycle at MET Norway ....................................................................................... 11 III.2 Observational Data ................................................................................................................................ 12

IV Validation framework ................................................................................................................................. 13 IV.1 Model results ........................................................................................................................................... 14 IV.2 Validation products ................................................................................................................................ 15 IV.3 Period and domain ................................................................................................................................. 18

V Validation results .......................................................................................................................................... 21 V.1 Validation vs. SST observations from drifting buoys ........................................................................... 22

V.1.1 Day 1 forecast ................................................................................................................................. 25 V.1.2 Day 6 forecast ............................................................................................................................... 236

V.2 Validation of SST vs. the AVHRR satellite-based product .................................................................. 24 V.2.1 Day 1 forecast ................................................................................................................................. 24 V.2.2 Day 6 forecast ................................................................................................................................. 25

V.3 Validation of sea ice concentration and ice edge vs. ice chart data ...................................................... 26 V.3.1 Day 1 forecast ................................................................................................................................. 26 V.3.2 Day 6 forecast ................................................................................................................................. 28

V.4 Validation of sea ice drift vs. satellite-derived observations ................................................................. 30 V.4.1 Day 1 forecast ................................................................................................................................. 30 V.4.2 Day 6 forecast ................................................................................................................................. 31

V.5 Validation of sea ice type vs. satellite-derived observations ................................................................. 30 V.5.1 Day 1 forecast ................................................................................................................................. 30 V.5.2 Day 6 forecast ................................................................................................................................. 31

V.6 Validation of sea level anomaly vs. Jason-2/3 altimeter data ............................................................... 35 V.6.1 Day 1 forecast ................................................................................................................................. 35 V.6.2 Day 6 forecast ................................................................................................................................. 36

V.7 Validation of bottom temperature vs. World Ocean Atlas data .......................................................... 37

V.8 Validation of mixed layer depth vs. ARGO profile data ...................................................................... 38 V.8.1 Day 1 forecast ................................................................................................................................. 35 V.8.2 Day 6 forecast ................................................................................................................................. 36




Date : 22 May 2017

Issue : 6.0


I EXECUTIVE SUMMARY

I.1 Products covered by this document

This document covers the following products:

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001_a, which is the real-time forecast of physical parameters in the Arctic.

The product is produced by the Arctic Monitoring and Forecasting Centre (Arctic MFC) using the TOPAZ4 numerical prediction system, which consists of the following elements:

Model: Nansen Center’s version of HYCOM V2.2.37

Spatial resolution: horizontal = ~12 km (1/6th to 1/8th deg), vertical = 28 Hybrid z-isopycnal layers.

Time period for the V1/V2/V3 version of the CMEMS product: since April 2015

Data assimilation: Deterministic version of the Ensemble Kalman Filter (DEnKF) with asynchronous assimilation (ensemble version of the First Guess at Adequate Time FGAT)

Assimilated data: see below.

Atmospheric forcing data: Data from ECMWF's HRES (High Resolution) Model are used. The data are on a global regular lat/lon grid (0.125 x 0.125 degrees), and are interpolated to the model grid in a pre-processing step.

The model products are available through the CMEMS Catalogue as daily averages on a 12.5 x 12.5 km grid at 12 depths.

I.2 Summary of the results

Validation of products from the operational forecasts has been performed as an activity of solely comparing model results and observations. The rational for this choice is that users who work with forecasts are expected to have a focus of the variability on time scales from hours to one or two weeks. In such a context, comparison with e.g. climatology is of less interest, with the exception of bottom temperature, for which observations are very sparse. The list of variables which will undergo validation will be expanded according to the Scientific Validation Plan for the Arctic MFC.

Based on the results provided in Chapter IV, we deem the results from this Arctic MFC product to be of a satisfactory quality. Improvements vs. the V0 products from the Arctic MFC have been reported earlier (see also http://cmems.met.no/ARC-MFC/Validation/V0toV1).

While no specific model discrepancy has been identified, there is room for improvement of all forecast products. Hence, we will continue the present strategy for improving the quality of products from Arctic MFC. This strategy includes (1) refining the horizontal and vertical (layer) resolution in the simulations, (2) expanding and refining the assimilation step, (3) improving the

http://cmems.met.no/ARC-MFC/Validation/V0toV1




Date : 22 May 2017

Issue : 6.0


performance of the ocean circulation model, and (4) examine if an improvement is possible by application of results from the Global MFC at the open boundaries.

I.3 Estimated Accuracy Numbers

The tabulated accuracy numbers in this section are statistical values for bias and root mean square (RMS) differences between model and observed quantities. Biases are calculated with observational values subtracted from model results, so e.g. a positive bias occurs when model output value is larger than the corresponding observation. RMS differences are computed from the squares of offsets between model results and observations. Moreover, for SLA, the RMS difference EAN is supplemented by the correlation EAN. Additional information regarding the EANs are given in section IV.

Field Name of Product Name of References

SST (Buoys)

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a Derived from subset of INSITU_GLO_NRT_OBSERVATIONS_013_030

SST (AVHRR)

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a Metop-A,B AVHRR data (Non-CMEMS product)

Sea ice concentra

tion

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_002

Sea ice edge


Sea ice drift


Sea ice type


SLA ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a Subset of SEALEVEL_GLO_PHY_L3_NRT_OBSERVATIO

NS_008_044

Bottom temperature

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a World Ocean Atlas data (Non-CMEMS product)

Mixed layer depth

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a Subset of

INSITU_ARC_NRT_OBSERVATIONS_013_031

T,S profiles

ARCTIC_ANALYSIS_FORECAST_PHYS_002_001a Subset of

INSITU_ARC_NRT_OBSERVATIONS_013_031

The table gives the names of the products that are used to calculate the Estimates Accuracy Numbers (EANs). The subsets referred to, are the surface drifting buoys from the global drifter




Date : 22 May 2017

Issue : 6.0


program for SST (Buoys), and the Jason-2 along-track data for SLA (switched to Jason 3 in Oct 2016).

Results that are tabulated below were retrieved on May 19th 2017, except for ocean bottom temperature for which results were retrieved in early August 2016.

For numbers designated with (†): during some months, SST Buoy Full domain results are questionable due to possible problems with observations in the Arctic Ocean. These data, which are suspected to be from buoys with instruments on the sea ice surface (or in snow on sea ice), thus not measuring water temperature, were not flagged as suspicious in the observational product. During periods were such observations are not suspected to be present, the SST estimated accuracy numbers indicate a similar quality for the Full domain case as for the two limited regions for which values are provided (see also figures in Section V.1, and the discussion in Sections V.1 and V.2).

I.3.1 BIAS

Forecast

Day1 Day6

SST Buoys Full domain (K) -0.3(†)

-0.3(†)

Nordic Seas (K) -0.3 -0.3

Barents Sea (K) -0.4 -0.5

AVHRR Full domain (K) -0.2 -0.2


Barents Sea (K) -0.1 -0.1

Tbottom WOA Full domain (K) 0.5 0.5

Sea Ice

Edge Edge length (103 km) -2.5 -2.6

Edge position (km) -14 -12

Type, first year ice

Arctic Ocean (106 km

2) -0.2 -0.1

Type, multi-year ice

Arctic Ocean (106 km

2) -0.2 -0.1

Drift Arctic Ocean (km/day) 2 2

American sector (km/day) 4 3

European sector (km/day) 2 2

Asian sector (km/day) 1 1

Mixed layer depth

Full domain (m) 19 41

Nordic Seas (m) -6 9

T profile 0-5m Full domain (K) -0.15 -0.09


5-100m Full domain (K) -0.3 -0.3





Date : 22 May 2017

Issue : 6.0


100-300m Full domain (K) -0.2 -0.06

Nordic Seas (K) 0.3 -0.3

300-800m Full domain (K) 0.5 0.5

Nordic Seas (K) 1.1 1.1

800-2000m Full domain (K) 0.3 0.3


S profile 0-5m Full domain (‰) -0.03 -0.03

Nordic Seas (‰) -0.12 -0.14

5-100m Full domain (‰) -0.06 -0.06

Nordic Seas (‰) -0.15 -0.17

100-300m Full domain (‰) -0.12 -0.07

Nordic Seas (‰) -0.12 -0.12

300-800m Full domain (‰) -0.02 -0.02

Nordic Seas (‰) -0.01 -0.02

800-2000m Full domain (‰) 0.009 0.007

Nordic Seas (‰) 0.01 0.01

I.3.2 RMS differences

Forecast

Day 1 Day 6

SST Buoys Full domain (K) 1.1(†)

1.1(†)


Barents Sea (K) 0.6 0.7

AVHRR Full domain (K) 0.8 0.9


Barents Sea (K) 0.7 0.7

Tbottom WOA Full domain (K) 1 1

Sea Ice

Concentration Concentration (%) 18 20

Edge Edge position (km) 45 50

Drift Arctic Ocean (km/day) 5 8

American sector (km/day) 6 7

European sector (km/day) 5 8

Asian sector (km/day) 5 9

SLA South of 66N (cm) 7 8

Nordic Seas south (cm) 6 6

T profile 0-5m Full domain (K) 0.6 0.6


5-100m Full domain (K) 0.6 0.7




Date : 22 May 2017

Issue : 6.0



100-300m Full domain (K) 0.8 0.8


300-800m Full domain (K) 1.0 1.0


800-2000m Full domain (K) 0.6 0.6


S profile 0-5m Full domain (‰) 0.2 0.2


5-100m Full domain (‰) 0.2 0.2


100-300m Full domain (‰) 0.3 0.12


300-800m Full domain (‰) 0.06 0.06


800-2000m Full domain (‰) 0.04 0.04


I.3.3 Correlation

Forecast

Day 1 Day 6

SLA South of 66N 0.6 0.5

Nordic Seas south 0.6 0.6




Date : 22 May 2017

Issue : 6.0


II EXTERNAL OPERATIONAL PRODUCTS

Validation of the sea surface temperature is performed using data from drifting buoys, which are compiled by the CMEMS INS TAC (InSitu Thematic Assembly Center). A post-processed version of the buoy data are freely available from IFREMER, from ftp://ftp.ifremer.fr/ifremer/cersat/projects/myocean/sst-tac/insitu/data/, with a temporal resolution of approximately 3 hours. These data are converted to daily mean values to conform with the TOPAZ model results prior to the validation, and observations which have questionable quality are disregarded. Note that the analysis is not restricted to night-time data.

Validation of the sea surface temperature is also performed using a remote sensing product (originally AATSR, now substituted by AVHRR) which is based on observations from a satellite borne instrument. The AATSR/AVHRR data are available from a password protected ftp site, operated by IFREMER. Although these data are included in one of the observational products that are assimilated in the Arctic MFC ocean model analysis, the AATSR and AVHRR products have much stricter cloud masks than the assimilated product, and thus comes with an increased data quality at the expense of the domain coverage. Note for document Issue 1.3 and later: The AATSR validation process was stopped in April 2012 upon the failure of Envisat and its data provision. It has been replaced by METOP-A AVHRR as recommended reference instrument.

The validation of sea ice concentration and sea ice edge position is performed based on ice charts for the Arctic Ocean that are available from the CMEMS OSI TAC / OSI-SAF EUMETSAT service. The primary source of data for these ice charts are SAR images. The ice charts are produced daily on week days, and divide the domain into regions with a discrete set of sea ice classes, as defined by WMO. The data are freely available from the CMEMS catalogue as product SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_002. SAR data are presently not included in the analysis and initialization of the Arctic MFC model, so these data constitute a set that is independent from the model results.

Observational results for sea ice type are available as part of CMEMS OSI TAC product SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_001. These data are derived from SSMIS brightness temperature contrasts, and ASCAT data.

High-resolution sea ice drift observations, derived using a pattern recognition algorithm, are available as CMEMS OSI TAC product SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_006. This observational product is derived from Synthetic Aperture Radar (SAR) imagery, and has a temporal resolution of 1 day as it compares images that are separated by about 24 hours.

Temperature and salinity profiles are available from the Argo drifters, with data included in CMEMS INS TAC product INSITU_ARC_NRT_OBSERVATIONS_013_031. The data are collected from deep drifters that surface every 10 days, and transmits the data to overflying satellites. This data set is also used to infer the observational mixed layer depth, which is set to the depth where the offset from the uppermost observation is 0.03 kg/m3.

ftp://ftp.ifremer.fr/ifremer/cersat/projects/myocean/sst-tac/insitu/data/

http://marine.copernicus.eu/web/69-interactive-catalogue.php?option=com_csw&view=details&product_id=SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_002




Date : 22 May 2017

Issue : 6.0


III PRODUCTION SUBSYSTEM DESCRIPTION

The Arctic MFC V2/V3 nominal system is the TOPAZ4 system based on an advanced sequential data assimilation method (the Ensemble Kalman Filter, EnKF) in its deterministic flavour (DEnKF, Sakov and Oke, 2009) and the Hybrid Coordinate Ocean Model (HYCOM version 2.2). The Arctic systems have been upgraded successively but there has not been any cold restart or “double diffusion” of parallel production chains.

III.1 TOPAZ4 Production Cycle at MET Norway

The model results are produced with the TOPAZ4 ocean data assimilation model system. Presently, TOPAZ4 data assimilation is run weekly on Thursdays to produce an analysis which is valid for the preceding Monday. The following Monday a one-week 100 member ensemble hindcast simulation is run to produce a best estimate for each of the preceding 7 days. Finally, a 10 days, 10-members ensemble forecast is run daily using the most recent analysis, forced by updated and perturbed atmospheric fields. The ensemble mean forecast is delivered to the users and used for the validation.

TOPAZ4 was developed and is maintained by the Nansen Environmental and Remote Sensing Center (NERSC, http://nersc.no/). It is run operationally for CMEMS Arctic MFC production at the Norwegian Meteorological Institute (MET Norway, http://met.no).

The table illustrates the weekly schedule for the TOPAZ system. The vertical axis indicates forecast lead time in days.

http://nersc.no/

http://met.no/




Date : 22 May 2017

Issue : 6.0


III.2 Observational Data

Observational data are assimilated into the TOPAZ system in the weekly analysis. All data are assimilated, but instead of “background check”, the observation error is increased for those observations too far from the model forecast, in order to moderate the impact of their assimilation. Presently the following data are assimilated:

Altimetric sea level anomaly (SLA) observations from the Jason-3, Cryosat, (since September 2013) AltiKa, and (since June 2014) Hy2a satellites. From May 2014 the following SLA-products are combined and assimilated:

◦ SEALEVEL_GLO_SLA_L3_NRT_OBSERVATIONS_008_017 (south of 50N)

◦ SEALEVEL_ARC_SLA_L3_NRT_OBSERVATIONS_008_025 (north of 50N)

These replace the obsolete SLA-products that were assimilated until the end of April 2014 (SEALEVEL_GLO_SLA_L3_NRT_OBSERVATIONS_008_001_a). In the transition to the new SLA-products the reference level for mean sea level height was adjusted from a 7-year mean to a 20-year mean – thus the new observations are adjusted before assimilation to compensate for this change (by subtracting an offset of 2cm).

OSTIA global SST observations.

OSISAF global sea ice concentration.

OSISAF global sea ice drift observations.

In situ observations of temperature and salinity profiles from the Arctic In Situ TAC since June 2013 (All profiles).




Date : 22 May 2017

Issue : 6.0


IV VALIDATION FRAMEWORK

Validation of the CMEMS Arctic MFC results for sea surface temperature is performed using data from drifting buoys, which are compiled by the CMEMS In Situ TAC via IFREMER. The buoy data have a temporal resolution of approximately 3 hours. These data are converted to daily mean values to conform with the TOPAZ model results prior to the validation, and observations which have questionable quality are disregarded. Note that the analysis is not restricted to night-time data, so daily model averages are compared to daily averages from buoy observations. Near-surface drift is validated using positional data from the same source of data (i.e., drifting buoys).

Validation of the CMEMS Arctic MFC results for sea surface temperature is also performed against satellite data from the currently identified reference instrument. Up to April 2012, this was the Envisat AATSR product. Although these data are included in one of the observational products that were assimilated in the Arctic MFC ocean model analysis at the time, the AATSR product had a much stricter cloud mask than the assimilated product, and thus comes with an increased data quality at the expense of the domain coverage.

In the absence of operational data, the validation of bottom temperature is performed against observed monthly climatologies from the World Ocean Atlas (WOA). Note that the WOA data were collected during the time-period 2005-2012, so there is an inconsistency in the time reference of the two dataset.

The validation of sea ice concentration and sea ice edge position is performed based on ice charts for the Arctic Ocean that are obtained from the CMEMS OSI TAC: SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_002. The primary source of data for these ice charts are SAR images. The ice charts are produced daily on week days, and divide the domain into regions with a discrete set of sea ice classes, as defined by WMO. SAR data are presently not included in the analysis and initialization of the Arctic MFC model, so these data constitute a set that is independent from the model results. The unavailability of observational data from this product on Saturdays, Sundays and holidays implies that no validation can be performed for these days.

The validation of Arctic sea ice drift is performed against global satellite observations obtained from the CMEMS OSI TAC: SEAICE_ARC_SEAICE_L4_NRT_OBSERVATIONS_011_006. The ice drift data are based on a pattern recognition algorithm which is applied to observations from Synthetic Aperture Radar (SAR) imagery.

The validation of sea level anomaly forecasts is performed using the SLA data from a single

altimeter: currently Jason-3, which flies in a non-sun-synchronous orbit at an inclination of 66 to Earth's equator. The data are obtained from the CMEMS SL TAC. Since Jason-3 is assimilated by the ARC MFC, this does not constitute an independent validation. Anomalies in the TOPAZ results are defined analogously to the anomaly product based on the Jason-3 data.

Our aim is to provide validation results of the present real-time forecast system. Validation is performed against independent (i.e. non-assimilated) products from CMEMS TACs, when a relevant observational product exists (OSI TAC, INS TAC), as described above.

We have focused the present analysis on quantities which are of particular interest to the Arctic region. These are sea ice variables, and sea surface temperature and sea level anomalies, but




Date : 22 May 2017

Issue : 6.0


note that other variables such as temperature/salinity profiles, sea level and near-surface trajectories are also included in our validation of the product. Snapshots that illustrate model results for sea ice variables are shown below.

The product data are validated against two SST observational products (in situ and satellite), independent sea ice concentration and edge observations from the Svalbard region, satellite Arctic sea ice drift observations, temperature and salinity from Argo profiles, SLA observations from a single altimeter and near-surface drift against positional data from drifting buoys. Validation metrics are calculated for each daily forecast horizon: Day 1 forecast to Day 10 forecast. Results are published at http://cmems.met.no/ARC-MFC/V2Validation/index.html with weekly updates.

Sea ice concentration, fractions in the range 0-1, as shown by the colour bar to the right of the map. Snapshot is for an arbitrary day in March, for a sub-domain of the TOPAZ4 full model domain. Provided for illustrative purposes only.

Sea surface temperature, in °C as shown by the colour bar to the right of the map. Snapshot is for an arbitrary day in March, for a sub-domain of the TOPAZ4 full model domain. Provided for illustrative purposes only.

IV.1 Model results

TOPAZ results are available as an aggregated “best estimates” product from the CMEMS data portal using the SUBSETTER download vehicle, and as a list of separate forecasts from the production cycle with weekly updated bulletin dates using the GetFile vehicle. The “best estimate” product is the average of daily means from the 100-member ensemble model, followed by the ensemble average from the 10-members ensemble forecast for future dates.

Presently, the model results that are validated are

sea surface temperature sea ice concentration position of the sea ice edge sea ice drift temperature and salinity profiles, averaged in selected layers as specified by the CMEMS

PQWG for joint use across MFCs (not shown in this document) sea surface elevation near-surface drift

For the validation of sea surface temperature and near-surface drift, model results for ocean temperature and velocity at the 5 meter level are used.

bottom temperature but since operational observations are not available, we apply data from the World Ocean Atlas (monthly climatologies) for this purpose.

http://cmems.met.no/ARC-MFC/V2Validation/index.html




Date : 22 May 2017

Issue : 6.0


IV.2 Validation products

The validation products are listed here:

Sea Surface Temperature - buoys

1. Number of observations 2. Bias 3. Error based on RMS differences

Sea Surface Temperature – satellites (AATSR, AVHRR)

1. Number of observations 2. Bias 3. Error based on RMS differences

Sea Ice – ice charts

1. Bias of sea ice edge length 2. Bias of sea ice edge position 3. Error based on RMS differences of sea ice edge distance 4. Error based on RMS differences of sea ice concentration 5. Confusion matrix with match-up of sea ice classes; six ice classes:

a) Ice free b) Open Water c) Very open drift ice d) Open drift ice e) Close drift ice f) Very close drift ice

6. Ice Classes Area 7. Shaded contour maps with geographical distribution of ice classes 8. Error based on RMS differences

Sea Ice – OSI SAF SSMIS product

1. Bias of sea ice edge length 2. Bias of sea ice edge position 3. Error based on RMS differences of sea ice edge distance 4. Error based on RMS differences of sea ice concentration 5. Confusion matrix with match-up of sea ice classes; six ice classes:

a) Ice free (sea ice concentration in range <0.01) b) Open Water (range 0.01-0.1) c) Very open drift ice (range 0.1-0.4) d) Open drift ice (range 0.4-0.7) e) Close drift ice (range 0.7-0.9) f) Very close drift ice (range >0.9)

6. Ice Classes Area 7. Maps displaying position of sea ice edge in various products 8. Sea ice areal deficit/excess (model/observation concentration differences >0.25) 9. Shaded contour maps displaying regions with excess, and deficit, sea ice concentrations




Date : 22 May 2017

Issue : 6.0


Sea Ice Drift

1. Number of ice drift trajectories 2. Ice drift distance 3. Error based on RMS differences of drift

Sea Ice Type

1. Areal bias of first year (FY) ice; multi-year (MY) ice 2. FY ice area; MY ice area 3. Confusion matrix with match-up of sea ice classes; six ice classes:

a) Open b) FY ice c) MY ice

4. Shaded contour maps displaying regions with FY ice; MY ice

Sea level anomaly

1. Number of observations 2. Error based on RMS differences 3. Correlations

Mixed layer depth

1. Number of observations (mixed layer depth is derived from profile data) 2. Bias of temperature and salinity in each layer 3. Error based on RMS differences of temperature and salinity in each layer 4. Anomaly correlations (using monthly climatologies from WOA)

Temperature and salinity profiles – ARGO profiles

1. Number of profiles 2. 5 depth intervals 3. Bias of temperature and salinity in each layer 4. Error based on RMS differences of temperature and salinity in each layer 5. Anomaly correlations (using monthly climatologies from WOA)

Temperature and salinity profiles – ITP profiles

1. Number of profiles 2. 3 depth intervals 3. Bias of temperature and salinity in each layer 4. Error based on RMS differences of temperature and salinity in each layer 5. Anomaly correlations (using monthly climatologies from WOA)

Bottom temperature – WOA

1. Bias 2. Error based on RMS differences




Date : 22 May 2017

Issue : 6.0


Near-surface drift

1. Number of profiles 2. Bias of 24h drift distance 3. RMS of separation of trajectories after 24h drift

Arctic MFC validation results are updated weekly, presently late on Thursdays. Results are displayed on-line at http://cmems.met.no/ARC-MFC/Validation/index.html, except for bottom

temperature (no operational data). Note that the validation software is re-run for each Thursday bulletin after one, two and three weeks, in order to incorporate observations that span the entire forecast period, and to include data that are available with a corresponding delay. Bias always refers to results when observational values are subtracted from model results, so e.g. a positive bias in SST occurs when model output is warmer than the corresponding observations.

The algorithm for validation of sea ice concentration is described in detail in a report that is available from the web site mentioned above (to be specific, it may be downloaded e.g. from http://cmems.met.no/ARC-MFC/V1Validation/SeaIceConcentration/SICvalNote.pdf).

The algorithm for computing the RMS distance between the model ice edge and the observed ice edge is an extension of the methods used to validate sea ice concentration. The ice edge is here defined as the boundary between ice classes Open Water (OW) and Very Open Drift Ice (VODI). In practice, the edge is identified by those grids which belong to class VODI (or higher concentrations) and has at least one neighbour (up, down, left or right) belonging to class OW (or ice free). An illustration of how the identified grids may line up are displayed below, with one product (e.g. observations) represented by the red grids, and the other represented by the blue grids.

Then, for each grid belonging to the model sea ice edge, e.g. the light blue grid above, the closest grid belonging to the observed edge grids is found, in our example this is the light red grid. The distance between the observed and modeled sea ice edge is then taken to be the distance between the centres of these two grids. Furthermore, in this document we present validation results for the bias of the ice edge position. A positive bias means that the modeled sea ice edge is on the open ocean side of the observed edge. Note that this metric can be interpreted as a proxy for sea ice extent.

http://cmems.met.no/ARC-MFC/Validation/index.html

http://cmems.met.no/ARC-MFC/V1Validation/SeaIceConcentration/SICvalNote.pdf




Date : 22 May 2017

Issue : 6.0


IV.3 Period and domain

Results from the ARC MFC product have been validated since 2011-12-21. These products achieved operational status on 2011-12-16. However, from this QUID version onwards, we have chosen to use results from the CMEMS phase only (i.e., starting 2015-04-01, discarding results from the MyOcean phases) for the estimated accuracy. As explained above, asynchronous data were used for validation of bottom temperature.

The ocean circulation model used in CMEMS' Arctic MFC covers the Arctic Ocean, the North Atlantic Ocean and adjacent ocean regions. The northern part is depicted in the figure below. Validation of sea surface temperature from model results is performed for three domains:

an extended domain indicated by the blue, green and red regions in the figure

the Nordic Seas, shown as the green region

the Barents Sea, depicted in red




Date : 22 May 2017

Issue : 6.0


For the ice drift validation, the Arctic has been divided into 3 sectors as shown on the map below:

65 °W – 55 °E (European, Green)

55 °E – 175 °E (Asian, Blue)

175 °E – 65 °W (American, Red)

The combinations of these sectors is referred to as the Arctic Ocean (e.g. in the tables in Sections I.3.1 and I.3.2).

Regions for which validation is performed for the various products are displayed on http://cmems.met.no/ARC-MFC/V2Validation/regions.

http://cmems.met.no/ARC-MFC/V2Validation/regions/




Date : 22 May 2017

Issue : 6.0


Table 1 Metrics with results in this document, and status of validation activity

Name of metric Model output Supporting observations

SST-D-CLASS4-BUOY-BIAS-<region>

SST-D-CLASS4-BUOY-RMSD-<region>

On model output grid (12.5 km, top 5 m layer), for forecast day 1 and 6

SST from drifting buoys

SST-D-CLASS1-IRAD-BIAS-<region>

SST-D-CLASS1-IRAD-RMSD-<region>

Same as above High-quality satellite data from OSI TAC

SIEP-D-CLASS1-CHART-BIAS-<region>

SIEP-D-CLASS1-CHART-RMSD-<region>

SIEL-D-CLASS1-CHART-BIAS-<region>

On model output grid, for forecast day 1 and 6

Same as above

SIUV-D-CLASS1-SAR-BIAS-<ArcticSector>

SIUV-D-CLASS1-SAR-RMSD-<ArcticSector>


SAR-based displacement calculations

SITY-D-CLASS3-SAT-BIAS-<region> On observation grid, for forecast day 1 and 6

SSMIS/ASCAT-based observations of first year, multi-year ice vs. derived representation from model product

SLA-D-CLASS4-ALT_JASON3-RMSD-<region>

SLA-D-CLASS4-ALT_JASON3-CORR-<region>


Single altimeter SLA data (Jason-3; Jason-2 prior to 2016-09)

bottomT-BOT-M-CLASS1-CLIM-BIAS

bottomT-BOT-M-CLASS1-CLIM-RMSD

On model output grid, regions with H<1500m

Monthly climatology from the World Ocean Atlas

MLD-D-CLASS4-PROF-BIAS-<region>

MLD-D-CLASS4-PROF-RMSD-<region>


Derived from Argo data

T-LAYER<X_Ym>-D-CLASS4-PROF-BIAS-<region>

S-LAYER<X_Ym>-D-CLASS4-PROF-BIAS-<region>

T-LAYER<X_Ym>-D-CLASS4-PROF-RMSD-<region>

S-LAYER<X_Ym>-D-CLASS4-PROF-RMSD-<region>

On model output grid, vertically averaged over 5 layers, for forecast day 1 and 6

Quality-controlled Argo data from INS TAC

<region> stands for the Extended, Nordic and Barents Sea area, <ArcticSector> corresponds to Arctic Ocean and European/Asian/American Sector. These regions were defined in detail above.




Date : 22 May 2017

Issue : 6.0


V VALIDATION RESULTS

The full set of operational validation products is maintained and accessible on the web pages found under http://cmems.met.no/ARC-MFC/V2Validation/index.html. The figures in this chapter are representative samples of time series of selected validation quantities, with bulletin dates provided on the x-axis. Subsections labeled ‘Day 1 forecast’ are validation of the daily mean from first the 24 hours after the data assimilation/model re-initialization, whereas ‘Day 6 forecast’ refers to the daily mean centered at a date that lags Day 1 by 5 days.

In some of the figures lines are thin in parts of the full time series. Thin lines are used to bridge curves when validation results are not available from one or more intermediate bulletin dates due to lack of observations.

http://cmems.met.no/ARC-MFC/V2Validation/index.html




Date : 22 May 2017

Issue : 6.0


V.1 Validation vs. SST observations from drifting buoys

V.1.1 Day 1 forecast

For the Extended domain (“N. Atlantic + Nordic/Arctic”) model values that are averaged in the observation space are within 0.5K of the observations, with the exception of a spike in September 2015 and a prolonged period in the second half-year of 2016. Furthermore, RMS values are about 1K or lower, with the same exceptions. A closer look at the comparison reveals that the misfit in the 2nd half-year of 2016 occurs mainly in response to buoy data from regions that are likely ice infested. Although these data have not been flagged as suspicious, we strongly




Date : 22 May 2017

Issue : 6.0


suspect that the reported temperature values are not from water, but from top of ice or in snow on ice. This suspicion is strengthened by the comparison with AVHRR data below.


As for the Day 1 case, the main finding here is the suspicious, apparent deterioration in the 2nd half of 2016. Again, we strongly encourage to examine the comparison with AVHRR data in the next sub-section, which tells quite a different story.




Date : 22 May 2017

Issue : 6.0


V.2 Validation of SST vs. the AVHRR satellite-based product


We find that the validation using the remote sensing data indicates an even higher quality than the validation based on buoy observations, not only in the “suspicious 2nd half of 2017” (see the previous subsection), but as a general result. In this context, it is worth pointing out that (a) the data coverage of the remote sensing product is much higher than that of the buoy data, and (b) these remote sensing data are part of the SST data set used in the model’s assimilation system, so the present validation is not performed with completely independent data. Further, note the




Date : 22 May 2017

Issue : 6.0


increasing magnitude of the negative bias during the summer of 2015, which is accompanied by high RMS differences. This situation improved slightly in the summer of 2016.


Here also, we find that the deterioration in the forecast accuracy from forecast day 1 to forecast day 6 is very small.




Date : 22 May 2017

Issue : 6.0


V.3 Validation of sea ice concentration and ice edge vs. ice chart data

The figures below rely on the metrics defined in Section II. The RMS for sea ice fraction are complemented by a perhaps more intuitive error in distance from the ice edge.





Date : 22 May 2017

Issue : 6.0


There is no long term trend in the validation results. Changes appear to be dominated by seasonal changes (e.g. RMS offsets for sea ice concentration have high values during summer), and short-lived events (e.g. numerous spikes in the ice edge position RMS difference).




Date : 22 May 2017

Issue : 6.0






Date : 22 May 2017

Issue : 6.0


We find that the deterioration in the forecast accuracy from forecast day 1 to forecast day 6 is very small. The exception to this is the quality of the ice edge position forecast, which has an increasing RMS offset from Day 1 to Day 6 when compared to the ice chart data.




Date : 22 May 2017

Issue : 6.0


V.4 Validation of sea ice drift vs. satellite-derived observations

The SAR satellite sea ice drifts have a resolution of about 10km, which is roughly the same as the model resolution (the CMEMS ARC product presently has a resolution of 12.5 km). For a complete discussion of the scale-dependence of sea ice drift statistics, see Rampal et al. (2008).


All drifts are calculated as 24 hours drift, based on the daily mean sea ice drift parameter. The American, European and Asian sectors correspond to 175°E - 65°W, 65°W - 55°E, and 55°E - 175°E, respectively. Bias values are computed from the length of the drift distance, disregarding any directional offset between model results and observations.




Date : 22 May 2017

Issue : 6.0


These validation results are indicative of a stable quality. Note however that results in the sub-regions can from time to time become a bit noisy, during periods when the data coverage is limited. The RMS difference is computed from the distance between the trajectory end points based on model results and observations. Thus, directional offsets are important for this metric, and it nicely accompanies the drift distance bias results.





Date : 22 May 2017

Issue : 6.0


The results for drift bias indicate that the forecasted trajectory length does not deteriorate from Day 1 to Day 6. However, the results for RMS offsets are somewhat deteriorated for this span in forecast time range. Hence, the deterioration of quality with an increasing forecast range is associated with the forecasted drift direction, not the speed forecast.




Date : 22 May 2017

Issue : 6.0


V.5 Validation of sea ice type vs. satellite-derived observations

The results for the separation between first year ice and multi-year ice from model and observations are not completely compatible. Furthermore, both products are subject to significant inaccuracies, but due to different processes. Thus, the two products should be viewed as complementary, and it is difficult to evaluate the quality of the model results by a comparison with the observational product. The observational product is not available during summer.





Date : 22 May 2017

Issue : 6.0


Two observations of the results above can be noted: (1) The differences during the summer of 2016 are much larger than during the summer of 2015; (2) The differences between model and observations for FY ice and MY ice are much larger than the differences in the sum of the two ice types.


The differences between model results and observations do not increase from Day 1 to Day 6, in fact, we note a small decrease in the differences.




Date : 22 May 2017

Issue : 6.0


V.6 Validation of sea level anomaly vs. Jason-2/3 altimeter data

The Jason-2 and Jason-3 satellite tracks are limited to south of 66N. See the maps of the areas North Atlantic and Nordic Seas on the following link: http://cmems.met.no/ARC-MFC/V2Validation/regions/


Validation results for sea level anomaly shows a constant product quality except in the period from April to September 2016, when the SLA observation errors were doubled to “buffer” the






Date : 22 May 2017

Issue : 6.0


change of mean SL at V2. Note that the Jason-2/3 data are assimilated by the model, so these are not independent data. Note also the seasonal reduction in correlations during spring.


Sea level anomalies are an integral measure of the temperature and salinity variability in the ocean. Due to the high ocean inertia, it is therefore not surprising that the validations of sea level anomalies indicates close to no quality loss from the day 1 forecast of sea level anomalies to the day 6 forecast.




Date : 22 May 2017

Issue : 6.0


V.7 Validation of bottom temperature vs. World Ocean Atlas data

The TOPAZ bottom temperature during four months time-period (April to July 2016) has been validated using the World Ocean Atlas (WOA) 2013 database, version 2. Note that the monthly climatologies (April to July) of WOA are estimated during the time-period 2005-2012. Hence there is an inconsistency in the time reference of the two dataset. In addition, the analysis is limited to regions where the bottom depth is 1500 m or less due to the non-availability of bottom depth temperature observations elsewhere. The figure shows that the model has a warmer bias in the regions around the Arctic, Barents Sea, Faroe Shetland Channel and Canadian Archipelago. On the other hand, the East Greenland Current and the Labrador Current are the prominent regions where the model temperature is lower than observations. The estimated accuracy numbers given in the tables in Section 1 in this document are areal averages.

Model minus WOA bottom temperature mean errors (left panel) and RMS differences (right panel). In the left panel, a positive mean error value indicates a model warm bias.




Date : 22 May 2017

Issue : 6.0


V.8 Validation of mixed layer depth vs. ARGO profile data

There are on average approximately 15 profiles available each day for the Extended domain north of 50°N. About 5 of these profiles originate in the Nordic Seas domain.


We note that there is a very strong seasonal variability in the Results. While the model results in the period June-November are nearly un-biased with respect to the Argo data, the remaining (winter) bias results exhibit extreme variations with both large positive and large negative values. The period with large fluctuations in the results coincide with the season with weak winter stratification due to intense cooling at the surface. Thus, very small discrepancies in the density profiles from model and observations lead to the large fluctuations in the validation results.




Date : 22 May 2017

Issue : 6.0



As was the case with the Day 1 validation results, the Day 6 validation results are unsurprisingly dominated by large fluctuations in the period with weak ocean stratification in this region. In the complementary period, there is little or no deterioration in the quality of model results.

Documents

QUALITY INFORMATION DOCUMENT For Arctic Ocean Physical ... · 1.4 02 Sep 2013 All New document format. Added reference to Altika data. G.A. Waagbø, B. Hackett, A. Melsom Laurent