QA4Seas - Quality Assurance for Multi-model Seasonal Forecast Products

Project start 01.07.2016
Project end 30.09.2018
Topics Weather
Region World
Status Completed projects

The European Commission’s Copernicus QA4Seas (Quality Assurance for Multi-model Seasonal Forecast Products, C3S_51 Lot3) aimed at developing a strategy for the evaluation and quality control (EQC) of multi-model seasonal forecasts provided by the Copernicus Climate Change Service (C3S). Being one of the project partners, MeteoSwiss has been active in different areas of the project with a specific focus on the scientific assessment of the C3S seasonal forecasts.

QA4Seas (C3S_51 Lot3) in a nutshell

The European Commission’s Copernicus Climate Change Service (C3S) aims at supporting adaptation and mitigation by providing authoritative climate data and information through its Climate Data Store (CDS). Observations and reanalyses, seasonal forecasts and climate projections provide information about past, present and future climate and climate change. A key aspect for the success of the C3S is the Evaluation and Quality Control function that provides quality assurance information for all products and ensures that users have access to the information they need to use the service. The QA4Seas (Quality Assurance for Multi-model Seasonal Forecast Products, C3S_51 Lot3) project aims at developing a strategy for the evaluation and the quality control (EQC) of multi-model seasonal forecasts provided by the C3S in response to the needs of a wide range of stakeholders of the CDS. Key components of the QA4Seas project are: a) a survey identifying the needs of CDS users, b) a scientific assessment of multi-faceted quality aspects of the seasonal forecasts available through the CDS, c) a gap analysis identifying scientific and technical gaps, d) the development of a prototype of the EQC system, and e) the development of a solution to include forecast/product provenance metadata. MeteoSwiss has been active in all areas of the project with a specific focus on the scientific assessment of the C3S seasonal forecasts.

MeteoSwiss contribution – scientific assessment of seasonal forecasts

Forecast quality of seasonal forecasts is limited and varies strongly in space, in time (both lead time and forecast valid time), by parameter and forecasting system. Further, seasonal forecast models suffer from systematic errors that need to be corrected in order to render the forecasts useable. Recommendations on how to adjust these errors have been developed based on an in-depth analysis of re-forecasts with the C3S forecasting systems from the European Centre for Medium-Range Weather Forecasts (ECMWF), Météo France, and the Met Office for the period 1993-2014. Seasonal forecasts are communicated in two distinct formats: a) ensemble forecasts, that is probabilistic forecasts of absolute values (e.g. probability for temperature to exceed 18° C and b) multi-category probability forecasts, that is forecast probabilities relative to the reference climatology (e.g. warmer or colder than normal). For the latter, the current forecast is compared to an archive of past forecasts to determine the probabilities for each category.

For the three variables considered within QA4Seas, that is near surface temperature, sea surface temperature, and precipitation,  bias adjustment is able to considerably increase forecast quality. However, only minor differences in forecast quality could be detected among the different methods that have been tested. In general, simple bias adjustment methods outperform more sophisticated approaches due to the short training period (1993-2014).  Multi-model combination further improves  the forecasts consistently, but the actual gain in forecast skill due to multi-model combination is rather small.  The panels on the left of the figure show an example of a multi-model seasonal forecast for El Niño, i.e. positive temperature anomalies in the tropical Pacific, the skill of the ECMWF seasonal ensemble before any processing, the bias adjusted ECMWF seasonal ensemble, and a multi-model combination of the three seasonal forecast models available from the CDS.

Based on the scientific assessment, we recommend to process ensemble and multi-category probability forecasts differently. While it is crucial to bias adjust seasonal ensemble forecasts, multi-category probability forecasts do not need any bias adjustment, because they are implicitly bias adjusted by calculating the category probability relative to the reference climatology. Further, the different nature of the two forecast products calls for different multi-model combination approaches. Multi-category probability forecast products can be directly combined by computing (weighted) averages of the forecast probabilities provided by the different forecast models, whereas multi-model ensemble forecasts can be either obtained by simple pooling of the ensemble members or by more sophisticated approaches based on mixture distributions. The panels on the right of the figure show the multi-category probability analogue of the example that has been introduced in the previous paragraph.

The QA4Seas scientific assessment identified quality, strength and limitations of seasonal forecast models and multi-model combinations thereof. Currently, MeteoSwiss issues ECMWF seasonal ensemble based multi-category probability forecasts of temperature for the north-east, the west, and the south of Switzerland. The results from QA4Seas confirmed this practice. Apart  from using a multi-model ensemble to compute the multi-category probabilities, it is unlikely that the current practice can be outperformed by more sophisticated approaches.

Sub-panels showing different forecast products.
Illustration of the generation of seasonal forecast products. 1st row: Example Nino3.4 multi-model (ECMWF, Met Office, and Météo France) forecasts initialized in November 2013. 2nd row: skill of the raw ensembles in terms of probabilistic verification measures (CRPSS for the continuous ensemble forecasts and RPSS for the multi-category probability forecasts). 3rd row: skill of the recalibrated ensemble forecasts. Multi-category probability forecasts are implicitly recalibrated. 4th row: Skill of the multi-model forecasts. Both CRPSS and RPSS are positively oriented, i.e. the larger the value the higher the forecast skill, with a perfect forecast resulting in a score of one and values below zero reflecting forecasts that perform worse than the reference climatology.