Users Guide

This Users Guide outlines the basic requirements of the Standardized Verification System (SVS) for Long Range Forecasts (LRF). Full details are outlined in the Manual on the Global Data-Processing System (attachments II.8 (new-updated 8/2006)). It is recommended that users download the Manual for use in conjunction with this Users Guide.

Index

• Variables to be Assessed
• Levels of Assessment
• Diagnostic Measures
• What the Lead Centre Provides
• How to Submit Results
• Format for Submitting Results
• Detailing your forecast system

Variables to be Assessed

• Air Temperature Anomaly at 2 metres above the surface
• Precipitation Anomaly
• Sea Surface Temperature Anomaly (For Level 1 (see below) only Nino 3.4 is required)

Levels of Assessment

The verification system is divided into three levels. Users are requested to assess their model(s) in the order from level 1 (first) to level 3 (last). The SVS suggests this staged implementation may be spread over a 2 year period. Please note: while key regions and variables to be assessed are proposed, producers are not limited to these parameters. (i.e., if your system does not produce a global outlook, you may still contribute for the areas in which it does, or for some more specific region (e.g., North America only.))

The three levels of assessment are:

• Level 1: Diagnostic measures aggregated over regions (Tropics, Northern extratropics, Southern extratropics)
• Level 2: Diagnostic measures evaluated at individual grid-points
• Level 3: Contingency tables provided for individual grid points.

Diagnostic Measures

The three diagnostic measures used by the core SVS for all levels of assessment are:

• Relative Operating Characteristics (ROC)
• Reliability Diagrams including frequency histograms
• Mean Square Skill Scores (MSSS)

It is recommended that all assessment be performed against the key data sets of observations. Regions over which forecasts should be assessed, as well as lead times and seasons required are all outlined in the Manual. Updates to the manual include a recommended hidcast period of 1981-2001.

The attached table provides a summary of the requirements of the core SVS - sections refer to section numbers in the old attachment II.9 of the Global Data Processing System. (NB: Please note, a revised attachment is now available.)

What the Lead Centre Provides

• The standard verification data sets in GRIB format. These cover the recommended hindcast period of 1981-2001.
• Software (fortran subroutines) for calculating all the verification scores, maps, frequency histograms and reliability diagrams.

How to Submit Results

To submit the verified data, place the ASCII files on an FTP area belonging to your Global Producing Centre. Once this has been done, please fill in the submission form to alert the Lead Centre of the FTP site address and directory in which your data is held. The Lead Centre will collect the data and plot up the results in a standard presentation format.

You will be contacted for final approval of the plotted results before these appear on the Lead Centre Web site.

In order to simplify this process, all scores need to be submitted in a standard format, with a standard naming convention.

Format for Submitting Results

In order to simplify the plotting up of results by the Lead Centre, all data must be submitted in a standard format with a standard filename. Due to the large amount of data that will be received, it is very important that all data sets are in the correct format. If unsure about any of the format requirements below please contact the lead centre for further details.

The filename should have the format:
{parameter}.{area}.{lead}.{season}.{enso_state}.{diagnostic}.{system_abbreviation}.{submission_number}.txt

where:
parameter is t2m (temperature at 2 metres), prp (precipitation), sst (sea surface temperature) or n34 (Nino 3.4)
area is t (tropics), n (northern extra tropics), s (southern extratropics) or g (grid) (include t for tropics even if calculation is for the NINO3.4 area - to keep the format the same)
lead is lead time for the verification; e.g., 3 for 3 months, minimum is 1 month, maximum 6 months
season is the season for which the model has been verified.
Level 1: Only the four conventional seasons, DJF, MAM, JJA and SON.
Level 2: Twelve rolling three-month periods may be submitted, e.g. MAM for March-April-May, AMJ for April-May-June MJJ for May-June-July.
enso state stratified according to ENSO years, a (all), e (El Nino) and l (La Nina).
diagnostic is: Level 1 - mssb (mean square skill score - bulk number for region examined), rocc (ROC curve), reld (reliability diagram), freq (frequency histogram) and rocs (ROC area for region).
Level 2 - msss (mean square skill score), mss1, mss2, mss3 (mean square skill score 1st, 2nd, 3rd terms) and roca (roc area - grid)
system_abbreviation is an acronym or name for the model or institution submitting the data
submission_number is 1 if only one model will be assessed from your institution, otherwise it increments by one for each model your institution verifies.

All data is to be submitted in ASCII format.

For ROC areas (rocs), files should simply contain the name of the organisation on the first line, the score for the lower tercile as a single number on the second, the score for the upper tercile as a single number of the third line and the score for the middle tercile as a single number on the fourth line (at this stage the middle tercile score will not be displayed so may be left out). For MSSS areas (mssb), files should simply contain the name of the organisation on the first line, and the score as a single number on the second. If you do not have data for some of the leads or seasons, please do not create that file. For missing files we will simply place a ----- in the relevant field. If you do not have any data for any of the fields for the given diagnostic no output file will be created.

ROC curves (rocc), Reliability diagrams (reld) and Frequency histograms (freq), data should be submitted in the following format:
Organisation name on the first line, number of data pairs on the second then...
For the ROC curve, "X" refers to false alarm rate, and "Y" refers to the hit rate, where for example, for ten probability bins, the first pair of values is for the 100% probability bin, the second value is for the 90% probability bin etc, in 10% increments. Please include 100% and 0% (if possible) even if these pairs are (0,0) and (1,1), so for this example there will be a total of 11 points.
For Reliability diagrams, "X" refers to the forecast probability bins, and "Y" is the observed frequency (both axis will range from 0 to 1 so please scale scores accordingly).
For the Frequency histograms, "X" refers to the forecast probability bins, and "Y" refers to the frequency of occurrence.

The first list for ROC curves, Reliability diagrams and the Frequency histograms will contain the results for the lower tercile, the second list will contain the results for the upper tercile and have exactly the same format (except the header [organisation name] is not repeated).

header                      (maximum of 80 characters)
npts                         (number of X/Y pairs)
x(1) y(1)                   (first X/Y pair)
.
.
.
x(npts) y(npts)           (final X/Y pair)
An example is available of the format for Level 1 ROC curve

All data is to be submitted in ASCII format.

Grid point data should be submitted in the following format, with the standard filename as described above. For ROC gridded areas the format is slightly different with the lower tercile data written out first as the first data set, the upper tercile should then be written out straight after the lower tercile, in the same format but with only the title row (no lat lon etc). Mask values and missing values should be set to 99999.9:

Latitude values should be in the range -90 (south) to 90 (north). Longitude should range from 0 to 360 for east-west. Note that the lon and lat values should not be followed by any N, S or E. Fractions of a degree should be written, for example, -89.5 or 200.25.

An example is available of the format for Level 2 MSSS at each gridpoint

The table below summarises the diagnostics and seasons for each level.

-- Level 3 --
All data is to be submitted in ASCII format (note this level is not yet operational).

For deterministic forecasts, 3x3 contingency tables at each grid-point:

nlat                                        (number of latitude values)
ylat(j),j=1,nlat                          (list of latitude values)
nlon                                       (number of longitude values)
xlon(i),i=1,nlon                        (list of longitude values)
header                                  (maximum of 80 characters)
(((data(h,i,j),h=1,9),i=1,nlon),j=1,nlat)      (data array, sorted by contingency table values (where values 1 to 3 refer to the observations corresponding to the below normal forecasts, 4 to 6 refer to observations corresponding to the near normal forecast category, and values 7-9 refer to the observations associated with the above normal forecast category), longitude and latitude values).

For probabilistic forecasts, nx2 contingency tables at each grid-point:

nlat                                        (number of latitude values)
ylat(j),j=1,nlat                          (list of latitude values)
nlon                                       (number of longitude values)
xlon(i),i=1,nlon                        (list of longitude values)
header                                  (maximum of 80 characters)
((((data(g,h,i,j),g=1,2),h=1,N),i=1,nlon),j=1,nlat)      (data array, sorted by contingency table values (where values 1 to 2 refer to the observed occurrences (1) and observed non-occurrences (2), and values 1,N refer to the corresponding observed or non observed occurrences for the N different probability bins), longitude and latitude values)

Detailing your forecast system

This information should be submitted as an ASCII text file, with the title:
sdetails.{system_abbreviation}.{submission_number}.txt
e.g., sdetails.bmrc.2.txt would refer to the second model submitted from the Bureau of Meteorology Research Centre (BMRC).

Information submitted should include, but is not limited to:

• Whether the system is numerical, empirical or hybrid
• Whether the system is deterministic of probabilistic
• Model type and resolution
• Ensemble size
• Boundary conditions specifications
• List of parameters being assessed
• List of regions for each parameter
• Lead times available for each parameter (see Manual for definition)
• Period of verification
• Number of hindcasts and the dates/years of these hindcasts
• Details of verification data sets used
• If different from standard, the resolution of fields used.