Australian Government - Bureau of Meteorology Home | About Us | Contacts | Help | Feedback |

Global | Australia | NSW | Vic. | Qld | WA | SA | Tas. | ACT | NT | Ant. |
Weather & Warnings | Hydrology | Climate | Numerical Prediction | About Services | Learn About Meteorology | Registered User Services |

NMOC Quarterly Summary April-June 2002

Summary of System Performance
Model Rainfall Verification

Analyses & Numerical Prediction | About Products | Map/Image/Chart Archives

ruler

These model rainfall verifications are based on RAINVAL, which verifies daily quantitive precipitation forecasts for NWP models against operational daily rainfall analyses.

(i) Cities time series

The three models verified are LAPS (0.375 deg. resolution), Meso LAPS (0.125 deg.) and GASP (~0.80 deg.). For validity of comparison the analysed and predicted rainfall values have been standardised to a 1 deg. grid.

mesolaps 0 to 24 comparison

 

laps 0 to 24 comparison

 

laps 24 to 48 comparison

 

GASP 0 to 24 comparison

GASP 24 to 48 comparison

GASP 48 to 72 comparison

 

(ii) Comparative Statistics

Comparative Statistics Averaged for 20020401-20020630 1 deg. resolution

 

 

 LAPS125
00-24
AUST 		LAPS375
00-24
AUST 		LAPS375
24-48
AUST 		GASP
00-24
AUST 		GASP
24-48
AUST 		GASP
48-72
AUST
Mean Abs Error (mm/d) 0.58 0.59 0.61 0.59 0.65 0.70
Bias Score 0.95 0.96 0.93 0.95 1.04 1.07
Probability of Detection 0.55 0.55 0.54 0.54 0.54 0.49
False Alarm Ratio 0.42 0.43 0.42 0.44 0.48 0.54
Equitable Threat Score 0.34 0.34 0.34 0.33 0.31 0.26

 

Note that the mean absolute error is the average of the daily values while the categorical statistics (Bias, POD, FAR and ETS) are computed from pooled space/time match-ups. The area of verification is the land area of Australia.

Definitions
Mean Absolute Error (MAE)
The MAE is the average magnitude of the forecast error at individual locations at grid points. Errors in the location and/or magnitude will cause the MAE to be large.
Bias Score
The bias score measures the relative area (or frequency) of predicted and observed rainfall, without regard to forecast accuracy. A perfect score indicates the predicted rainfall area (or frequency) is the same as observed. However, there may still be an error in location.
Bias = hits+false alarms
hits+misses

Bias=1 indicates no bias

Bias<1 underforecasts rainfall

Bias>1 overforecasts rainfall

Probability of Detection (POD)
The probability of detection measures the success of the forecast in correctly predicting the occurrence of events,
POD =       hits      
hits+misses		 Limits:0-1, Perfect score: 1

The POD is sensitive only to missed events, not false alarms. POD can be increased by issuing a larger number of rain forecasts on the assumption that a greater number will be correct, usually at the cost of more false alarms.

False Alarm Ratio (FAR)
The false alarm ratio measures the fraction of event forecasts which were actually non-events,
FAR =     false alarms     
hits+false alarms		 Limits:0-1, Perfect score: 1

FAR is sensitive only to false predictions and not to missed events. This score can always be decreased by underforecasting the number of severe events, but only at the cost of more missed events.

Equitable Threat Score (ETS)
The ETS is a modification to the Critical Success Index that takes into account the number of correct forecasts of events (hits) that would be expected purely due to chance. The equitable threat score can be written as:
ETS =        ZH-FM         
(F+M)N+(ZH-FM)		 Perfect score: 1

where Z is the number of zeros, F is the number of false alarms, M is the number of misses, H is the number of hits, and N is the total number of forecasts made. The minimum value of ETS is -1/3. The equitable threat score is in common usage in the United States.

 

(iii) Bar Chart of Comparative Statistics

The chart shows the ratio of forecast to analysed values of rain area, average rain rate, rain volume and maximum rain rate for LAPS, Meso LAPS and GASP. A score of 1 represents a perfect forecast.

RAINVAL has been developed by Beth Ebert and John McBride of BMRC. Further details are available.

References

Ebert, E.E. and J.L. McBride, 1997: Methods for verifying quantitative precipitation forecasts: application of the BMRC LAPS model 24-hour precipitation forecasts. BMRC Techniques Development Report No. 2

McBride J.L. and E.E. Ebert, 2000: Verification of quantitative precipitation forecasts from operational numerical weather prediction models over Australia. Wea. Forecasting 15, 103-121.

Weymouth, G. et al., 1999: A continental-scale daily rainfall analysis system. Aust. Met. Mag., 48, 169-179.

ruler


Home | About Us | Learn about Meteorology | Contacts | Search | Help | Feedback
Weather and Warnings | Climate | Hydrology | Numerical Prediction | About Services | Registered Users | SILO
© Copyright Commonwealth of Australia 2008, Bureau of Meteorology (ABN 92 637 533 532)
Please note the Copyright Notice and Disclaimer statements relating to the use of the information on this site and our site Privacy and Accessibility statements. Users of these web pages are deemed to have read and accepted the conditions described in the Copyright, Disclaimer, and Privacy statements. Please also note the Acknowledgement notice relating to the use of information on this site. No unsolicited commercial email.