|
||||||||||||||||||||||||||||||||||||||||||
Analyses & Numerical Prediction
The operational Australian region Limited Area Prediction System known as LAPS_PT375 was upgraded to a new version in NMOC Melbourne on 2 April 2003. This upgrade involves the use of a new "Bulk Explicit Microphysics" (BEM) rainfall scheme that has been developed in the BMRC Model Development Group.
The upgraded version underwent a parallel operational trial in NMOC during the period December 2002 to March 2003. Objective assessments were carried out against analysis and observations. A statistical evaluation of rainfall forecasts indicates that the upgraded LAPS_PT375 generally performed better than the previous version throughout the 72-hour forecast range, particularly for frontal rain events where there is less likelihood of spurious widespread light rain. Forecast performance in tropical regions however has been essentially unchanged.
The general configuration of LAPS_PT375 has not changed from that described in Analysis and Prediction Operation Bulletin No 56 (APOB56) of 24 September 2002. In brief, it continues to be based on a 3-cycle analysis/forecast scheme in which observational data are inserted every 6 hours (model time). At each data insertion, an analysis is performed followed by a soil-moisture adjustment and then an initialisation and prediction. The prediction model is integrated forward to either the next analysis time or the end of a longer forecast. The only changes in this upgrade have been the introduction of the BEM scheme in the forecast model and some minor data analysis changes, which are discussed below.
The Bulk Explicit Microphysics rainfall scheme has been developed in BMRC by Dr Richard Dare. A brief description follows; more details of the BEM scheme are available within a BMRC report currently in press (Dare 2002).
The label "bulk" refers to the gross representation of individual hydrometeor classes, where each is generally represented by a mixing ratio alone, with no prognostic estimation of number concentration or size distribution. "Explicit" refers to the property of the scheme to represent actual microphysical processes occurring at each grid point within a three-dimensional model grid, rather than, for example, a single parameterisation approximating a series of processes.
The purpose of the BEM scheme is to provide a comprehensive representation of microphysics that is an improvement on the parameterisations used previously in atmospheric numerical models. The code has been constructed in a modular format allowing the user to control activation of individual components of the scheme. While a large number (56) of microphysical processes are available within the BEM scheme, only a small number (9) of these have been used in trials within LAPS over the past 1-2 years. This limitation also applies to operations and has been applied for two reasons: (1) to minimise the additional computational requirements, and (2) to limit the change in complexity of the representation of microphysics during this initial upgrade. A schematic of the 9 processes used is shown below (fig. 1).
This limited configuration of BEM includes representation of cloud water, rain water and ice crystals. In terms of realism, this improves upon the previous representation of large-scale precipitation in which water vapour was condensed at saturation to produce rain at the ground with no detailed representation of clouds (water and ice) and only a very simple representation of fallout of rain. Note that the scheme as implemented here deals with large-scale precipitation only - convective precipitation is still parameterised by the Tiedtke mass-flux scheme in the model.
Figure 1: Schematic representation of the 9 microphysical processes used in the present version of the BEM scheme.
After the previous LAPS_PT375 system upgrade in September 2002, it became evident that widespread over-raining was a problem in the new model. Several data analysis changes were made during October 2002 to deal with this problem - the soil moisture initialisation was switched back to using the Pescod scheme based on rainfall observations and climatological surface evaporation rate estimates, an assimilation code correction was made to make use of more of the surface moisture synop observations, and GMS moisture bogus data was switched off. As a result, the over-raining was reduced significantly.
During the parallel testing period of the BEM scheme some problems became evident in the operational LAPS_PT375 lower tropospheric moisture analysis over SE Australia. An improvement was made, by BMRC, to the METAR moisture observation processing over higher topography areas. This involved an appropriate use of "superobbing", which entails combining similar nearby observations into one observation. The change was incorporated in the LAPS_TEST parallel trial system between 22 Jan and 13 Feb 2003, after which it was introduced into the operational LAPS_PT375 assimilation scheme.
The general operational configuration of the LAPS_PT375 system remains as described in APOB48 (e.g. the model domain, resolution, vertical level structure have not been changed) and readers should refer to that report for details.
One point to note is that the use of the BEM scheme in LAPS_PT375 has resulted in a slight increase in the elapsed time for the prognosis step on the NEC SX-5 from approximately 13.2 minutes (real time), for the old system, to 15.6 minutes (using 8 CPUs & 2.4 Gb memory).
NMOC conducted an assessment of BEM in LAPS_PT375 in a parallel trial during December 2002 - March 2003 prior to operational implementation. By far the most significant differences occurred in the rainfall forecasts and these were verified against observational rainfall using the RAINVAL rainfall verification package. In addition, conventional verification targets, such as the mean sea level pressure (MSLP) and the geopotential heights (HGHT), as well as surface weather parameters such as the screen temperature and 10m winds were also verified. The results are discussed below.
1. Rainfall verification
The following rainfall verifications were performed using the RAINVAL statistical verification package, which verifies daily quantitative precipitation forecasts for NWP models against daily rainfall analyses. RAINVAL was developed by Beth Ebert and John McBride of BMRC (McBride & Ebert, 2000). A variety of statistical scores are available from this system for judging aspects of rainfall forecast performance. Further details, including a glossary that explains the strengths and weaknesses of the various statistical scores presented here, can be found at http://www.bom.gov.au/bmrc/mdev/expt/rainval/rainval_gui/rainval_gui.shtml.
Table 1 shows RAINVAL statistics for the previous operational LAPS_PT375 and the BEM LAPS_TEST model forecasts averaged over all Australian gridpoints for the period 18 December 2002 to 24 Mar 2003. It can be seen that the BEM forecasts show improvements in almost all measures of skill over the entire 72 hour forecast range. The operational scheme had a tendency to forecast excessive light rainfall, which the BEM scheme suppresses, and this shows up in the Bias Score (a simple measure of rainfall area) with values much closer to the ideal value of 1 than the operational scheme. This reduced rainfall area results in improved (i.e. reduced) False Alarm Ratios but on the other hand it also results in decreased Probabilities of Detection. The more sophisticated measures of skill such as the Critical Success Index, Hanssen & Kuipers Score and Equitable Threat Score all show general improvement for all forecast periods with BEM.
| Observed | Operational LAPS_PT375 | BEM LAPS_TEST | |||||
|---|---|---|---|---|---|---|---|
| 00-24 hr | 24-48 hr | 48-72 hr | 00-24 hr | 24-48 hr | 48-72 hr | ||
| Rain Area (km2 * 103) | 1599 | 2064 | 2148 | 2212 | 1660 | 1741 | 1753 |
| Avg Intensity (mm/d) | 11.5 | 9.9 | 9.7 | 9.8 | 11.3 | 10.8 | 10.7 |
| Rain Volume (km3) | 18.3 | 20.5 | 20.9 | 21.6 | 18.7 | 18.8 | 18.7 |
| Max Intensity (mm/d) | 89.1 | 66.8 | 73.7 | 76.2 | 68.1 | 72.7 | 69.9 |
| Mean Abs Error (mm/d) | - | 3.0 | 3.3 | 3.7 | 2.8 | 3.1 | 3.4 |
| RMS Error (mm/d) | - | 7.4 | 8.3 | 9.3 | 7.4 | 8.1 | 8.9 |
| Correlation Coefficient | - | 0.55 | 0.48 | 0.40 | 0.55 | 0.50 | 0.41 |
| Bias Score | - | 1.29 | 1.34 | 1.38 | 1.04 | 1.09 | 1.10 |
| Probability of Detection | - | 0.78 | 0.77 | 0.72 | 0.72 | 0.71 | 0.67 |
| False Alarm Ratio | - | 0.30 | 0.43 | 0.48 | 0.30 | 0.34 | 0.39 |
| Critical Success Index | - | 0.52 | 0.49 | 0.44 | 0.55 | 0.52 | 0.47 |
| Hanssen & Kuipers Score | - | 0.60 | 0.57 | 0.49 | 0.61 | 0.58 | 0.51 |
| Equitable Threat Score | - | 0.38 | 0.34 | 0.28 | 0.43 | 0.39 | 0.33 |
Table 1: Average RAINVAL stats for all available days (92) and all gridpoints during Period 18 December 2002 to 24 March 2003. Verification done on a 0.75° grid, number of gridpoints = 1094. The best model results in each case are coloured red.
Figure 2 below shows the 0-24 hr dkill scores for the two models for various rain amount thresholds. It cam be seen that for all the scores there is a significant general improvement at the lower threshold values. At the very highest thresholds (≥20 & ≥50 mm/day) however, the skill score change is neutral or possibly very slightly worse.
 |
 |
 |
| a) Accuracy | b) Bias score | c) Probability of Detection |
 |
 |
 |
| d) False Alarm Ratio | e) Critical Success Index | f) Equitable Threat Score |
 |
 |
|
| g) Heideke Skill Score | h) Hanssen & Kuipers Score |
Figure 2: Comparative RAINVAL verification scores as a function of various rainfall thresholds (≥0,1,2,5,10,20 and 50 mm/day), averaged over all gridpoints for the period 18 December 2002 to 24 March 2003.
2. S1 Skill Score, Bias and Root Mean Square error for MSLP and HGHT
Table 2 below presents the mean S1 skill score, bias and root mean square (RMS) error of the operational LAPS_PT375 (OPS) and the new BEM LAPS_TEST for the period 17 Dec 2002 - 16 Feb 2003 over the Australian verification region. The overall result shows little significant change.
| Table 2 | Mean S1 Skill Score, Bias and RMS Error for MSLP and HGHT comparing the operational LAPS_PT375 model and the new version using BEM |
|---|---|
| Region: | Standard Australian verification grid |
| Total Period: | 12Z 17 Dec 2002 - 12Z 16 Feb 2003 using both 00 and 12 UTC runs |
| Verifying Analyses: | Self |
| Improvement: | absolute value of OPS LAPS_PT375 - absolute value of BEM LAPS_TEST. A positive improvement implies that the new BEM version skilled better than the operational version. |
| Forecast period | Verification statistic | Field | OPS | BEM | Improvement |
|---|---|---|---|---|---|
| +24HRS | S1 Skill Score | MSLP | 20.9 | 20.8 | 0.1 |
| 850 HGHT | 20.6 | 20.6 | 0.0 | ||
| 500 HGHT | 15.2 | 15.2 | 0.0 | ||
| 250 HGHT | 13.5 | 13.5 | 0.0 | ||
| BIAS | MSLP | 0.4 | 0.4 | 0.0 | |
| 850 HGHT | 3.0 | 2.9 | 0.1 | ||
| 500 HGHT | -1.0 | -2.8 | -1.8 | ||
| 250 HGHT | 3.9 | 4.3 | -0.4 | ||
| RMS | MSLP | 1.4 | 1.4 | 0.0 | |
| 850 HGHT | 10.4 | 10.4 | 0.0 | ||
| 500 HGHT | 14.3 | 14.3 | 0.0 | ||
| 250 HGHT | 20.6 | 20.8 | -0.2 | ||
| +48HRS | S1 Skill Score | MSLP | 26.2 | 26.2 | 0.0 |
| 850 HGHT | 25.5 | 25.7 | -0.2 | ||
| 500 HGHT | 19.6 | 19.7 | -0.1 | ||
| 250 HGHT | 17.7 | 17.8 | -0.1 | ||
| BIAS | MSLP | 0.4 | 0.4 | 0.0 | |
| 850 HGHT | 2.3 | 2.0 | 0.3 | ||
| 500 HGHT | -6.5 | -9.8 | -3.3 | ||
| 250 HGHT | 2.4 | 3.3 | -0.9 | ||
| RMS | MSLP | 1.8 | 1.8 | 0.0 | |
| 850 HGHT | 13.6 | 13.7 | -0.1 | ||
| 500 HGHT | 19.5 | 19.8 | -0.3 | ||
| 250 HGHT | 27.9 | 28.2 | -0.3 | ||
| +72HRS | S1 Skill Score | MSLP | 31.0 | 30.8 | 0.2 |
| 850 HGHT | 30.2 | 30.3 | -0.1 | ||
| 500 HGHT | 23.7 | 23.8 | -0.1 | ||
| 250 HGHT | 21.8 | 21.8 | 0.0 | ||
| BIAS | MSLP | 0.5 | 0.5 | 0.0 | |
| 850 HGHT | 2.4 | 1.8 | 0.6 | ||
| 500 HGHT | -6.3 | -11.1 | -4.8 | ||
| 250 HGHT | 3.3 | 4.2 | -0.9 | ||
| RMS | MSLP | 2.3 | 2.3 | 0.0 | |
| 850 HGHT | 16.6 | 16.7 | -0.1 | ||
| 500 HGHT | 24.6 | 24.8 | -0.2 | ||
| 250 HGHT | 35.6 | 35.7 | -0.1 |
Units for bias and RMS are hPa for MSLP and m for HGHT. S1 skill score is non-dimensional.
3. Surface Weather Parameter Verification against the Aviation METAR Observations
Model forecast surface weather elements such as surface (2m, or screen level) temperature and dewpoint and surface (10m) wind direction and speed are routinely verified in NMOC against METAR observational data. RMS errors and biases are calculated at regular intervals out to a specified forecast period using all available Australian METARS (approximately 297).
Figures 3 and 4 show the time series plots of the mean bias and RMS errors in surface temperature and dewpoint averaged over all METAR sites and over all hourly model forecast periods out to +72 hrs. Results are shown for the period 15 December 2002 to 20 March 2003. It can be seen that, overall, there is very little difference in the errors of the BEM LAPS_TEST model relative to the operational LAPS_PT375 errors, the most significant differences occurring during the period 22 January to 13 February during which LAPS_TEST assimilation scheme was used to test an improved METAR moisture observation procedure involving superobbing. This procedure was incorporated into the operational LAPS_PT375 assimilation scheme on 13 February, after which the errors for the two models become almost identical. The 10m wind speed and direction errors for the two models were almost identical throughout the trial period and are not shown.
|
|
| a) | a) |
|
|
| b) | b) |
| Figure 3. Screen (2m) dry bulb temperature comparison between the operational LAPS_PT375 (LAPS) and the BEM trial (LAPS_TEST) errors averaged over all (approx. 297) available METAR sites and over all hourly model forecast periods out to +72 hrs. a) shows Bias and b) Root Mean Square (RMS) Error | Figure 4. Screen (2M) dewpoint temperature comparison between the operational LAPS_PT375 (LAPS) and the BEM trial (LAPS_TEST) errors averaged over all (approx. 297) available METAR sites and over all hourly model forecast periods out to +72 hrs. a) shows Bias and b) Root Mean Square (RMS) Error |
Terry Skinner of NMOC undertook a synoptic comparison between operational LAPS_PT375 and BEM LAPS_TEST during the parallel trial period. Since the only change to the model was the introduction of the BEM rainfall scheme there was little impact on the mean sea level pressure and wind forecasts. The following features emerged from the assessment of rainfall:
The test period was mainly dominated by monsoon activity over Northern Australia. Rain events over Southern Australia were minimal. There were several clear examples of over-raining (i.e. false alarms) in the operational LAPS_PT375 forecast which did not appear in the BEM LAPS_TEST version. An example of this is shown in figure 5 where the 24-hour rainfall analysis from NMOC up to 9am 8 Jan 2003 is compared against the corresponding +24 and +48 hr rainfall forecasts from the operational and test models. In the operational model forecasts there were significant false alarms over VIC, SA and WA, which were not evident in the test forecasts.
In monsoon rainfall situations there is little discernable difference between the BEM and operational in the vicinity of the monsoon low - both exhibit heavy falls. Even with BEM there are cases where both the test and operational models miss prediction of significant areas of convective tropical rainfall (10-25mm), e.g. N Kimberley
Figure 5a: BEM LAPS_TEST rainfall forecast for 0-24 hours up to 0000 UTC Wed 8 January 2003 |
Figure 5b: Operational LAPS_PT375 rainfall forecast for 0-24 hours up to 0000 UTC Wed 8 January 2003 |
Figure 5c: Verifying rainfall analysis over Australia for 24 hours up to 0000 UTC Wed 8 January 2003 from NMOC |
It is planned to implement the BEM scheme in MESO_LAPS_PT125 and MESO_LAPS_PT050 in the near future. Further tuning of BEM is underway in BMRC to try to improve the rainfall forecasts in the tropics. There are also plans to soon commence ingesting LAPS_PT375 model output into the MARS archive system.
Dare, R., "The BMRC Bulk Exp;icit Microphysics (BEM) Scheme " BMRC Research Report (in press) 2002.
McBride, J.L. and Ebert, E.E., "Verification of quantitive precipatation forecasts from operational numerical weather prediction models over Australia", Wea Forecasting 15 103-121, 2000.
