Home | About Us | Contacts | Help | Feedback | SEARCH Global | Australia | NSW | Vic. | Qld | WA | SA | Tas. | ACT | NT | Ant. | Weather & Warnings | Hydrology | Climate | Numerical Prediction | About Services | Learn About Meteorology | Registered User Services |

Global Guide
Main Menu

Global Guide to Tropical Cyclone Forecasting:
CHAPTER 3: TROPICAL CYCLONE MOTION

3.5 VERIFICATION METHODS

One of the more neglected components of motion forecasting is the routine preparation of best tracks and detailed forecast performance statistics at the end of each season. Whilst this is a time-consuming task, which uses valuable resources, it provides the best possible storm archive, together with invaluable information on overall forecast performance and the relative performance of individual techniques. Such information is essential for:

• Developing new techniques and monitoring forecast improvements resulting from new techniques, observing systems, and the like;
• Providing a basis for objectively estimating forecast uncertainty, which is used in deciding the timing and extent of warnings and watches, especially for critical regions;
• Providing feedback on the relative accuracy of available forecast techniques, perhaps including stratification into different synoptic, latitudinal, or seasonal types (eg Neumann and Pelissier, 1981);
• Providing feedback to external organisations, such as the global modelling centres, on their performance in the region.

3.5.1 Best-Track Preparation

The preparation of the best track is undertaken using the procedures outlined in Section 3.2. An additional major advantage at this stage is the knowledge of the whole storm track, so that the post-analysis can make good use of interpolating between past and future movement in deciding the optimal position. A major question for the analyst is the degree of detail that is to be included in the final track. For example, fine-scale, trochoidal oscillations can be resolved by radar data, but generally not in systems over the open ocean.

Recommendation: Best track archives should include the maximum possible detail. Although the detail will vary by observing system, this provides the maximum possible information available for later research and development use. Care needs to be taken, however, not to provide detail that is within the noise level of the observing systems, and thus largely one of analyst interpretation. As a general rule, 6 hourly positions provide a sufficiently detailed description of the track.

3.5.2 Estimating Forecast Error

Analysis and forecast errors can be developed from several perspectives. The most common are the absolute track error, but the across and along track errors also are used (Fig. 3.20). The absolute track error is the great circle distance from best track location to the operational analysis or forecast position. This must be a great circle distance, simply averaging the meridional and zonal components leads to error biases.

One essential component of the error analysis is to normalise all forecast errors by a standard technique. CLIPER was recommended for this purpose by IWTC-II, and an example has been provided in Fig. 3.15. Since CLIPER is based entirely on current position and motion and derives its forecast from past storms, techniques that do not improve on this have no real skill. This normalisation also allows a ready comparison between "easy" and "difficult" forecast years and different ocean basins (Pike and Neumann, 1987).

Figure 3.20: Illustration of the calculation of absolute and along/across track forecast errors (after JTWC, 1992).

The method of presentation of the forecast errors is one of personal preference. Histograms showing the frequency of errors in defined ranges provide a ready indication of the most frequent errors, together with the skewness towards large errors (eg Fig. 3.18). Plotting error components in scatter plots with either meridional and zonal (Fig. 3.21), or along and across track axes, provides very useful information on relative bias and clustering of errors (for example consistently slow techniques).

One curious anomaly in the approach used for defining forecast error at most operational centres is that errors are defined from the analysis time rather than the time of availability of the forecast. For a delay of an hour or so, this has little real effect. However, for a global model that is not available until 12 or more hours after analysis time, the real forecast period is reduced accordingly so that the 24 hour model forecast is only a 12 hour real forecast. The current approach is biased against simpler models, which may have slightly worse errors from analysis time, but because of their early availability are more accurate in real terms.

Recommendation: Forecast error statistics should be routinely prepared by all forecast offices and promulgated to the international community. The forecast period should reflect the time of forecast availability rather than the base analysis time.

Figure 3.21: Scatter plot of 24 h zonal and meridional forecast errors for Australian tropical cyclones with intensity greater than 970 hPa in the 1992-93 season (Woodcock, personal communication, 1993).

3.5.3 The Concept of Forecast Difficulty⁽¹⁾

Neumann (1981, 1985) introduced the concept of forecast difficulty to assess forecast improvements over the North Atlantic basin. The concept is based on the use of residual errors of CLIPER to provide a threshold skill level and a basis for determining forecast difficulty. These CLIPER errors are directly proportional to operational forecast errors; for example, both have lowest errors when tropical cyclones move slowly and along persistent tracks. The index of motion steadiness presented in Section 1.3.4.5 addresses persistency of direction, but it does not include a sensitivity to translational speed.

Thus, CLIPER is an excellent measure of forecast difficulty and normalising all forecasts by the CLIPER errors provides a homogeneous indication of forecast skill. IWTC-II recommended that all cyclone motion forecasts be normalised by concurrent CLIPER forecasts to provide a universal standard for comparison.

Pike and Neumann (1987) developed CLIPER models based on the same criteria for each of the tropical cyclone basins⁽²⁾ and compared residual errors (Fig. 3.22). Their Forecast Difficulty Level (FDL) varies widely, from a maximum in the Australian/southeast Indian and southwest Pacific regions to a minimum in the north Indian basin. This implies that, given similar information on the current and past positions of a cyclone, the most accurate forecasts can be expected in regions with low FDL. For example, mean 72-h forecast errors are 665 km for the North Atlantic, but only 460 km for the eastern North Pacific. After accounting for the FDL, forecast skill is comparable in the two basins.

Note that the FDL relates only to the intrinsic nature of the cyclone tracks. For example, no account is taken of operational factors such as the accuracy of centre fixes, which will affect the persistence forecasts. This qualification is particularly applicable to the north Indian basin, where cyclones are relatively weak and centre positions difficult to locate accurately. Moreover, the FDL does not address inherent warning problems, such as storm surge potential, communications to coastal populations and evacuation.

A further problem is that not all CLIPER techniques are equally skilful. When small data sets are used, or historical data have considerable uncertainty, for example, the CLIPER equations may be unstable.

Recommendation: Provided its limitations are understood, the FDL provides an excellent means of comparing ocean basins, different forecast techniques, and seasonal changes in forecast errors. We strongly recommend the use of FDL in all comparisons of forecast errors from different data sets, for different techniques and for long term forecast trends.

Figure 3.22: Forecast difficulty level vs. forecast period for specified basins (Pike and Neumann, 1987).

1. Provided by C. Neumann

2. Pike and Neumann combined basins 6 and 7 of Table 1.2 into a single basin.

Contents Chapter 4