Darwin Regional Specialised Meteorological Centre
(Darwin RSMC)

UPGRADE OF THE GLOBAL ANALYSIS AND PREDICTION (GASP) SYSTEM

The installation of the NEC SX-4 supercomputer has enabled experimental development and evaluation of a substantially higher resolution GASP system. The GASP T79 system was implemented operationally on the NEC SX-4 on 18 December 1997 with some modest upgrades in both the analysis and model components. The present operational upgrade has been the research and development focus of the BMRC Medium Range Prediction Group through 1998. Dr Xieng-Kun Meng has been chiefly responsible for operational implementation within NMOC.

Horizontal Resolution Upgrade

The horizontal resolution has been increased from T79 to T239 representing a change in the smallest equatorial half wavelength from 250 to 80 km.

Figure 1. Computational grid for the GASP T239 system

Vertical Resolution upgrade

From the 19 level sigma = p/p* vertical resolution (where p* is surface pressure) of

0.991, 0.950, 0.900, 0.850, 0.800, 0.750, 0.700, 0.600, 0.500, 0.400,

0.300, 0.250, 0.200, 0.150, 0.100, 0.070, 0.050, 0.030, 0.010

to the upgraded 29 sigma = p/p* level vertical resolution of

0.991, 0.975, 0.950, 0.925, 0.900, 0.875, 0.850, 0.800, 0.750, 0.700,

0.633, 0.566, 0.500, 0.433, 0.366, 0.320, 0.290, 0.260, 0.230, 0.200,

0.170, 0.140, 0.110, 0.090, 0.070, 0.050, 0.030, 0.020, 0.010

The vertical domain is still bounded by the two level values of sigma = 0.991 and 0.010. The increase in resolution is focussed at pressures greater than 850 hPa approximately, across the tropopause (approx 30 hPa level spacing) and with a modest increase in stratospheric resolution.

Numerical methods

Both the GASP prediction model and the analysis system now are implemented at a resolution of T239/L29; the nonlinear incremental normal mode initialisation continues to be applied after every analysis step but is only applied up to wave number T79 for the first 3 vertical modes.

The model now incorporates a significant number of options, some of which are being exercised within this present upgrade. A key upgrade is the implementation of a semi-implicit semi-Lagrangian time stepping algorithm; this allows the use of a time step of 600 secs at the T239 resolution which was in fact the time step permitted with the previous semi-implicit Eulerian at T79. The semi-Lagrangian scheme is a three-dimensional scheme with a non-interpolating algorithm in the vertical direction. With the semi-Lagrangian scheme the resolution of the dynamics grid is defined by a linear criteria as opposed to the quadratic criteria of the Eulerian model; consequently T239 horizontal resolution of the model utilises 240 Gaussian latitudes and 480 equi-spaced longitudes. The resolution of the grid on which physical processes are modelled is similarly this 240 x 480 grid. (The GASP T79 model has been using a 120 x 240 grid resolution for both the dynamics and physical processes; so for T239 the grid point processes are modelled on a grid of twice the resolution with the dynamics modelled at a spectral resolution which is tripled.) The grids used in the model are termed thinned grids; here the number of points around a latitude circle is reduced such that east-west grid point spacing is kept approximately constant as the grid approaches the polar latitude; this has a negligible impact on precision of model integration but provides a significant saving in computation time.

The resolution of the model dynamics and physics grid (with thinning) is shown in Figure 1; the spectrally represented model topography in the Australian region and over the globe are shown in Figures 2(a) and (b) respectively.

Figure 2a Australian region topography

Figure 2b Global topography

Analysis Component

The analysis component of the GASP system is the recently coded implementation of the Multi Variate Statistical Interpolation (MVSI) scheme as first implemented operationally in Dec 1997. (A design aspect of the MVSI framework within the GASP system has been to support the unification of the LAPS regional assimilation software.) The new version of MVSI is in fact generalized and in its full implementation will permit, for example, direct assimilation of radiances from the TOVS instruments on the polar orbiters; in the present upgrade we are initially continuing to use the NESDIS provided temperature and moisture retrievals. The observational data handling of MVSI system has been extensively upgraded and now allows use of for example TOVS data at pressures < 10 hPa, a prerequisite for proper implementation of radiance retrievals.

The analysis is now performed on the 240 x 480 (unthinned) Gaussian grid with a reduced prediction error specification (approximately halved), relative to the GASP T79/L19 system, which has the effect of giving the model 6 hour first guess higher weight. The amended prediction error specification has been based on the recent statistics of comparisons between observations and six-hour forecasts. A reduction in the scale of the horizontal covariance function is planned but not implemented as yet. The horizontal covariance function remains as at T79 with a horizontal scale given by the Gaussian covariance function with a scale of 500 km.

There have been several changes in the observations used by the analysis. The major one has been the use of the 1000-850 hPa layers from NESDIS TOVS retrievals of temperature and moisture over the sea, along with a sea-ice check. Over sea-ice the TOVS temperature retrievals are only used only at pressures below 300 hPa. Previously TOVS temperature and moisture retrievals were used only above the 850 hPa level over both sea and ice. The use of TOVS temperature data only at pressures above the 100 hPa level over land is unchanged. The observation data files have been upgraded to allow more information to be presented to the analysis, and all reports within a six hour window are included; previously, if there were multiple reports from ships, drifting buoys or automatic weather stations only the latest observation was available in the observation files. With the new scheme, the observation closest to the analysis time is selected from the file for use by the analysis.

The analysis continues to be performed over a series of regions, or analysis subvolumes, approximately 1000 km square, using data selected from a radius of 1500 km. Whereas previously a maximum of 500 observations were chosen and 90% of the observations had to be within the analysis subvolume, the new scheme uses up to 950 observations and only 50% have to be inside the subvolume. Furthermore, the subvolumes now overlap by 1/3 to 1/2 of their width, rather than abutting each other. These changes make a significant reduction in the high frequency noise that could be possible in the T239 system from simply using a sequence of abutting local analyses to produce a global analysis.

Characteristics of the system

One notably different aspect at T239 is the more finely resolved precipitation available from the model predictions. Figure 3 shows a recent example of observed rainfall to 9 am 26 November 1998 matched with corresponding predictions from the T79 and T239 GASP predictions using the output from the rainfall verification system developed by Beth Ebert in BMRC.

Figure 3 Precipitation verification for T79 and T239 GASP

(The top left hand panel shows the operational analysis of 24 hour rainfall accumulation to 9am on 26 November; subsequent panels show GASP T79/L19 24 hour precipitation accumulation for this valid time from 1, 2 and 3 days prior and GASP T239/L29 24 hour precipitation accumulation for this valid time from 1, 2,3,4 and 5 days prior. The T239 predictions are labelled HRGASP).

A corresponding satellite image at this same time is shown in accompanying Figure 4. There is clearly a more detailed representation of the precipitation patterns in the higher resolution GASP system that is consistent with both the rain gauge analysis and the satellite imagery.

Figure 4 GMS-5 IR imagery for 2300 UTC 26 November 1998

Predicted satellite brightness temperature fields for the new operational GASP T239/L29 forecasts continue to be available operationally. The scheme includes the spectral response functions which now encompass all infrared channels on GMS-5, GOES-8/9 and METEOSAT. Apart from inter-calibration problems and a sparsity of data, it is now possible to validate the cloud/water vapour fields on a close to global basis. The 72 hour Brightness Temperature predictions valid for 2300UTC 26 November 1998 are shown in Figure 5 for both the T79/L19(5a and 5b) and T239/L29(5c and 5d) GASP systems. These can be compared with the precipitation figures and satellite imagery of Figures 3 and 4.

Figure 5a T79L19 Brightness Temperature analysis

Figure 5b T79L19 Brightness Temperature 72hr prediction

Figure 5c T239/L29 Brightness Temperature analysis

Figure 5d T239/L29 Brightness Temperature 72hr prediction

Performance of the GASP T239L29 system in parallel operational trials

(a) Objective Verification

The full T239/L29 GASP system has been running in parallel to operations since late July 1998 and a range of issues has been addressed and resolved within this period. The system as it now exists is considered to be robust and to be delivering generally improved performance relative to the GASP T79/L19 system. A comparison of a range of quantitative verification statistics of the GASP system at the two resolutions of T79/L19 and T239/L29 for the period November 9 1998 until November 29, 1998 (42 cases) is shown in Figure 6. Here the verifications for each system are relative to their own assimilation analyses and include both the 2300 UTC and 1100 UTC based forecasts for this period.

(b) Subjective comparison of synoptic performance.

The following summary of the characteristics of the T239 predictions was compiled by Terry Skinner and is based on forecasts from 9 November onwards, when the trial configuration of T239 GASP was finalised.

Analyses:

• prior to smoothing the analyses can be noisy especially in the tropics.
• tropical cyclones (e.g. Billy and Thelma) have been poorly represented (as is the case in T79 GASP).

Predictions:

The following comments refer to 5 day predictions compared with NMOC manual analyses:

• the model tends to over-deepen lows in the westerlies over the Southern Ocean as well as over the Tasman Sea
• phase properties of troughs in the westerlies are generally good
• there is a tendency to over-deepen easterly dips over Western Australia and the continental interior, but there have been some good forecasts of easterly dips over South Australia and NSW.
• the WA heat trough is generally well-handled
• anticyclones to the south of Australia tend to be too strong (i.e. there is spurious ridging to the south)
• good capability in forecasting the interaction of easterly dips over the continent and the westerlies
• in cases of anticyclones over eastern Australia or the Tasman Sea there is a tendency for a positive pressure bias over Queensland, and the Queensland ridge/trough system is retrogressed with too weak an amplitude in the inland trough
• some MSLP fields are noisy, often associated with forecast precipitation, when plotted at high resolution, but the noise does not lead to "blow-ups" in later forecast intervals

• the model does not capture the intensity of tropical cyclones.

Rainfall prediction:

• areas of light rain are far too extensive, as in the T79 GASP
• broad-scale precipitation systems including frontal rain are generally well-defined
• peak heavy tropical convective rainfall is underestimated, although less so than in T79 GASP
• topographically associated rainfall rates are underestimated e.g. west coast of Tasmania or north coast of Queensland
• tendency to over-forecast large areas of heavy rain in non-frontal situations

Figure 6 Comparison of objective verification scores for the GASP predictions at T79 (labelled GASP) and at T239 (labelled GASP25). The comparison is based on 42 cases comprising 00 and 12 UTC predictions for the period 00 UTC 9 to 1200 UTC 28 November 1998.

Figure 6a (ii) S1 Skill Score for the Australian region for 500 hPa geopotential height.

Figure 6a (iii) S1 Skill Score for the Australian region for 200 hPa geopotential height.

Figure 6b (i) Anomaly correlation for the Southern Annulus region for MSL

Figure 6b (ii) Anomaly correlation for Southern Annulus region for 500 hPa geopotential height.

Figure 6b (iii) Anomaly correlation for Southern Annulus region for 200 hPa geopotential height.

Figure 6c (i) Anomaly correlation for the Northern Annulus region for MSL

Figure 6c (ii) Anomaly Correlation for Northern Annulus for 500hPa geopotential height.

Figure 6c (iii) Anomaly Correlation for Northern Annulus for 200 hPa geopotential height.

Figure 6d (i) RMS error for the Tropics region for MSL.

Figure 6d (ii) RMS error for the Tropics region for U(zonal) wind 850 hPa.

Figure 6d (iii) RMS error for the Tropics region for U(zonal) wind 200 hPa.

Product Availability:

Most output products will continue to be available in the same formats and at the same resolution. Due to limitations of space in the real-time database the T239 products have been post-processed to the same 2.5 by 2.5 degree resolution, which is the resolution at which the overseas global models are available. It is planned to add higher resolution fields at more frequent output intervals, particularly those showing high spatial variability such as precipitation and surface parameters, as database resources permit. The grid files output for external access are also provided at this resolution although higher resolution files can be considered as a product in the future.

Charts on DIFACS are produced from the real-time database grids, except for precipitation which is derived from the original model output file with light smoothing applied. Precipitation, with its high spatial variability, is a field showing significant impact from the higher resolution. These charts and other colour charts are viewable through the internal Web on the communications server at http://comms.ho.bom.gov.au.

COMPUTATIONAL ASPECTS of the GASP T239/L29 system:

The implementation of the GASP system at T239/L29 has been possible through the significant computational capacity now available for research and operations in the Bureau. The installation of the NEC SX-4/32 supercomputer in the HPCCC facility has provided the capacity to operationally implement such systems and importantly the capacity to run the extensive parallel testing necessary for operational development.

The SX-4/32 has 32 processors with 8GBytes of shared memory; the computational aspects of the system at T79/L19 and T239/L29 on the NEC SX-4 are compared in the following:

The optimisation of the GASP system software has been a substantial aspect of recent work; a detailed report on this is available from Gerry Embery of BMRC. Figures 7(a) and (b) show

1. the Gflops obtained for the prediction model at T239/L29 as a function of the number of CPUs used; and
2. the speed up as a function of increasing number of CPUs used.

Individual CPUs have a peak rating of 2 Gflops per processor; with the existing levels of parallelism in the model of 98.5% the use of 16 processors provides effective wall-clock time and efficient overall use of the machine. The MVSI analysis component of the GASP system has been multitasked (although not as efficiently as in the case of the model) and at present 8CPUs are used. The present GASP system utilises approximately half of the available resources in its operational configuration.

Figure 7a Gflops as function of the number of CPUs used

Figure 7b Speed up as function of the number of CPUs used

Future Developments

Some enhancements to the GASP T239/L29 system currently being investigated include:

• Implementation of physical retrievals from TOVS radiances with a particular focus on the use of observations from the Advanced Microwave Sounding Unit (AMSU) on NOAA-15
• Use of ERS-1 scatterometer winds within the GASP analysis system with the lowest model and analysis level relocated nearer the surface
• Refined radiative transfer code with more sophisticated cloud optical properties
• Raising the top model level to improve the representation of the stratosphere and provide a better vertical profile for use in physical retrievals from TOVS radiances
• Development of the tangent linear and adjoint of the adiabatic component of the GASP code, enabling the generation of singular vectors to be used in generating the initial conditions for ensemble predictions.

1d VAR 1 dimensional Variational analysis

BMRC Bureau of Meteorology Research Centre (Australia)

BoM Bureau of Meteorology (Australia)

CRC Co-operative Research Centre

ERS-1 European Remote Sensing

GASP Global Assimilation and Predication

GFDL Geophysical Fluids Dynamics Laboratory

GMS Geostationary Meteorological Satellite

GOES Geostationary Operational Environmental Satellite

HPCCC High Performance Computing and Communication Centre

IR Infrared Red

LAPS Limited Area Prediction System

MSL Mean Sea Level

MVSI Multi-Variate Statistical Interpolation

NESDIS National Environmental Satellite, Data, and Information Service

NOAA National Ocean Atmosphere Administration

TOVS TIROS Operational Vertical Sounder

UKMO United Kingdom Meteorological Office