Chapter 2 - Climate Data and Monitoring

The Climate Record > Climate Data and Data Management

Responsibility for the primary Australian climate database rests with the National Climate Centre (NCC) of the Bureau of Meteorology. Various other sections of the Bureau of Meteorology manage data related to climate (e.g. ozone, subsurface oceanographic measurements, imagery and other radiative information from satellites). Other national archives incorporating climate data are maintained by a number of other agencies. These include:

• The Australian Oceanographic Data Centre, which is formally responsible for archiving and distributing Australian oceanographic data,
• The National Tidal Facility of Australia, which is responsible for the maintenance of extensive sea level data sets from Pacific, Indian and Southern Ocean sites and from the ASEAN region; and
• The Australian Antarctic Data Centre which maintains a comprehensive archive of data associated with Australia’s Antarctic Science Program.

The NCC maintains the climate database, known as the Australian Data Archive for Meteorology (ADAM), through the development and maintenance of data ingestion, quality control, storage and access systems. Several climate data variables are archived within ADAM including observations of temperature, wind, atmospheric pressure, rainfall, rainfall intensity and humidity, from surface, upper air and marine observing systems. Most data are ingested electronically through the Bureau of Meteorology’s communication systems but some require manual entry. Hourly, daily and monthly summaries are stored, along with some international surface and upper air messages from other National Meteorological Services. Since 2001, the main areas of development in climate data management in the NCC have been the:

• completion of the Climate Archive Project (CLIMARC) project to computerise pre-1957 daily and hourly climate records for 50 stations across Australia;
• development of systems to manage solar radiation data from in–situ and satellite sources;
• ingestion and management of one minute data from a set of automatic weather stations (Figure 2.15), followed by the commencement of the ingestion of pre-2003 data held in various holdings within the Bureau of Meteorology;
• refinement of a more accurate and more portable system for digitising rainfall intensity traces from pluviograph charts;
• dispatch of marine data from real time ship reports to the WMO for inclusion in the Marine Climatological Summaries Scheme; and,
• digitisation of aviation data from manuscripts (METARS), which commenced in late 2002.

Figure 2.15 One-minute temperature (dark line) and maximum wind gust data on a hot summers day in Melbourne. These data are now being routinely archived in the national climate database (ADAM).

Increased size of data holdings and increased complexity of analysis and extraction processes meant that extra processing power was needed, and ADAM hardware was replaced in 2001 and 2002 with new systems providing at least a tenfold increase in performance. The Redundant Array of Inexpensive Disks RAID) storage disks, which have been in use since the database first became operational in 1994, were replaced with a Hewlett Packard X256SAN (Storage Area Network) to secure a higher degree of reliability and performance from the database. Upgrades to ADAM software over the past two years have also continued to improve the accessibility of the data.

Staff from the NCC promote the use of climate data, climate databases and standards for climate data management through a range of opportunities. For example, the NCC provided the address Changing Technologies in Climate Data Management" at the WMO Commission for Climatology Technical Conference in November 2001.

Data Quality

An increased focus, in recent years, on improving data quality has led to a wider and more integrated handling of data quality issues within the Bureau of Meteorology. While most operational data quality control occurs within the NCC, quality monitoring activities also occur in other areas of the Bureau of Meteorology. Liaison between the relevant areas, and also with the Bureau of Meteorology’s Regional Offices, is an essential ongoing activity.

Several ongoing projects are targeted at improving data quality, including development of a Quality Monitoring System (QMS), which will provide a range of improved statistical and graphical tools to help detect and correct problem climate data. With the increasing demand for quality controlled data in near real time, these projects will assist with the development of more automated quality control procedures that not only detect suspect data but also make skilled corrective estimates. The QMS is a collaborative initiative between the NCC and the Bureau of Meteorology’s Observations and Engineering Branch which is responsible for installing and maintaining observation stations. The system will provide valuable performance information on observation networks, which should help improve their management and ultimately the data provided by them. The increasing volume of data and number of observational platforms are another reason why automated quality control procedures will be required. However, manual quality control will still be necessary for the production of high quality datasets and for the processing of historical data that still have to be entered into the electronic database.

The increased automation of the observing network in recent years has also led to the introduction of a greater range of "at source" quality control procedures. Electronic field books, for example, have been widely implemented throughout Australia, providing basic quality control of the observations in real time. Further steps have been taken to reduce data losses from automatic weather stations such as establishing systems for the routine recovery of missed messages stored in memory. Also, the software used within automatic weather stations (AWSs) now includes a data filter to reduce the number of erroneous "data spikes" being reported. This same software upgrade enables stand-alone AWSs to transmit the Met Data Format (MDF) which, unlike the previously used message format, allows maximum and minimum temperatures to be reported with standard precision and time intervals. Access to these data will assist the quality checking of AWS data. AWS performance monitoring systems have been established to help identify calibration errors, missing data and communication problems. These systems form part of a comprehensive station metadata database that aids the quality assurance process of AWS data. Further, feedback on Australia’s GCOS Surface Network (GSN), provided through the reports of the GSN Monitoring Centre, has led to improved reporting of monthly CLIMAT messages from Australian stations.

Metadata

There has been increased recognition in recent years of the importance of maintaining metadata, which is, in effect, data about the data and includes such information as how an observation has been made, on the instruments, site details and observation practices. Recognised as an important part of the Bureau of Meteorology’s climate record in their own right, metadata are archived, in addition to the actual climate data, in a specially designed metadata database (SitesDb) within ADAM. Recent metadata have assisted in producing an updated and improved version of the Australian high-quality annual mean temperature dataset. Work has progressed to incorporate significant older metadata into the database and to trial the electronic imaging and storage of material held on paper files. This latter material includes site diagrams and photographs, some of which are deteriorating. The electronic imaging will preserve the information they contain, and will also provide easier access for researchers. With undigitised station history files currently numbering more than 22,000, the building of the metadata database is expected to take many years to complete.

Other initiatives on metadata include: the development of site summary products, where the essential information (including changes) on an observation site is condensed to a few pages suitable for data users; an audit of historical operating procedures and practices, and archived equipment specifications; and documenting the historical changes in the observation of temperature (with similar work on rainfall observations expected to follow).

CLIMARC

The CLIMARC project commenced in 1999 and was completed in 2002. CLIMARC developed from a concern that only a small amount of pre-1957 daily and hourly data had been digitised for use in computers and archived in the Bureau of Meteorology’s climate database. The project provided for the processing of daily and hourly climate data for 51 key locations (64 different sites) across Australia (Figure 2.16) followed by their quality control and storage in the climate database (ADAM). The stations were selected especially for their relevance to agriculture and usefulness for climate applications and research (including the length of their data series). The earliest data entered was from 1858 for Bathurst Gaol, NSW.

The CLIMARC data will provide valuable inputs to crop simulation and seasonal climate prediction models and will extend our knowledge about changes in climate extremes. The Queensland Departments of Primary Industries (QDPI) and Natural Resources (QDNR) collaborated with the Bureau of Meteorology on CLIMARC, with the principal funding being provided by Agriculture Fisheries and Forestry-Australia and Australia's rural R&D Corporations under the Climate Variability in Agriculture Program. To cater for the many changes in observing practices and in the units of measurement that have occurred since observations began in the late 1800s, the CLIMARC project developed advanced data entry and quality control procedures. As opportunities allow, the skills developed in the CLIMARC project will be used in future efforts to digitise historical data from other stations, as much data still remain only in paper form. A similar initiative is now underway for aviation reports (METARS).

Figure 2.16 Map showing the 51 locations where historical climate records were entered into ADAM through the CLIMARC project.

National Archives

Australia Project Part of the Bureau of Meteorology's data archiving function relates to the management of historical paper archives (books, forms, charts, letters, maps and graphs). The retention of these paper records is a crucial issue for the integrity of the Australian climate record, given that the Bureau of Meteorology’s digital archives are far from complete. Until recently, most of this material was held by National Archives of Australia (NAA) in facilities across Australia. In recent years the NAA has been under pressure to rationalise its facilities and reduce the overall volume of its holdings. This has required the Bureau of Meteorology and other organisations with large volumes of records to review and reduce their holdings and develop strategies to manage them differently.

Following several reviews, the Bureau of Meteorology identified non-essential records that could be destroyed and material that could be captured electronically. Work is progressing on the production of scanned images of this latter material. The project aims to preserve the images as digital files within Bureau of Meteorology data bases and as microfilm in NAA offices, and to eliminate the original paper records in NAA holdings. Imaging of these important scientific records will also help ensure that they are more easily accessible to Bureau of Meteorology staff and to the wider community. In other cases, consideration is being given to alternative storage arrangements for material that does not constitute primary records but nevertheless has historical significance. During 2003, alternative off site storage arrangements and/or electronic imaging will be arranged for some other material that is currently held within the NCC.