About the water data transfer format (WDTF)

What is WDTF?

WDTF is an XML data transfer format designed to support the water industry to share and deliver water data to the Bureau of Meteorology as required under the Water Regulations 2008.

The current version of the WDTF supports data for Regulations categories 1, 2, 3, 4, 5, 7, 9, 10a and 10b.

See also: What is XML

The format has been developed through the Water Information Research and Development Alliance (WIRADA).

Why was a data transfer file format required?

In July 2008, the Bureau of Meteorology was scheduled to receive data from more than 250 organisations listed in the Regulations. At the time there wasn't any national or international format standards which could be used. We found that the data providers were using various types of formats ranging from mature in-house or commercial data management systems to adhoc spread sheets configured for a combination of data storage and reporting purposes. Segments of the water resources monitoring industry with common data storage products were able to exchange data with common formats, but the data meaning was not consistent between the organisations.

The variability in data formats coupled with the potential for format instability presented significant challenges for the Bureau when it came to using the data. These issues also meant that data interchange between organisations and researchers was going to be problematic, which required project specific bespoke data manipulation solutions. The Bureau identified that a data transfer format was need to support the transfer of regulations data — this format is known as the water data transfer format (WDTF).

Existing data transfer standards and formats.

The Bureau has an existing format for Hydrological data transfer, this format is called 'Hydrological comma separated' (HCS). This format serves the Bureau well in the established and mature flood warning role but was limited in its ability to support the full range of data being asked for by the regulations.

In September 2007 a 'Water Resources Information Model workshop' was held in Australia with participants from Australia, New Zealand, USA, Canada, Netherlands and the UK. This workshop identified the need to develop standards based information model and transfer formats for water.

The Bureau established a Water Industry Research and Development Alliance (WIRADA) project to investigate existing national and international activities in the area of hydrological data transfer and establish standards based direction for hydrological data transfer. The project reviewed a number of existing formats and standards, adopting the Open Geospatial Consortiums 'Observations and Measures' standard as a basis for establishing a data transfer format.

WDTF initial development

The scope of data being transferred for WDTF was derived from the regulations. The regulations requirements were reviewed with industry representatives to establish the data requirements for implementation in a transfer format. During this process input was sought from representatives from CSIRO, Greenspan, Kisters, Rubicon and Stephen Rowe and associates. A number of content workshops were held to define the data content of WDTF. The data content was then encoded into XML through the WIRADA project. The XML encoding is based on a GML Simple Features (GML- SF) implementation of the Observations and Measurements (O&M) standard. GML- SF is a profile which only uses a subset of GML, reflecting the most common WFS (Web Feature Service) use of GML3.

See also: Complement to review of Water Data Standards.

Walker, G. , P. Taylor, S. Cox and P. Sheahan (2009). Water Data Transfer Format (WDTF): Guiding principles, technical challenges and the future. In Anderssen, R.S., R.D. Braddock and L.T.H. Newham (eds) 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation. Modelling and Simulation Society of Australia and New Zealand and International Association for Mathematics and Computers in Simulation, July 2009, pp. 2377-2383. ISBN: 978-0-9758400-7-8.

Future directions

Components of WDTF address similar requirements to existing standards. Work is being completed to harmonise these efforts using a standards based, model driven approach. The design of WDTF is compatible with this model driven approach. The WDTF data requirements are a superset of those being considered in the harmonisation activity. Once completed, the harmonised components will be integrated into WDTF. It is anticipated that this work will be the basis for release version 2.0.

The harmonisation work is being completed through the OGC's hydrology domain working group (Hydrology DWG). The Hydrology DWG is a forum for collaboration and development of standards for hydrological data. The purpose of the Hydrology DWG is to provide a venue and mechanism for seeking technical and institutional solutions to the challenge of describing and exchanging data describing the state and location of water resources, both above and below the ground surface. The path to adoption will be through OGC papers and standards, advanced to ISO where appropriate, and also through the World Meteorological Organization's (WMO) and it's Commission for Hydrology (CHy) and Water Information Systems (WIS) activities.

WDTF release dates

• WDTF package v0.1 released — 26 August 2008
• WDTF package v0.2 released — 22 September 2008
• WDTF package v0.3 released — 24 March 2009
• WDTF package v1.0 released — 30 October 2009