Information Delivery

How do I give specified water information to the Bureau?

Information delivery methods are different for Historical Information and for New/Ongoing Information

How do I deliver Historical Information

CD, DVD or on an external storage device. The Regulations also provide for historical information to be given by File Transfer Protocol (FTP) or via Web Services. However, given the potentially large file sizes, we advise against FTP. Each zip file sent via FTP should not be greater than 30MB. The Bureau does not currently have the operational capacity to poll Web Services, please alert us if you are considering giving water information via Web Services so we can discuss the arrangements for this process.

Address to send Historical Information on CD or DVD:

Manager Water Data
Water Data Management Branch
GPO Box 2334
ACT 2601

For a definition of Historical Information, see Historical Information and New Information.

How do I deliver Ongoing/New water information

For a definition of Ongoing/New Information, see Historical Information and New Information.

Time Series Data and Meta-data

Time Series Data and Meta-data should be provided by FTP or via Web Services. The Bureau does not currently have the operational capacity to poll Web Services, please alert us if you are considering giving water information via Web Services so we can discuss the arrangements for this process.

Water Information that is not Time Series Data

This information should be provided by FTP. However, where information is required at discrete and widely spaced time intervals, such as at the end of each financial year it can be given by CD, DVD or on an external storage device.

 

File Transfer Protocol (FTP)

The Bureau of Meteorology has registered all persons named in the Regulations on its FTP service, to support the giving of regular water information (hourly, daily, weekly, monthly, annually), as required in accordance with the Regulations under the Water Act 2007. This service will allow you to copy and transfer the water information files you are required to give to the Bureau of Meteorology on an ongoing basis.

Each agency has the following Folder structure:

All incoming data files whether time series, metadata or other must be contained in zip files for FTP transfer. Zip files should be uploaded directly to the data, metadata, or timeseries folders, as above. Please do NOT attempt to create subfolders within these folders. The file transfer mode of the FTP client must be set to binary. The total size of the zip files transferred in any one hour period should not exceed 30MB, or the upload may fail.

The zip files should be named using only alphanumeric, hyphens, and underscores in filenames, i.e. no spaces or metacharacters. (See Data file content and naming conventions for information on how to name the data files within a zip file.) Zip file names should be chosen to be unique for each transfer, e.g. by encoding date and time or sequence number in the name, unless the data is intended to replace previously supplied data, in which case the previous name should be reused.

To access the service you will need to connect to the internet. Your organisation will receive by email the User Id and password which you will require to access the service.

The FTP service is available 24 hours per day, every day.

More information about the Bureau of Meteorology's FTP service can be found at http://www.bom.gov.au/reguser/ under FTP Access procedures.

Windows Explorer can also be used as a simple FTP client to connect to the Bureau of Meteorology's FTP service.
Download FTP instructions.

Supply of data to the Bureau by organisations who outsource data collection

Some organisations named in the Water Regulations 2008 employ service providers such as contractors or other agencies to collect and store data on their behalf. With few exceptions, this information is defined under the regulations as being in the named organisation's control, although it may not be stored in their database. The named organisations are therefore required to ensure its provision to the Bureau. In the case where information is being sourced from a third party service provider, the Bureau can, on instruction from the named organisation, provide a separate FTP login to the service provider. The service provider can then supply data into their own FTP account on behalf of a named person. Please contact us if this situation applies to your organisation.

After you have uploaded a zip file

After you have uploaded one or more zip files to the Bureau's FTP server, the data files will be processed by Bureau systems. The process will assess data files and will either accept (import) or reject the files. The actions completed are recorded in log files that are on the FTP site. The log files are the method of confirming acceptance or rejection of data files with data providers.

The Bureau's FTP data retrieval and receipt process.

• The FTP folders are scanned and uploaded files are examined once every hour. Any files in the middle of being transferred are ignored until the next hour.
• If a file is in the correct format and is a valid zip archive it is accepted for further processing and moved from the upload folder.
• If a file is not in zip format or is not a valid zip archive it is rejected, and deleted from the upload folder.
• For each accepted file, a line will be added to an 'acceptedYYMM.txt' logfile that is created within the upload directory.
• For each rejected file, a line will be added to a 'rejectedYYMM.txt' logfile that is created within the upload directory.

About FTP data delivery log files.

• Logfiles are a cumulative record of all the files accepted and rejected for a given year (YY) and month (MM).
• The current month logfiles may be deleted without loss as they will be recreated the next time any files are processed, however a previous month's logfiles will no longer be recreated after a new month has begun.
• Data suppliers should delete log files from the FTP site once they are no longer required.
• Where the accumulation of log files becomes excessive the Bureau may delete older log files. In the event that the Bureau deletes old log files we will not delete the current month and previous month's files from the FTP site.
  

 

Data Format

WDTF

The Water Transfer Data Transfer Format (WDTF) is our preferred mode of data delivery and we are promoting this as the standard format for the transfer of water data. This format will facilitate the transfer of Time Series data, and Metadata.

WDTF and Commerical Systems

There are a number of commercial companies that have produced export tools to facilitate the forwarding of data to the Bureau. Organisations wishing to update their data manangement systems to support WDTF can apply for funds from the Bureau under the Modernisation and Extension of Hydrologic Monitoring Systems Program.

Non time series data Format

There is no specified format for water information, other than time series data and metadata.

When you fill in the Data Survey please describe the systems in which you currently store that information. An example of water information that is not time series data is Information Sub-Category 3f.

Data file content and naming conventions - ad hoc data formats

To assist with the ingestion of data into Bureau systems, there are some basic guidelines for data file content file name structure.

Data file content

It is theoretically possible to deliver all data relating to many monitoring stations in a single file. However the Bureau has set a constraint that a data file should only contain data for a single site and the data for that site be split across two transfer file types. If required, data of each file type can be provided independently of the other. The two file types and their contents are described below.

Data file naming

Data files are named to produce unique file names that inform of their source and contents. The file name is constructed by the concatenation of several elements, in order. File names are considered to be case insensitive.