Australian Satellite Data Archive

This document describes the Australian Satellite Data Archive (ASDA) format for storing satellite pass meta-data in the Parameter Value Language (PVL). Fields are classified as mandatory, recommended, or optional.

The contents of the PVL header is recommended to be:

Anything with a boldfaced name below is an actual PVL value — those with the keyword group are aggregation block groups.

Preamble

The preamble contains the ASDA Version number for this archive file.

ASDA_Version, mandatory
Supplies the version number and the date upon which the version was settled.
Example: "V1.0 March 1997"
ASDA_Information, recommended
Supplies information about the header and the designer(s).
Header_Contents, mandatory
Lists the sequence of entries in this archive file.
Example: (Format, HRPT_Data_Description)

Format Group

The Format Group defines the data storage format within the archive file. This is a mandatory field, as it tells a user or user program how to extract the data from the file bit-stream.

tape_physical_record, mandatory on tape systems
PVL integer of blocksize used on tape. It is the first actual field (after the version field) which contains data, and defines the tape blocking size. A user who is unsure of the block size will be able to read in the first hundred bytes or so and simply read off (in plain 7-bit ASCII) the actual block size to use.
Content Groups, mandatory
This group gives the size of each content block of the archive. It also gives an indication of the record size and record type of the data. The record type given should directly correspond to an entry in a data description group of the same name as the content group. For example, the HRPT data stream:
HRPT_Data group in the Format group indicates that a data block of HRPT data exists in the archive file and has records of length 13864 bytes which are formatted as HRPT_Line which is described in the HRPT_Data_Description group.

Example of Format Group

This example is for an archive file containing the PVL header and HRPT data which is unpacked.

begin_group = Format
 File_Contents = (PVL_Header, HRPT_Data);
 begin_group = PVL_Header;
  length = 65536 <bytes>;
 end_group = PVL_Header;
 begin_group = HRPT_Data;
  length = 72383944 <bytes>;
  record_size = 13864 <bytes>;
  record_type = HRPT_Line;
 end_group = HRPT_Data;
end_group = Format;

Per-Data Description Group

One of these groups exists for each data section type in the archive. For example, in a NOAA archive there should be a HRPT_Data_Description group and possibly a NOAA-1B_Description group.

Scene_Description Group, recommended
AVHRR_scene, recommended if polar orbiter scene
this is a PVL set containing the latitude and longitude coordinates of the top left corner and bottom right corner of the AVHRR scene
Example: AVHRR_scene = {(-10.3,140.1),(-45.3,150.3),(-9.6,142.1),(-45.2,154.3)}
unique_identifier, recommended
this is the composite foreign key that is used to look up the archive catalogue to determine archive file location.
Example: unique_identifier = "NOAA-14,RAW,1996-04-30T10:03:45Z,Hobart"
CEOS_inventory
160 characters, as per the Committee on Earth Observations Satellites (CEOS) Inventory Exchange Format (IEF), version 2.1, February 1992

Example:

N14AVHHRP 950502 034500 ?????? HBTHBT01731A ????????????? 0hrpt   16? 
Xc01mar961 -50.82 184.00-58.05 137.24-10.32 133.29-06.11 160.43A? 
???????????????????????2
Satellite Group, mandatory
This group defines the identity and state of the satellite during the time of image capture.
name, mandatory
The unique code for the satellite.
Example: name = "NOAA-19"
identity, mandatory
the identity (as given by ...) of the satellite
orbit, mandatory if polar orbiter
Orbit number of satellite when data was sensed defined at the acquisition time.
pass_direction, mandatory if polar orbiter
direction in orbit, "ascending" or "descending"
acquisition_start, mandatory
satellite clock at start of data acquisition
acquisition_end, mandatory
satellite clock at end of data acquisition
Navigation Group, mandatory
This group gives all available navigation information for the satellite.
Instruments Group, recommended
This group gives all available instrument information for the satellite.
Station Group, recommended
This group defines the identity and state of the station that captured the image.
name, recommended
the name of the station
identity, recommended
the identity (as given by ...) of the station
location, recommended
latitude/longitude coordinates of the receiver
station_clock_accuracy, recommended
description of the source of the station clock
Example: "XNTP daemon under UNIX"
acquisition_start, recommended
station clock at start of data acquisition
acquisition_end, recommended
station clock at end of data acquisition
Data Description Group, mandatory
The data description group has no formal structure. It should describe the data that is stored from the satellite. The description will mostly be straight ASCII (PVL Character Set) with a few tables. It is intended that this be used as a programmer or scientist reference. This description should include:
  1. A full data set name and/or reference to the document used to compile this description.
  2. The organisation of the data, be it by lines, by fields, by grid or whatever.
  3. The method by which the data was generated. References to the instruments as described in the "Satellite Group", "Instruments Group" above where appropriate.
  4. The storage format of the data. This should include the bit-field descriptions of data values where appropriate.

Example

The HRPT data contains information described in the HRPT_Line data description. A partial HRPT_Line data description group could be:

 begin_group = Data_Description;
  begin_group = HRPT_Line;
   size = 13864 <bytes>;
   elements = (pre_sync, identity, time, telemetry, back_scan, space_data, 
               sync, TIP, spare, AVHRR, post_sync, fill, error_codes);
   begin_group = pre_sync;
    elements = 10 <bits>;
    number_elements = 6;
    description = " first 60 bits from a 63-bit pseudo noise generator,
                      generator polynomial x6+x5+x2+x+1, start all 1's,
                      bit 1, element, 1 first";
    format = '1010000100 0101101111 1101011100 0110011101 1000001111
                0010010101';
   end_group = pre_sync;
   begin_group = identity;
    elements = 10 <bits>;
    number_elements = 2;
    description = 'minor frame number, spacecraft address and frame status';
    format = 'element 1(7)
                bit 1    0 = internal_sync   1 = AVHRR sync
                bit 2,3  00 = not used       01 = minor frame 1
                         10 = minor frame 2  11 = minor frame 3
                bit 4-7  spacecraft address  (4 = MSB, 7 = LSB)
                bit 8    0 = frame stable    1 = frame resync occurred
                bit 9,10 spare, bit 9 = 0    bit 10 = 1
                element 2(8)  spare';
   end_group = identity;
   begin_group = time;
    elements = 10 <bits>;
    number_elements = 4;
    description = 'time code day to milliseconds from spacecraft clock';
 .
 .
 .
Data_Quality Group, recommended
The Data Quality Group, at a minimum, has to define how the user of this data is to extract the data quality information from either the data stream or from this meta-data file.
description, recommended
Description of how to determine the quality of the data. For HRPT, this would describe what the contents of the Error Field in each line mean.
bad_lines, option
Number of lines that are determined by the ingestor to not be good. This entry is used typically when the data quality information is embedded within the data stream.
line_quality_table, option
The line quality table is used to store the data quality information externally to the actual data stream. Typical entries are for fly-wheeled (repeated by reception hardware - typically not received) lines and lines which are known to have corrupted data.

Version History

The following details the revisions of this document.

Draft version 2, 27 May 1996
Initial public release.
Draft version 3, 13 November 1996
Clarification of the Base Elements of the elements field of the Data Format, particularly with respect to binary data.
Draft version 4, 12 December 1996
Group discussion with P.J. Turner, B.A. Berzins, R.T. Jones and P.C. Tildesley found that the following needed to be rectified in the current format used:
  1. The boundaries between data storage and data format had been made less clear by the BoM extensions to the Format group.
  2. The overall structure of the ASDA format should be rearranged to allow multiple data formats within one archive file.
The following changes have been made:
  1. The Ancillary group has been replaced by the HRPT_Data_Description group (this corresponds directly to the name given to the raw data in the Format group plus the _Description tag). The Data_Description group has been moved into the HRPT_Data_Description group. Thus, there are two high-level groups now - the Format group and the HRPT_Data_Description group. If a SST product was included in the archive, the appropriate Format description would be added along with a top-level SST_Data_Description group describing the data.
  2. The BoM Format group extensions have been incorporated into the Data_Description group, allowing programmatical access to that information.
  3. A size parameter has been added to data format groups that consist of complex elements which is the total size of those elements.
Draft version 5, 18 April 1997
Version 1.0 of the archive header has been finalised with no major changes from 12 December. Some reformatting of the HRPT description entries was made.

C Tool to access ASDA Format Files

This tool allows a user to easily access information stored in an ASDA file:

  • display a given parameter (possibly within a given set of blocks)
  • display the data size of a given element or element group
  • convert the Format group to plain text/python source

The relevant files are available as a compressed tar file: asdaTool.tar.Z

Example of usage

When run on one of the archive files the Bureau produced, the following results were obtained:

% asda example_header.asda -d Header_Contents
(Format, Ancillary, Data_Description)
% asda example_header.asda -d ASDA_Version
V0.95 2 April 1996
% asda example_header.asda -s
260560696 bits/3.25701e+07 bytes (31 Mbytes)
% asda example_header.asda -s HRPT_Data HRPT_Line
110912 bits/13864 bytes (13 Kbytes)
% asda example_header.asda -s HRPT_Data HRPT_Line avhrr
102400 bits/12800 bytes (12 Kbytes)

Programmer API

The source is divided into two sections at the moment. The first section (asda_Display.c) handles the displaying of parameters from ASDA PVL files:

int asdaDisplay(char *filename, int argc, char **argv, FILE *out)
The filename specifies the ASDA file to read the information from. The argc and argv entries specify the blocks to open and the parameter to display (blocks in order followed by parameter) and the out file stream is the stream to display the information on.

The second section (asda_Format.c) handles the Format group of the ASDA PVL file:

long asdaFormatSize(char *filename, int argc, char **argv)
Returns the size in bits of the format group element specified in the argv array (the full path to the element must be given) from the ASDA file filename.
void asdaFormatText(char *filename, FILE *out)
Parses and displays the Format group structure from filename to the stream out. This is more an example of how to parse the Format group than anything actually useful. Currently though, the textual output format is Python source code.