Changes to solar data

Recent changes to the various solar data products provided by the Bureau are summarised below. However, changes to individual observations which occur as part of the Bureau's ongoing quality control process are not listed. See also Updates to One Minute Solar Data. Please read the Bureau's Disclaimer which is relevant to this information.

An attempt has been made to broadly classify the change in terms of an estimated magnitude of the change. However, it is your responsibility to review this information to determine the significance and impact for your work.

  1. H
    High degree of change; affecting the majority of data and/or a change in magnitude that is not associated with routine adjustments.
  2. M
    Medium degree of change; a large number of data changed and/or which may include reprocessing which is part of normal operation.
  3. L
    Low degree of change; a relatively small number of data changed, and/or which includes periodic changes as part of normal operation. This also includes notifications of general interest.
Subscribe to an RSS feed to be notified of updates to this page.

Please contact us if you require further information about these changes.

Recent changes to solar data

M
2016-Nov: Affecting product IDCJAD0111
New hourly satellite-derived solar irradiance data to 31 Oct 2016 now available.

General

The gridded hourly global horizontal irradiance (GHI) and direct normal irradiance (DNI) dataset has been updated to include data to the end of October 2016. The associated GHI metadata and DNI metadata have been revised to include new observations provided from the Japanese Himawari 8 satellite.

H
2016-Jan: Affecting product IDCJAD0111
All hourly satellite-derived global and direct normal solar irradiance data have been updated.

General

Enhancements have been made to the full dataset of gridded hourly global solar irradiance (GHI) and gridded hourly direct normal solar irradiance (DNI). These enhancements include the provision of 24 grids per day (up from 18); night time values no longer indicated as missing values; and improved correction of data to remove bias errors associated with the conversion from satellite image to irradiance value. Previously, the Bureau product code IDCJAD0111 refered to both data sets combined. From this point there will be a unique code for each parameter: IDCJAD0026 for gridded Hourly Global Solar Irradiance, and IDCJAD0027 for gridded Hourly Direct Normal Solar Irradiance.

Technical discussion

A number of changes have been made to the dataset to improve accuracy and useablity of the data. These are described below, and in the IDCJAD0026 GHI metadata and IDCJAD0027 DNI metadata documents.

  • Files are supplied for 24 hours of each day. Previously files were supplied only for the 18 hours of a day for which some region of Australia is in daytime at some time of the year.
  • Nighttime values are given as zero rather than as missing data. Missing values will remain when a daytime value can’t be produced because, for instance, input data are missing or rejected or model limitations are exceeded.
  • Values for which the solar zenith angle is between 83 and 90 degrees, for which cases the satellite model is not valid, and corresponding to the period of up to approximately one hour after sunrise or before sunset, are filled by adjusting the satellite GHI value from an adjacent hour. The fill method assumes that the clear-sky index is the same as that of the next (for morning) or previous (for afternoon) hour, where the clear-sky index is the ratio of the GHI to the clear-sky GHI predicted for the same solar zenith angle by a simple model.
  • If the satellite data for an hour are missing then a file is supplied which contains missing or zero values for daytime or nighttime respectively.
  • The above changes, taken together, result in this dataset being a complete hourly time series, with values given as missing only for daytime cases of missing or rejected satellite data.
  • The Bureau of Meteorology ground network used for bias modelling and uncertainty estimation includes the eight stations that operated for approximately two years during 2012-2014 under the Solar Resource Mapping Project sponsored by the Australian Renewable Energy Agency.
  • The bias model has an improved functional form and is based on the expanded station network.
  • The uncertainty estimates are based on 25-minute averages of the surface observations, rather than the 1-minute averages used previously, to better recognise the spatial averaging inherent in the satellite data.
L
2015-Nov: Affecting product IDCJAC0022
January to July 2007 one minute solar data for the Darwin site (014015) have been reprocessed.

The Bureau's ground solar data undergo a number of quality assurance tests, and are occasionally reprocessed as testing techniques are improved. Observations from the Darwin site covering the period January to July 2007 have been reprocessed after a fault in the tracking equipment was detected. Furthermore, during this period earthing issues with the global pyranometer were also detected and removed.

H
2015-Apr: Affecting product IDCJAD0111
Hourly satellite-derived global and direct normal solar irradiance data for the period between Jan 01, 2013 and Sep 30, 2014 have been reprocessed.

General

Hourly grids of global solar irradiance (GHI) and direct normal solar irradiance (DNI) across Australia are provided by a computer model which analyses satellite images. High quality observations from the Bureau's ground stations are subsequently used to fine-tune these estimates (bias correction). A bug in the software used to generate these data prevented the bias correction from being applied to observations covering the period between Jan 01, 2013 and Sep 30, 2014. The affected global and direct normal solar irradiance data (product IDCJAD0111) have been updated with the appropriate bias correction.

Technical discussion

The solar irradiance values produced raw from the Bureau's satellite data processing have a substantial bias, as determined from comparisons with surface-based irradiance measurements from the Bureau's radiation network. This bias is modelled as a function of irradiance and solar zenith angle, and is used to adjust the satellite estimates to improve agreement with the ground observations.

The bias correction varies with the values of irradiance and solar zenith angle. For GHI the bias correction results in a moderate reduction and most strongly affects the lower values. The change is amplified for DNI, for which it most strongly affects the middle values. To get an overall measure of the change in the corrected data after the bias correction was applied, the hourly data were integrated into daily exposures, then averaged over the affected period. The mean daily global horizontal exposure without the bias correction was, compared to the bias corrected data, 6-10% high over most of the country, except up to 19% in coastal NSW, southern Victoria and Tasmania. The mean daily direct normal exposure without the bias correction was 15-30% high, except up to 40% in southern Victoria and northern Cape York, and up to 50% in Tasmania.

Download the updated hourly gridded global and direct normal solar irradiance data for the affected 21 month period (4 compressed files, 5 Gbyte in total). This free update will only be available until 30th June 2015. The data will be superceded by an enhanced version of the entire gridded solar data set which will be available at normal cost recovery pricing.

M
2014-Aug: Affecting products IDCJAC0003, IDCJAC0016, IDCJAD0106, IDCJAD0105, IDCKAS????
Routine bias correction update for satellite-derived solar data.

General

The satellite-derived daily solar exposures for September 2013 to June 2014 (2013-09-01 to 2014-06-30) have been updated using a new bias correction based on surface-based exposure measurements.

Technical discussion

One of the steps in the production of the satellite-derived daily solar exposures is an adjustment to reduce biases with respect to observations from the Bureau's high quality ground stations. The ground data are available only after a delay of up to several months due to the partially manual nature of their quality control. Thus, the initial bias correction of the daily satellite data cannot be done against the corresponding ground data. Instead, for near-real-time production the bias model is based on a temporal extrapolation of previous satellite-to-ground comparisons. At a later date, a more effective routine bias adjustment is done once the relevant ground data become available.

H
2014-Apr: Affecting products IDCJAC0003, IDCJAC0016, IDCJAD0106, IDCJAD0105, IDCKAS????
The following products have been updated: solar data from Climate Data Online, recent solar maps and grids available online, and daily and monthly solar grids provided on USB.

General

The daily global solar exposure data accessed via Climate Data Online and also available as gridded data are provided by a computer model which analyses satellite images. High quality observations from the Bureau's ground stations are used to fine-tune these solar exposure data. The information used for fine-tuning the computer model data needs to be updated periodically. However, there was a delay in this updating process which led to a less effective adjustment for model bias. The daily global solar exposure data (products IDCJAC0016 and IDCJAC0003) have been updated and made available via Climate Data Online from 3 April 2014.

Technical discussion

The daily solar exposures produced raw from the Bureau's satellite data processing have a substantial exposure-dependent bias, as determined from comparisons with surface-based exposure measurements from the Bureau's radiation network. This bias can be modelled as a linear function of exposure, with the model coefficients changing on a time scale of months for a given satellite, and this model can be used to adjust the satellite exposures. The surface-based measurements are available only after a delay of several months due to the partially manual nature of their quality control. Thus the near-real-time satellite exposures must be adjusted with a temporal extrapolation of the bias model.

The temporal extrapolation of the bias model should be updated periodically as new surface-based observations come in, so that the extrapolation is not too far ahead. However, a prolonged delay between updates reduced the effectiveness of the bias adjustment process, particularly in the early summer period (see Figure Data-difference for an example). The raw exposures have been processed with a refreshed bias model derived from the surface observations, and made available via Climate Data Online from 3 April 2014.

An example of the error associated with the delay in updating the bias model.

Figure Data-difference. An example typical of the deviation associated with the delay in updating the bias correction model is shown in this figure, for the solar exposure estimated for the Alice Springs Airport. The graph shows the difference between previous data with the old bias correction, and data with the updated bias correction.

The impact of the delay in updating the bias correction factor is evident in the graph as a steady increase in the difference during 2013. Data prior to May 2011 had been previously updated with the relevant bias correction, and hence show no difference. Deviations starting from May 2011 can be attributed to the normal extrapolated bias adjustment process, which shows a peak (least effective correction) around the early summer period. It is important to note that there is still an uncertainty in the updated data. The RMS difference between satellite and surface data varies between about 1.0 MJ·m-2 in winter and about 2.0 MJ·m-2 in summer.

The change in exposure value between before and after the revision will vary linearly with exposure.

Download the updated daily gridded global solar exposure data for the affected period (340 Mbyte, compressed). This free update will only be available until 30th June 2015. The data will be superceded by an enhanced version of the entire gridded solar data set which will be available at normal cost recovery pricing.

H
2014-Mar: Affecting product IDCJAC0022
Minimum 1 second direct irradiance (over 1 minute) currently unavailable in all one minute solar data.

In previous data updates affecting all stations and years, the Minimum 1 second direct irradiance (over 1 minute) values in each data file were instead the Maximum 1 second direct irradiance (over 1 minute) statistics for the same time period. The Maximum 1 second (over 1 minute) statistics are correct and duplicated across both fields. As this duplication occurred when the database was created, it will be necessary to reprocess all relevant data, a task which will take a substantial period of time to complete. Please note that as of this data update, the Minimum 1 second direct irradiance (over 1 minute) values have been replaced with blanks. Users will be notified via RSS feed when the updated data becomes available.

M
2013-Aug: Affecting product IDCJAC0022
A subset of one minute solar data from the Darwin site (014015) have been reprocessed.

The Bureau's solar instruments are regularly calibrated by comparison to a pair of reference pyrheliometers, which are in turn traceable to the National Radiometric Standards. This quality assurance process detected a change in the sensitivity of the Bureau's reference instruments. As a consequence, the sensitivities of field instruments derived from these references at the Darwin site (014015) have been updated. Global and diffuse irradiance data have been reprocessed using the updated sensitivities, with changes within 4% of the originally released values. The period in question is from 27 May 2009 until 28 May 2013. In addition, approximately four weeks of global and diffuse data between 11 Dec 2006 and 24 Apr 2007 have been removed after additional information has revealed that a suspected lightning strike adversely affected instrumentation at the Darwin site.

M
2013-Feb: Affecting product IDCJAC0022
Additional uncertainty information for one minute solar data has now been added.

Uncertainties for nighttime longwave irradiance have now been added where previously unavailable. Additional quality control of some historical observations may have occured since the last release.

L
2012-2014: Affecting product IDCJAC0022
Temporary expansion of the Bureau's solar observation network provides additional one minute solar data.

Eight new or re-established ground monitoring stations have been funded for a period of eighteen months through a collaborative project between Geoscience Australia and the Bureau, as part of the Australian Government Solar Flagships Program funded by the Australian Renewable Energy Agency. The primary goal for expanding the network is to provide valuable additional data to help enhance the computer model which generates the satellite-derived solar exposure provided by the Bureau and through Geoscience Australia. The data from these stations will also be publicly available via the Bureau's One Minute Solar Data product.