Standardization of Data Object Design (DOD) within ARM’s software products

An important capability of the ARM program is the ability to deliver consistent data to the atmospheric science user community.  In an effort to improve that capability, an effort was undertaken to standardize the names and attributes within the ARM data.  This capability will improve the data stream development process and provide better data analysis tools for the user community.  Currently, there are inconsistencies across data streams.  Development of standards for field and global names and attributes will improve the usability of the ARM data.  This document outlines the goal of this task and the guidelines for the construction of the field names, mandatory field and global attributes.

The goals are:

• creating a formula for naming fields
• defining required field attributes
• defining recommended field attributes
• providing resolution on bin, height, altitude, and time-averaging differences

Process for Developing Standards:
A review of the current VAP Data Object Design (DODs) was performed to determine the scope of this task.  An initial document was constructed at PNNL and presented at the Developers meeting at the Brookhaven National Lab.  With the consensus of the developers and managers, a bi-weekly meeting has been held with Robin Perez, Sherman Beus, Brian Ermold, Matt Macduff, Chitra Sivaraman and Krista Gaustad.

Guidelines for construction of field names:

1.  Names will consist of lower-letter, digits and underscores and begins with a letter.  Upper case is not to be used. (site/facility designation is the exception)
2.  The field names will be constructed by joining the names to the qualifiers using underscores.
3.  One has to be reasonable when picking field names.
4.  Field names will be concise.

Hierarchy:
If a conflict arises, then the following hierarchy shall be used.

1. [super prefix] e.g. qc, aqc, be
2. [prefix] e.g. interpolated, calibrated, instantaneous
3. [measurement] e.g. vapor_pressure, pressure, temperature
4. [subcategory] e.g. head, air, up, short, hemisphere
5. [medium] e.g. earth, satellite, sea
6. [height] e.g. 10m,05km
7. [enumeration] e.g. e, w, n, s, a, b, 1, 2
8. [quantity] e.g. mean,<field>_std is the preferred format for standard deviation
9. [source name] e.g. smos, smet

It is important to be reasonable. Field names should be as concise as possible.  For example, "temperature" should be "temp" and name hierarchy should only be used when necessary for field differentiation.
Example of field names using some of the hierarchy:

• qc_temp_ambient_10m
• temp_swat
• wspd_10m
• qc_rh_std
• qc_vapor_pressure_std_aeri

Dimensions
If a dimension is used, then a field level variable with the name of the dimension should be added with a long name and units attribute.  Examples of dimensions are bin, height, range, depth.
Example: dimensions:
                                    time = UNLIMITED ; // (2878 currently)
                                    range = 1999 ;
      variables:
                        float range(range) ;

range:long_name = "Distance from transceiver to center of corresponding range_bin" ;
range:units = "km" ;

Attributes:

In general, the attribute names will be in lower case. The words should be separated by an underscore. A single lengthy comment attribute is preferred instead of multiple comment attributes (reference_line_n) with one line of explanation in each comment.
Mandatory field attributes

• long_name
• units

Standard field attributes

• valid_min
• valid_max
• valid_range
• valid_delta
• resolution
• comment
• comment_n (used for multiple distinct comments within a single field)
• missing_value
• precision
• accuracy
• uncertainity

Other possible attributes (not all inclusive):

• actual_wavelength
• corrections
• filter_wavelength
• FWHM (capital letters ok)
• height - If there are multiple sensors within the instrument suite at different heights, then a field name with the height specification is preferred. Example:
• float wspd(time) ;

wspd:long_name = "Mean Wind Speed" ;
wspd:units = "m/s" ;
wspd:sensor_height = 2m;
Global attributes should be used to specify the heights when all sensors are at the same height.  Example: sensor_height =  “10 m AGL”;

Mandatory global attributes, standard (*) global attributes

• command_line
• records command line used to run the ingest or VAP
• example: command_line = “mwravg –d 20070101 –a 5 –f nsaC1”;
• formerly Command_Line
• command_line_comment *
• optional global attribute
• records the exceptional switches used in the command line. The –d (date), -f (facility), -a (average) need not be explained.
• example: command_line_comment = “-D updates the glue database file, -C will process only the data below ~18km”

• process_version
• records the version of the ingest or vap running on production
• process_version should be used alone (ingest_software or ingest-software is obsolete) unless the DOD is a revised DOD and used as an input to an existing BW VAP and the VAP has not been updated to use process_version.
• example: process_version = “process-ingest-smos_ingest-8.1-1”
• dod_version
• records version of the DOD represented in this file
• example: dod_version=”1440smos-b1-2.0” . Until the vaps integrate with the database, the value
  will be -9999.0
• site_id
• three letter site designation (gec, fkb, nim, nsa, sgp, twp, pye)
• example: site=”sgp”
• facility_id
• Fn(n): City, State or Fn(n): City, Country or Fn(n): Region, Country
• example: facility_id = “E10: Tyro, Kansas” ; or “M1: Black Forest, Germany” ;
• data_level *
• records data level (a0, a1, b1, c1, s1)
• example: data_level = “a1”;
• formerly proc_level
• input_datastreams_num (vap only)
• example: input_datastreams_descriptions = “A string consisting of the datastream(s),  datastream version(s), and  datastream date (range).”;
• input_datastreams_description (vap only)
• records the number of datastreams with unique versions and date ranges
• example: input_datastreams_num = ”10”;
• input_datastreams (vap only)
• records the itemized list of input datastreams, versions, and date ranges.  The versions refers to the process_version.  The datastream version(s), versions and date ranges are separated by a space, colon and another space.  The individual entries are separated by a space, semicolon, followed by a new line. If there are multiple files in a day and not all are used, the individual ranges used should be itemized as separate entries.  The separator between dates in a given range is a hyphen.  If we have a single date, then no hyphen or end date should be included and the date range is simply a single date.
• example: input_datastreams = “sgpsondewnpnC1.a1 : 6.1 : 20010208.232700-20010210.053400 ;

sgpmwrlosC1.b1 : 1.17 : 20010209.000000 ;
sgp1twrmrC1.c1: Release_1_4 :  20010209.000000 ;
sgparscl1clothC1.c1 : Release_2_9 : 20010209.000000”;

• input_source (ingest only)
• records name of FIRST raw file (and path) used in daily netcdf file
• example: input_source = ”/data/collection/sgp/sgpswatsE10.00/1167508800.icm”;
• serial_number (ingest only)

• records SN of instrument(s) used to collect data
• if multiple instruments, specify instrument, if not just give SN
• example: :serial_number = "PIR1-DIR:       31312F3",

  "PIR2-DIR:       30167F3",
  "Diffuse PSP:    33271F3",
  "NIP:            31876E6",
  "PSP-DS:         33703F3",
  "SKY-IR:         1845" ;

• resolution_description (ingest only)
• boilerplate definition of ARM resolution
• example: resolution_description = "The resolution field attributes refer to the number of significant digits relative to the decimal point that should be used in calculations. Using fewer digits might result in greater uncertainty.  Using a larger number of digits should have no effect and thus is unnecessary.  However, analyses based on differences in values with a larger number of significant digits that indicated could lead to erroneous results or misleading scientific conclusions.  Resolution for lat = .001, resolution for lon = .001, resolution for alt =1”;
• averaging_interval *
• records averaging interval applied by instrument or datalogger.
• example: averaging_interval = “20 seconds” or averaging_interval = “1 minute”;
• formerly averaging_int
• averaging_interval_comment *
• records beginning, middle or end of the averaging interval.
• example: averaging_interval_comment = “The time assigned to each data point indicates the <beginning/middle/end>… of the averaging interval”;
• sampling_interval *
• records sampling interval in use by instrument or datalogger.
• example: sampling_interval = “400 microseconds”
• formerly sample_int
• missing_value *
• defines missing value or fill value (-9999) used
• example: missing_value = “-9999”;
• sensor_height *
• records height of sensors above ground level. If multiple sensors at different heights exists, use field-level attribute. See example in other possible attributes.
• example: sensor_height =  “10 m AGL”;
• formerly  sensor_location
• For qc requirements, please refer to the documents in http://engineering.arm.gov/~gaustad/qc_standards/final_doc/update_c_datastream.htm for vaps and http://engineering.arm.gov/~gaustad/qc_standards/final_doc/ingest_requirements.htm for ingests).
• zeb_platform (for zebra processes) *
• records full datastream name sssiiiiFn.dn
• example: zeb_platform = “nsamettwr4hC1.b1”;
• history
• records the user name, machine name, the date and version of the ingest or vap that is recorded in process_version global attribute. The date should be in the following format.  <d(d)-<Mmm>-<YYYY>,<h(h)>;<mm>;<ss>;
• example: history = “created by user dsmgr on machine ruby at 1-Jan-2007,2:43:02 using $State: ds-zebra-zeblib-4.16-0” ;
• last global attribute in all ARM netcdf files

Obsolete global attributes:

• ingest_software
• This should be used only if it is a revised DOD and used as an input to an existing BW VAP and the existing VAP has not been updated to use process_version.
• ingest-software
• This should be used only if it is a revised DOD and used as an input to an existing BW VAP and the existing VAP has not been updated to use process_version.

Version Discussion:

Historically, the ARM DOD’s have had several version attributes in the DOD.  The versions of the various components of the system and softwares are particularly important when a datastream is reprocessed and differences occur in the output. 

Solutions:

Since the user has to usually look further to reproduce an exact version of the data, the DOD will provide only the minimum information to help the user.

• Provide information about how the data was generated - the date, the user and the machine that generated the data. This information is available in the "history" global attribute.
• Provide process_version. The full package number of the source code with version number.
• Provide dod_version. The base name, data level and the version number of the DOD.

At the discussion, these assumptions were made.

• It is not the responsibility of the netcdf file to report the version of the compiler, libraries etc.
• The netcdf file needs to report enough information so the data user can understand if they care.
• If there is a discrepancy in the data, the user has to dig for more information even if the netcdf file reports the version of the compiler, libraries, source code, etc.

Guidelines to source field:

When multiple inputs or algorithms are used to compute data fields, it may be useful to indicate the source of the input or algorithm.  In such a case, a new variable indicating the source of the data could be added.  The source field is optional and can be added at the discretion of the developer or translator or when a user might find it helpful.  “User” in this case could be the end-user of the product or the users as designated by the development team during the evaluation stage.  

To standardize the way VAPs write the source field, the following guidelines shall be followed:

1. The name of the field shall be “source_<field>” where <field> is the variable field name.

Example: 

aod(time, height)

source_aod(time, height)

If the source is constant with height, the source field shall be a function of time only.   Note that if the sources do not change overtime, an attribute could be added to the field itself indicating the source instead of creating a separate field.

2. The source_<field> should be accompanied by source_<field>:description with the following text.

Example:

source_be_aod_500:description = "This field contains integer values which should be interpreted as listed. A value of -1 represents no source available." ;

3. Values of the source field shall be of type “integer”.

4. If there are no data samples for a particular time, the source value shall be set to -1 (negative 1).  The value ‘0’ (zero) shall not be used.  If a source preference ranking is appropriate, lower numeric values will indicate higher preference. 

5. The meanings of the possible integer source values shall be indicated in the source field attributes “value_n” with the syntax “datastream_name: field_name”. The site and facility may be included as part of the datastream_name and is left to the discretion of the developer or translator.

Example:

value_1=”mfrsr.c1:aerosol_optical_depth”

value_2 =”mfrsr.b1:aerosol_optical_depth”

6. The source field attribute “value_n_description” is optional and can be used to provide more details on how the data was computed.

Example:

source_be_angstrom_exponent:value_5_description = "Fill gaps of 3 days or less via interpolation" ;

Guidelines to name quick-look plot filenames:

The standard convention for VAP quick-look plot filenames created at the Data Management Facility should be as follows.

datastream.level.date.time.description.extension

Note: The delimiter should be a ‘.’ (period) except within the description when it should be an (_) underscore.

Example:

sgp30ebbrE9.b1.20100101.000000.latent_heat_flux.png

Summary:

Standardizing the field names in the ARM program's VAPs and ingests presented in this format could significantly increase current data user satisfaction and acceptance of the ARM data sets within the atmospheric science and educational communities. Implementation efforts will begin when new VAPs and ingests are being developed and getting ready to be released.