HDF-EOS FAQ

Hierarchical Data Format (HDF) is a data format designed to facilitate sharing of scientific data. Features include platform independence, user extendibility, and embedded metadata for units, labels, and other descriptors. Standard data types include: multidimensional array, text, table, raster image, and palette. Both C and FORTRAN functions are available to access and process the data.

HDF was developed by the The HDF Group (THG). THG maintains its own support page. General information about scientific data formats can be obtained from the Scientific Data Formats FAQ.

1.2 What is HDF-EOS? How does it differ from standard HDF?

The HDF-EOS data format is standard HDF with ECS conventions, data types, and metadata added. HDF-EOS adds three geolocation data types (point, grid, and swath) which allow the file contents to be queried by earth coordinates and time. An HDF-EOS file also contains ECS core metadata which is essential for ECS search services. An HDF-EOS file can be read by any tool that processes standard HDF files, although HDF-EOS geolocation or temporal information generally is not accessible using standard HDF calls.

A data product need not fit any of the grid, point or swath models to be considered HDF-EOS. If ECS metadata are included, it is a valid HDF-EOS file.

HDF-EOS is implemented as a C library extension of the standard HDF library (with FORTRAN bindings). HDF-EOS ordinarily is based on the latest release of HDF.

1.3 What are the advantages of using HDF-EOS?

HDF-EOS is the official data format for EOSDIS data production and archive. Use of HDF-EOS insures that data can be accessed by EOSDIS scientists and non-scientists from multiple disciplines. Use of HDF-EOS also can eliminate duplication of software development efforts, especially for analysis and visualization software. EOSDIS data providers must supply written justification for deviating from the HDF-EOS (or HDF) format.

2. GEOLOCATION DATA TYPES

2.1 What is the swath data type?

The swath data type is used to represent time-sequences of scan lines, profiles, or other array data. A swath has an "along-track" direction that is monotonic with time and usually a "cross-track" direction associated with each time instance. The cross-track direction need not be a straight line; a circular scanning instrument can utilize this data type by specifying its coordinates as a function of time. This data type will be most useful for products at EOS processing levels 1 or 2.

2.2 What is the grid data type?

The grid data type is used to represent locations on a regular grid, based upon one of several earth projections. This data type will be most useful for products at EOS processing levels 3 or 4. Map projections presently supported include: Mercator(space oblique), Universal Transverse Mercator (UTM), interrupted Goode, polar steroegraphic, Lambert azimuthal (equal area), Lambert conformal conic, polyconic, transverse Mercator, Hotin oblique Mercator, and integerized sinusoidal grid.

2.3 What is the point data type?

The point data type is used to represent a network of locations that are irregularly spaced. The user may define hierarchical "levels" that are similar to linked tables in a relational database. This feature allows the point data type to be used with ocean buoy data that change locations with time.

2.4 What function calls are available for the geolocation data types?

The functions fall into one of five categories: access, definition, I/O, inquiry, and subset (Section 2.6). Access routines open/close files and attach/detach names to files. Definition routines create new instances of a data type and define mapping parameters. I/O routines read/write data and their attributes (dimension, type, etc.). Inquiry routines retrieve parameter values. Subset routines create subsets in the space/time domain (see 2.6). All ECS search and order services will be built on top of these function calls.

2.5 Can data of more than a single type or resolution reside in a single HDF file?

You can mix grid, point, swath, and standard HDF structures in a single HDF-EOS file. The library allows a user to extract and manipulate any single object. In addition, structures of different resolution can be stored in the same file.

2.6 What is subsetting and what types of subsetting are possible?

Subsetting is the extraction of a user-specified range or parameter from the data. HDF-EOS permits subsetting by geographic location or time on geolocation variables (point, swath, or grid). The field names "Longitude", "Latitude", "Colatitude", and "Time" are understood by HDF-EOS to be geolocation variables. Subsetting also can be carried out along the vertical dimension (or any other dimension that is monotonic) for grid and swath data. Standard HDF calls permit subsetting by array row or column.

Subsampling is the extraction of a range of data with a "skip" factor which selects every Nth data point in the specified range. "Bands" of data can be extracted from an array using this method.

2.7 What format is used for time?

HDF-EOS uses the TAI93 (International Atomic Time) format. This means that time is stored as the number of elapsed seconds since January 1, 1993 (negative values represent times prior to this date). An 8 byte floating point number is used, producing microsecond accuracy from 1963 (when leap second records became available electronically) to 2100. The SDP Toolkit provides conversions from other date formats to and from TAI93. Other representations of time can be entered as ancillary data, if desired. For lists and descriptions of other supported time formats, consult the Toolkit documentation or write to sdps-support@earthdata.nasa.gov.

2.8 What if my data do not fit any of the three geolocation data types?

If none of the geolocation types are appropriate, a standard HDF data type can be used (such as multidimensional array or raster image) and ECS metadata can be created. However, ECS geographical and temporal subsetting tools will not work on the standard HDF types. It is expected that Level 0 and some Level 1 data may not fit any of the standard HDF-EOS data types. Other heritage data also may not fit. These datasets can be converted to a geolocation type at a higher processing level. Zonally averaged data also may not fit any of the geolocation types. If you are unsure whether one of the geolocation data types is appropriate, send an e-mail message to sdps-support@earthdata.nasa.gov.

2.9 Are there any upper limits to the number of fields associated with a geolocation structure?

A maximum of 256 "fields" can be associated with any swath or any "level" of a point structure.

3. METADATA

3.1 How are metadata inserted, viewed, and extracted from HDF-EOS files?

The SDP Toolkit provides the easiest method for handling metadata. This library contains calls for writing, manipulating and accessing metadata in HDF-EOS files. EOSView (see Section 7) can be used to view metadata. For more information consult Appendix J of the SDP Toolkit Users Guide or write hdfeos@eos.hitc.com.

3.2 What types of metadata are embedded in an HDF-EOS file and what are the added storage requirements?

An HDF-EOS file must contain ECS "core" metadata which is essential for ECS search services. Core metadata are populated using the SDP Toolkit, rather than through HDF-EOS calls. "Archive" metadata (supplementary information included by the data provider) may also be present. If grid, point, or swath data types have been used, there also will be structural metadata describing how these data types have been translated into standard HDF data types. Metadata resides in human-readable form in the Object Descriptor Language (ODL). Structural metadata uses 32K of storage, regardless of the amount actually required. The sizes of the core and archive metadata vary depending on what has been entered by the user.

3.3 What are the options for adding ECS metadata to standard HDF files?

For data products that will be accessed by ECS but which remain in native HDF, there is a choice of 1) adding no ECS metadata in the HDF file, 2) inserting ECS metadata into the HDF file, or 3) "appending" ECS metadata to the HDF file. "Append" means updating the HDF location table so that the appended metadata becomes known to the HDF libraries/tools.

3.4 Some DAACs currently provide descriptor files that give background information about the data. Will this information be included in an HDF-EOS file?

Yes. The descriptor file will be retained. It can be viewed by EOSView if it stored either as a global attribute or a file annotation.

4. FILE PROCESSING

4.1 What is the maximum file size allowed?

Files must not exceed 2 Gigabytes (uncompressed) in either HDF or HDF-EOS.

4.2 How large is an HDF-EOS file relative to the raw data file?

The two are similar in size (uncompressed) for files larger than 100's of KBytes. The HDF-EOS file stores the data objects in binary form, but includes metadata of various kinds (see 3.2).

4.3 Is file compression possible within HDF-EOS?

Each field within an HDF-EOS file can be individually compressed using Run Length Encoding (RLE), Huffman, or gzip. A compressed field will be uncompressed dynamically when HDF-EOS calls are made to access it. A file may contain a mixture of compressed and uncompressed fields. Metadata are not compressed.

The UNIX "compress" command also can be used to compress an entire HDF-EOS file. This command compresses the metadata in addition to the data. UNIX compression will be not be used in ECS for storage purposes, though it might be used for data distribution. No other file compression is planned for the near future.

4.4 Are external (HDF subordinate) files possible in HDF-EOS?

The subordinate feature of HDF currently is not supported by ECS. The issue is under study at this time.

4.5 How do HDF tags work within HDF-EOS files?

HDF-EOS uses the same tags as HDF files, as it is built entirely on HDF data structures. HDF-EOS tags can be viewed with standard HDF tools and function calls.

5. ACCESS AND AVAILABILITY

5.1 What platforms & operating systems are supported?

HDF-EOS is supported on Linux, MacOSX, MacIntel, SUN, Cygwin, etc

5.2 How do I link HDF-EOS source code to my program?

Untar the HDF-EOS tar file and put libhdfeos.a in your link path. Complete information on installation and linking is contained in the HDF-EOS Users Guide

5.3 Where can I get sample HDF-EOS files for inspection?

Test files are available on request from sdps-support@earthdata.nasa.gov. Alternatively, see hops.stx.com:8080/diblpd.html or ulabhp.gsfc.nasa.gov/~jpals.

5.4 What version of HDF-EOS is currently being distributed?

The current version is 2.16, released in July, 2009. It is based on HDF Version 4.2r4. The The current version of HDF-EOS5 is 5.1.12, released in July, 2009. It is based on HDF5 Version 1.8.3.

6. DOCUMENTATION

6.1 What documentation is available for HDF-EOS and how do I obtain it?

The HDF-EOS User's Guide for the ECS Project is the standard documentation. It is available on-line in two parts: Volume 1: Overview and Examples and Volume 2: Function Reference Guide. Documentation for standard HDF also is available on-line.

6.2 What is the relationship between the documentation for HDF-EOS and that for HDF and for the SDP Toolkit?

The HDF-EOS documentation provides all the information needed to use HDF-EOS. SDP Toolkit documentation is contained is a separate document.

7. OTHER SOFTWARE TOOLS

7.1 What software tools are presently available for reading or visualizing HDF-EOS files?

EOSView presently is the only tool that supports the geolocation subsetting capabilities of HDF-EOS. Many other browsing and visualization tools support standard HDF files, but geolocation subsetting is not possible. Browsing tools available without charge include: HDF WWW Scientific Data Browser, Mosaic Browse-a-Rama, and a proptotype Java-based browser from THG. Full visualization tools that read standard HDF tools and are available without charge include: LinkWinds from JPL, Envision from UIUC, and VCS from Lawrence Livermore Laboratory.

Commercial visualization tools that support standard HDF include: IDL from Research Systems Inc., PV-WAVE from Visual Numerics, Transform from Fortner Research, Inc., EASI/PACE from PCI, Data Explorer from IBM, and many others.

7.2 What PC or Mac tools are presently available for reading or visualizing HDF-EOS files?

The HDF Browser from Fortner Research Inc. is available for Windows and Macintosh. There is no charge for this package. It supports only standard HDF and thus does not permit geolocation subsetting.

7.3 What software tools are currently under development?

IDL, IBM Data Explorer, HDF Browser, and LinkWinds have expressed interest in the development of interfaces to HDF-EOS files. Of these, only HDF Broswer from Fortner Research has made a commitment. ECS is currently developing the Java Earth Sciences Tool (JEST) for viewing HDF-EOS files.

8. OTHER RESOURCES

8.1 How do I obtain HDF-EOS? Who maintains the libraries?

HDF-EOS was developed and is maintained by Raytheon under contract to EOSDIS Core Systems (ECS). The current version (Version 2.16) was released in July 2009. Information on downloading can be obtained by sending e-mail to sdps-support@earthdata.nasa.gov.

Page tree

HDF-EOS FAQ