Page tree
Skip to end of metadata
Go to start of metadata

Recommendation:

Only compression filters that are officially supported by a default installation of the current netCDF4 software distribution should be used in Earth Science data products in netCDF4 or netCDF4-compatible HDF5 formats.

Recommendation Details: NetCDF4 has enabled access to non-default (i.e., non-DEFLATE) HDF5 compression filters starting from version 4.7.0.  However, the filter identification and access are currently obscure (~five digit IDs) and non-portable (no guarantees client software will be able to decompress them). DEFLATE is currently the only compression filter that is guaranteed to work with default (non-customized) netCDF4 installations, and so DEFLATE is the only compression filter that should be used in interoperable Earth Science data products in netCDF4 or netCDF4-compatible HDF5 formats. Use of the shuffle filter is not prohibited since it is not a compression filter and is supported by the netCDF4 default installation. Combining the shuffle and the DEFLATE filters can noticeably improve the data compression ratio.

This recommendation may evolve as netCDF4 extends its default compression filter support to other filters. Description of netCDF4 filter support can be found at https://www.unidata.ucar.edu/software/netcdf/docs/md__Users_wfisher_Desktop_docs_netcdf-c_docs_filters.html .

Awaiting ESO Approval

This recommendation has been finalized by DIWG but has not yet received final ESO approval.

17 Comments

  1. How about "Only compression filters that are officially supported by a default installation of the current netCDF software distribution should be used in Earth Science data products. Currently, DEFLATE is the only such filter."

    1. Thanks Charles Zender, changed as per your comment.

  2. Here is the link to the definitive netCDF4 documentation for filters:

    https://www.unidata.ucar.edu/software/netcdf/docs/filters_8md_source.html

    A careful read shows that the default netCDF installation does not install any filters except for DEFLATE, and that user intervention is necessary to install additional filters in a findable way. That's fine because it's a new netCDF feature. But it's terrible for interoperability, hence this recommendation.

  3. Should we add a link to the release notes from netCDF 4.6.0 that describes the changes to the compression filters?

  4. Sounds transient. Which link do you mean?

    1. I would like to read the part of the netCDF 4.6.0 release notes that describes the changes to the compression filters.

  5. I notice that LZW compression is specifically mentioned as acceptable for GeoTIFF in another recommendation. That is fine though it seems to conflict with the first sentence of this recommendation: "Only compression filters that are officially supported by a default installation of the current netCDF4 software distribution should be used in Earth Science data products" which I suggest we consider altering to "Only compression filters that are officially supported by a default installation of the current netCDF4 software distribution should be used in Earth Science data products in netCDF or HDF5 format", or something similar to avoid to the apparent conflict between the two recommendations.

    1. I agree with Charlie's suggested revision.

  6. I implemented that change and Siri's suggested change

  7. Update the netCDF docs link in the recommendation.

  8. I like this recommendation as is, however...

    I think the title does not correctly summarize the content. The recommendation is to use officially supported codecs, of which there is only one, DEFLATE. Should more codecs become officially supported, the title will be erroneous, but the content of the recommendation would be accurate. I suggest the title/summary be changed to "Use Only Officially Supported Compression Filters on ...". Perhaps this can be done in the editorial stage since it does not change the content of the recommendation.

  9. best to reference unidata's github repo for the doc on filters:  https://github.com/Unidata/netcdf-c/blob/master/NUG/filters.md