Recommendation:

Only compression filters that are officially supported by a default installation of the current netCDF-4 software distribution should be used in Earth Science data products in netCDF-4 or netCDF-4-compatible HDF5 formats.

Recommendation Details: NetCDF4 has enabled access to non-default (i.e., non-DEFLATE) HDF5 compression filters starting from version 4.7.0.  However, the filter identification and access are currently obscure (~five digit IDs) and non-portable (no guarantees client software will be able to decompress them). DEFLATE is currently the only compression filter that is guaranteed to work with default (non-customized) netCDF4 installations, and so DEFLATE is the only compression filter that should be used in interoperable Earth Science data products in netCDF-4 or netCDF-4-compatible HDF5 formats. Use of the shuffle filter is not prohibited since it is not a compression filter and is supported by the netCDF4 default installation. Combining the shuffle and the DEFLATE filters can noticeably improve the data compression ratio.

This recommendation may evolve as netCDF4 extends its default compression filter support to other filters. Description of netCDF-4 filter support can be found at https://docs.unidata.ucar.edu/netcdf-c/current/filters.html .

Awaiting ESO Approval

This recommendation has been finalized by DIWG but has not yet received final ESO approval.

17 Comments

  1. Charles Zender

    How about "Only compression filters that are officially supported by a default installation of the current netCDF software distribution should be used in Earth Science data products. Currently, DEFLATE is the only such filter."

    1. Aleksandar Jelenak

      Thanks Charles Zender, changed as per your comment.

  2. Charles Zender

    Here is the link to the definitive netCDF4 documentation for filters:

    https://www.unidata.ucar.edu/software/netcdf/docs/filters_8md_source.html

    A careful read shows that the default netCDF installation does not install any filters except for DEFLATE, and that user intervention is necessary to install additional filters in a findable way. That's fine because it's a new netCDF feature. But it's terrible for interoperability, hence this recommendation.

    1. Peter Leonard

      The Web page https://www.unidata.ucar.edu/software/netcdf/docs/filters_8md_source.html now seems to work again.

  3. Peter Leonard

    Should we add a link to the release notes from netCDF 4.6.0 that describes the changes to the compression filters?

  4. Charles Zender

    Sounds transient. Which link do you mean?

    1. Peter Leonard

      I would like to read the part of the netCDF 4.6.0 release notes that describes the changes to the compression filters.

      1. Peter Leonard

        This may be it: https://www.unidata.ucar.edu/software/netcdf/docs/filters_8md_source.html .

  5. Peter Leonard

    The link

    https://www.unidata.ucar.edu/software/netcdf/docs/md__Volumes_Workspace_Ward_releases_netcdf-c-4_87_82_netcdf-c_docs_filters.html

    no longer words, and it looks like what used to be there has been moved to

    https://www.unidata.ucar.edu/software/netcdf/docs/md__home_wfisher_Desktop_gitprojects_netcdf-c_docs_filters.html

  6. Charles Zender

    it's up to you

  7. Charles Zender

    I notice that LZW compression is specifically mentioned as acceptable for GeoTIFF in another recommendation. That is fine though it seems to conflict with the first sentence of this recommendation: "Only compression filters that are officially supported by a default installation of the current netCDF4 software distribution should be used in Earth Science data products" which I suggest we consider altering to "Only compression filters that are officially supported by a default installation of the current netCDF4 software distribution should be used in Earth Science data products in netCDF or HDF5 format", or something similar to avoid to the apparent conflict between the two recommendations.

    1. Peter Leonard

      I agree with Charlie's suggested revision.

  8. Charles Zender

    I implemented that change and Siri's suggested change

  9. Aleksandar Jelenak

    Update the netCDF docs link in the recommendation.

    1. Aleksandar Jelenak

      Done. (For now.)

  10. Charles Zender

    I like this recommendation as is, however...

    I think the title does not correctly summarize the content. The recommendation is to use officially supported codecs, of which there is only one, DEFLATE. Should more codecs become officially supported, the title will be erroneous, but the content of the recommendation would be accurate. I suggest the title/summary be changed to "Use Only Officially Supported Compression Filters on ...". Perhaps this can be done in the editorial stage since it does not change the content of the recommendation.

  11. SiriJodha Khalsa

    best to reference unidata's github repo for the doc on filters:  https://github.com/Unidata/netcdf-c/blob/master/NUG/filters.md