Recommendation:

A list of all missing granules should be provided on a permanent Web site (e.g., the DOI landing page) for each Earth Science data product.

Recommendation Details: It is not uncommon for an Earth Science product to be missing some data granules.  For example, data gaps can occur because of missing L0/1 data (e.g., the instrument was not operating, the instrument was in a mode that did not produce useful observations, or a permanent loss of telemetry occurred).

A list of all missing granules should be provided on a permanent Web site (e.g., the DOI landing page) for each Earth Science data product, because it can save users much time if the missing data are clearly identified and explained.  This list should also explain the reason for each and every missing granule.  A list of missing granules that is machine readable would be very useful.

This recommendation applies to instruments that are expected to acquire data on a regular basis (e.g., MODIS, MOPITT, CERES).  This recommendation does not apply to instruments like ASTER, which have an irregular observation schedule.

Awaiting ESO Approval

This recommendation has been finalized by DIWG but has not yet received final ESO approval.

12 Comments

  1. Peter Leonard

    Coming up with a machine-readable format for listing missing granules is a topic for future discussion.

    1. Christopher Lynnes

      I strongly recommend that one such format be Spatiotemporal Asset Catalog for two reasons:

      1. It would include spatial and temporal descriptions of the missing granules, which are the two most important aspects users want to know, and
      2. The STAC community is prone to metadata harvesting, and have not seriously considered how to handle deleted or reprocessed data. This would demonstrate the value of documenting the data that are not there, whether due to those that have been pulled due to quality issues, or the ones that were never collected or processed.

      1. Peter Leonard

        If we are going to be this specific about a machine-readable format, then it should be a separate recommendation, in which case the "A list of missing granules that is machine readable would be very useful" sentence should be modified or dropped.

  2. Peter Leonard

    I now think it would be best to rename this recommendation to something like "Documenting Data Gaps".

    Data gaps are usually due to missing L0/1 data (e.g., the instrument was not operating, the instrument was in a mode that did not produce useful observations, or permanent loss of telemetry).

    All data gaps should be documented regardless of the reason, including situations beyond the control of the data producer.

    1. Peter Leonard

      Perhaps "Documenting Missing Granule Ranges".

      1. Peter Leonard

        Or perhaps just "Documenting Missing Granules".

  3. SiriJodha Khalsa

    I believe some further refinement of the recommendation is warranted.  Specifically, adding the words "where appropriate" instead of stating that the recommendation applies to all Earth science data products.  Again, ASTER is a prime example. Yes, there is a scheduler that prioritizes a list of targets, but it's automatic and one cannot predict precisely what scenes will be acquired in a given orbit, so it's impossible to identify what should have been acquired but wasn't.

    1. Peter Leonard

      In my mind, "missing" granules are granules that would have been expected to have been produced under normal operating conditions, but were not produced due to something unexpected.

      ASTER worked slightly differently than most instruments in that observations were only collected for a small part of each orbit with a big gap in time between observations - this defines normal operating conditions for ASTER.

      Whenever ASTER did not make an expected measurement (e.g., an orbit without any measurements made), then this would represent a missing granule.

      How does the ASTER Team 1) list the observations that were carried out as planned, and 2) list the observations that would have normally been expected but were not made?

      1. Hampapuram Ramapriyan

        I just sent an email message to Tom Maiersperger at LP.DAAC asking how they handle the ASTER case. I will share his response when I get it.

  4. Hampapuram Ramapriyan

    Maybe the recommendation should be confined to global datasets where it is expected that the data are collected constantly. Examples besides ASTER are ICEBridge, other suborbital investigations, and field campaigns where the expectation of what should be collected is not clear.

    1. Peter Leonard

      The difference between ASTER, which collected observations for roughly only 8 minutes per orbit, and an instrument on a sun-synchronous satellite that collects observations over the entire sunlit part of Earth (e.g., 49 minutes per orbit) is merely one of degree.

      Yes, we should state somewhere in this recommendation that it only applies to instruments on satellites that make observations in a predictable manner, though I think this includes ASTER.

  5. Vardis Tsontos

    On a practical note, how will these lists of expected but missing granules be compiled and whose responsibility is it to deliver updates to those lists on an ongoing basis (the mission, or the DAAC archive/distributing the data)?  The DAACs provide information on periods of data outage, and certainly could additionally list specific missing granules for a given dataset that could be exposed via the dataset's landing page.  However, I think the missions are best positioned to know reliably what orbits could not be processed for whatever reason, and provide that information to the DAACs both for the record and in order to inform users.  I mention this because implementational considerations associated with this recommendation will result in additional requirements likely for both missions and DAACs.

Accessibility