EGI Programmatic Access FAQ

9/15/17

  1. What is the use case for having both bbox and bounding_box ?
    These separate parameters allow for specifying a spatial search constraint independently from specifying a spatial subsetting area for the results. Bounding_box is used for CMR searching and Bbox is used to specify subsetting area. When performing subsetting, these most likely will be the same. However, when not subsetting, the bbox parameter will not be used.
    For a spatial search, CMR returns granules whose spatial coverage intersects the Bounding_box. A user can then spatially subset those granules to an expanded coverage area using Bbox.

  2. How do I know what coverages are available for a collection?
    Query the capabilities service for the collection. See the sample script cap.sh.

  3. How do I specify dataset groups or multiple datasets (coverages) for processing?
    Use the coverage= kvp to specify datasets or groups. Multiples can be specified separated by commas. The coverage values are case sensitive. The syntax uses slashes to delineate the group hierarchy, always starting with a slash. Any embedded spaces must be url-encoded using + or %20. Here are some examples:
    coverage=/MOD_Grid_Snow_500m/NDSI_Snow_Cover
    coverage=/MOD_Grid_Snow_500m
    Coverage=/MOD_Grid_Snow_500m/NDSI_Snow_Cover,/MOD_Grid_Snow_500m/Snow_Albedo_Daily_Tile
    Coverage=/AerosolParameterAverage/Absorbing+optical+depth
    Coverage=/AerosolParameterAverage/Absorbing%20optical%20depth


  4. How do I work with a large number of granules?
    When there are multiple granules returned from the CMR query, they are returned in sets called pages. The default page size is 10 granules. The page_size KVP allows the user to change the page size. The page_num KVP is used to select which page is used. EGI processes the page of granules as a single request and streams back a zip file with the resulting data files for all the granules in the page. Note that the system configuration parameter MAX_GRANS_FOR_SYNC_REQUEST limits the size of a request in EGI. Exceeding that number returns an error. Page_size should always be set to less than or equal to this request limit.

  5. How do I know what the request granule count limits are?
    Query the capabilities service for the collection. See the sample script cap.sh. The programmatic access request size limit is identified as maxGransSyncRequest.
    To work with more granules than this limit, you must divide the granules into batches with a size less than or equal to the limit (using the page_size parameter), and make a series of Programmatic Access requests for each batch using the page_num parameter. Remember, the page_size parameter should be less than or equal to maxGransSyncRequest.

  6. How do I know how many granules are returned from the CMR query?
    Make a CMR only request with the desired search constraints. CMR provides an XML response containing the granule count in the “hits” element. Use this value along with page_num to issue the necessary count of service requests.
    Alternatively, just issue a series of service requests for successive pages until there are no more granules in the response. When CMR returns no granule in a response page, the EGI returns an HTTP redirect pointing to CMR. This redirect returns the CMR XML response indicating the total number of hits.
    Another way to get the count of matching granules is to submit the full request with a page_size=0. This will automatically return the redirected CMR url, which can be used to supply the hit count as above.

  7. How do I know how to specify the version parameter?
    The version is stored in CMR as a string. For ECS ESDTs, the version is usually an integer number with up to three digits. Depending on how the number was formatted when the collection was established in CMR, the number may have leading zeros. Example, MOD10A1 version 5 and MOD10A1 version 006. When specifying the version for the CMR query in a Programmatic Access request, the version string must match exactly. To find the version string value for a collection, issue a collection query directly to CMR using the shortname. For example:

    curl ‘https://cmr.earthdata.nasa.gov/search/collections.json?short_name=MOD10A1&pretty=true’ | grep version_id
    View the response and examine the version_id element for each version of that shortname.

  8. Why do I need a token?
    ESDIS requires that data distribution shall only go to Earthdata Login registered users.
    The user supplied token is used as proof that the user has registered with the Earthdata Login system. It is used for:
    - enabling user access to ACL protected granule results in CMR
    - enabling Programmatic Access delivery of results only to registered users
    - metrics collection
    You can register here: urs.earthdata.nasa.gov

  9. How do I create a token?
    To get a token, the user must interact with the Earthdata Login system to supply a registered username and password, and receive back a unique token string. The token is valid for a limited period determined by Earthdata Login. Here is the CMR wiki page describing how to obtain a token. The sample getatoken.sh script uses this technique and parses out the token string and saves it in a file.

  10. How do I get the full resolution browse files for ASTER L1T ?
    Use these parameters: subagent_id=FRI and format=[ BOTH | VNIR | TIR ]

  11. How do I find the latest granule that meets my criteria?
    Use the sort_key[] parameter for the CMR query.
    Example:

    curl “https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10A1&version=006&sort_key%5B%5D=-start_date”

  12. How do I find granules using PSAs?
    Use the attribute search parameter for CMR in your CMR KVPs. This parameter is “Attribute[]=type,name,value”. Note that the brackets must be escaped or encoded.
    For example, to use the NominalPassIndex product specific attribute:

    curl -OJ “https://n5eil01u.ecs.nsidc.org/egi/request?short_name=ae_l2a&attribute%5B%5D=int,NominalPassIndex,374&Coverage=/High_Res_B_Swath/Res5B_Surf&bbox=0,-40,75,80&format=GeoTIFF”
    (The O and J options will save the returned GeoTIFF into an appropriately name file.)

  13. How do I find MODIS data for a specific sinusoidal grid tile ?
    The sinusoidal grid formatted collections have Product Specific Attributes (PSAs) that indicate the tile, HORIZONTALTILENUMBER and VERTICALTILENUMBER.
    For example, to find the latest MOD10A1.006 granule covering the East Coast of the US:

    curl “https://n5eil01u.ecs.nsidc.org/egi/request?short_name=MOD10A1&version=006&attribute%5B%5D=int,HORIZONTALTILENUMBER,11&attribute%5B%5D=int,VERTICALTILENUMBER,5&sort_key%5B%5D=-start_date”

  14. How do I get the stitched result of a GLAS request ?
    The GLAS subsetter produces two types of results when processing multiple granules. The primary result is the set of individual granule results packaged in a zip file. This is what is returned by default in the stream result from programmatic access request. The second type of result is a single h5 file containing the stitched results from all the granules processed. To obtain this file, use the request_mode=async parameter to force the result into a file rather than the default stream. The http response stream will then contain an xml formatted structure containing a url to an esir location where the stitched result can be downloaded. If the email= parameter is used, then the same information will be sent in the notification email.

  15. How do I URL encode parameter values ?
    The programmatic access url characters should be encoded using standard “Percent Encoding“
    The problematic characters encountered in programmatic access urls are spaces and square brackets.
    spaces: spaces in data layer names can be encoded as %20 or replaced with a plus sign (+). The latter is more readable.
    brackets: left and right brackets should be encoded as %5B and %5D
    This simple sed command can perform this translation, as used in the example pa.sh script.

    sed ‘s:[:%5B:g;s:]:%5D:g;s: :%20:g’

  16. How do I specify bands to process for a multidimensional data set ?
    Bands are referenced using the “Coverage=” KVP, using an indexing syntax with square brackets.

Example using MISR MIL3DAE.004:
Coverage=/AerosolParameterAverage/Absorbing optical depth/OpticalDepth[1]/Band[1]

Example using SMAP SPL3FTA:
Coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw/Bands[1]

The brackets and embedded spaces must be encoded when used in the http url.
See “FAQ: How do I url encode parameter values?”

Some general rules:
* The Layers are separated with slashes, always starting with a slash. Trailing slash is optional.
* For any multi-dimensional layers (parameters or bands), the dimension is specified inside of square brackets.
* Dimension indexes start at 1. 0 is not a valid index and will cause an error.
* A range of dimensions can be specified like [2-4]. This specifies dimensions 2,3,4.
* The band layer spelling is data product dependent. (e.g. “Bands” for SPL3FTA, “Band” for MIL3DAE)
* The layers are case sensitive.
* Brackets must be encoded when used in the http url. [=%5B ]=%5D
* Embedded spaces must be encoded or replaced with a + ( =%20 or =+ )
* A dimensioned layer must use the bracket syntax if any other layer uses brackets.
* A coverage will include all dimensions of all layers lower than the lowest specified layer.
* Empty brackets specify all index values.

GOOD EXAMPLES
pa7.sh page_size=1 short_name=SPL3FTA format=GeoTIFF version=003 coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw/Bands[1]
pa7.sh page_size=1 short_name=SPL3FTA format=GeoTIFF version=003 coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw/Bands[]
pa7.sh page_size=1 short_name=SPL3FTA format=GeoTIFF version=003 coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw/
pa7.sh page_size=1 short_name=SPL3FTA format=GeoTIFF version=003 coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing+optical+depth/OpticalDepth[1]/Band[1]
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing+optical+depth/OpticalDepth[]
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing+optical+depth/OpticalDepth[1-2]/Band[1-2]
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing+optical+depth/OpticalDepth[1-2]/Band[]
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing+optical+depth/OpticalDepth[]/Band[]

BAD EXAMPLES
pa7.sh page_size=1 short_name=SPL3FTA format=GeoTIFF version=003 coverage=/Freeze_Thaw_Retrieval_Data/freeze_thaw/Bands
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing%20optical%20depth/OpticalDepth[]/Band
pa7.sh page_size=1 short_name=MIL3DAE version=4 format=GeoTIFF coverage=/AerosolParameterAverage/Absorbing%20optical%20depth/OpticalDepth/Band[]

  • No labels