Versions Compared

Old Version 1

changes.mady.by.user user-787a4

Saved on

compared with

New Version 2

changes.mady.by.user user-787a4

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Programmatic Access of EOSDIS DAAC Hosted Services

User Documentation


Anchor
Overview
Overview
Overview

Anchor
whatisprogrammaticaccess
whatisprogrammaticaccess
What Is EGI Programmatic Access ?

Programmatic Access is a capability enhancement to the Data Access services at EOSDIS Service Interface (ESI) enabled DAACs. (The ESI enabled DAACs are NSIDC, LP-DAAC and ASDC.)
As seen in the diagram below, it adds an ability for EGI (the ESI Gateway Interface) to access CMR (the Common Metadata Repository), and extends the accessible protocols to include WCS (Web Coverage Service) compatibility. This improves the scriptability of the EGI component, which is the exposed user program interface into the data access services. 

The Programmatic Access interface is used to locate and access DAAC hosted data, optionally performing services on the data. It combines into one interface functions that formerly required multiple interfaces: searching CMR for science granules of interest, and then obtaining the science data files with optional services applied. The interface supports synchronous, streaming synchronous, and asynchronous modes of operation. The Programmatic Access Interface requires client authentication through Earthdata Login. 



The EOSDIS Services Gateway Interface Context


Anchor
whatdoesitlooklike
whatdoesitlooklike
The EGI Request

EGI Programmatic Access is exposed as a REST interface using the HTTP protocol. It takes the form of an HTTP URL, containing a series of key-value-pairs (KVPs) as query parameters that specify the operands and the operations performed. The URL incorporates these elements:

...

This request and its response can be communicated with the DAAC using any HTTP client program, including browsers, command line utilities, and custom programs. A flexible command line utility named “curl” is commonly used for this purpose. The examples on this page show how to use curl to request programmatic access services and receive the resulting output files. Alternatively, a user can write a custom program in any language to implement their desired workflow, using a curl library or the native http communication functions of the language. An example demonstrating programmatic access of CMR over HTTP written in Python can be found here: https://git.earthdata.nasa.gov/projects/HDS/repos/cmr/. This can be readily extended to encompass the EGI Programmatic Access functionality described herein.

Anchor
egiresponse
egiresponse
The EGI Response

The response returned by the EGI depends upon the request parameters. By default, a request is submitted in 'streaming synchronous' mode. This means that a response will not be returned to the client until the request has finished processing, and the results of the request will be returned directly to the client - usually a zip file containing all the resulting granules.

...

Requests that invoke CMR queries but do not otherwise specify any form of granule processing or reformatting will return the XML formatted CMR response directly to the client.


Anchor
usingcurltocommunicatewithegi
usingcurltocommunicatewithegi
Using curl to Communicate With EGI

Anchor
thecurlprogram
thecurlprogram
The Curl Program

Curl is an extensive program (and library) that supports many internet protocols and options. Here we are focusing only on the HTTP protocol and options that are useful for EGI programmatic access. For more documentation, see the curl man page on your system.

...

                        in our development environment

Anchor
exampleshellscript
exampleshellscript
Example Shell Script

The pa.sh (updated) example shell script implements the EGI Programmatic Access Request/Response flow. The script must be edited to utilize the desired EGI endpoint. The HOST and MODE variables should be changed to match the target environment. (Note, this is a bash shell script developed for use on Unix type hosts.)

...

  1. Perform the HTTP request using curl 
  2. Examine the curl status return code
    Non-zero status indicates curl encountered an error, so we display this information and exit. The most common problems are “connection timeout” and “file already exists” when attempting to write the output file.
  3. Examine the HTTP response status code
    a. if return_code=200 then all is OK and the result is in the correctly named output file (from the OJ options)
    b. if return_code=303 then we have a redirect case; examine the redirection URL contained in the 'Location' header. 
      • if the redirect url is for CMR, then follow it and flow CMR results to the screen. Note that in this case we also clean up the empty output file created by the initial curl.
      • if the redirect url is not for CMR, then follow it with -O -J to save result with the correct EGI zip file name
    c. otherwise, any other return code indicates an error and the output file should be the xml format error info


Anchor
edl
edl
Earthdata Login Integration

If the EGI is configured behind Earthdata Login, any client accessing the EGI interface will need to handle the Earthdata Login OAuth2 user authentication process. Detailed examples of how to do this in various programming languages can be found here.

...

Code Block
> curl https://f5eil01.edn.ecs.nasa.gov/egi_DEV01/request?token=38C67342-5D86-15C2-F207-3F0F6D244092&<parameters>


Anchor
session
session
Establishing an Earthdata Login Session

When Earthdata Login is required for EGI access, EGI requests with very long URLs, for example, those using large shapefiles, can fail if your client program (e.g. curl, wget) has not already logged in and established a session. This is a due to the nature of the OAuth2 authentication process when credentials must be provided. In this case, it is easy to establish a session prior to sending your request simply by sending an empty request as follows:

...

Note: The use of CMR tokens eliminates the need to authenticate via Earthdata Login, so the same URL length limitations do not exists. 



Anchor
programmaticaccessparameters
programmaticaccessparameters
Programmatic Access Parameters

Programmatic Access parameters, or key-value pairs (KVPs) are represented in a request as URL query parameters, separated with an ampersand '&'. KVPs can be EGI parameters, CMR parameters, or a mix of both. Any parameter not recognized as a valid EGI parameter is passed to CMR.

...

The following table gives some useful CMR parameters; however any CMR supported parameter can be used. 

Anchor
cmrparameters
cmrparameters
Useful CMR parameters: (WCS Endpoint Parameters)

Short_name=aaaa

Specifies the short name of the collection used to find granules for the coverage requested. Can be used multiple times to return granules from multiple collections

Version=nnn

Specifies collection version. The version is treated like a string and must match the version field for that collection in CMR. Multiple versions can be specified.
Note: the Version parameter is also used by WCS clients to specify the version of WCS. When version=1.0.0, EGI understands it to be specifying the WCS versions. Otherwise Version is passed to CMR as a search parameter.

Discussion: The format of the version parameter depends on the the metadata that was provided to CMR when the collection was created. See the FAQ. 

Updated_since=<datetime>

Can be used to find granules recently updated in CMR. 
Example datetime: 2016-09-01T12:00:00Z

Bounding_box=n,n,n,n

This specifies a search filter to find only granules having a spatial extent that overlaps this bounding box, specified in decimal degrees of latitude and longitude. 
Order is lower left long, lower left lat, upper right long, upper right lat. 
This order is referred to as WSEN and is the same order as used for the EGI subsetting Bbox.

Time=<datetime>,<datetime>

Specify data datetime range filter for the CMR query. "Time" is the WCS compatible equivalent of the CMR "Temporal" parameter. 
Note, see the Time keyword for EGI described below for temporal subsetting usage.

The start and end time values should be specified as a compound date and time: year-month-dayThours:minutes:secondsZ. Year-month-day is mandatory, the time part is optional. Year-month-day must use 4 digits for the year, two digits for the month and two digits for the day, separated by hyphens. If the time part is used, it must start with T and it must include three subfields for hours, minutes and seconds, separated with colons. The trailing Z is optional. It is standardly used to indicate GMT time, however all time parameters used in EGI Programmatic Access are GMT times. If the time part is not used, it is equivalent to T00:00:00. 

The "time=" KVP must always contain two datetime values to specify start and end of a time window, separated with a comma. Here are some valid examples:
time=2016-01-01,2016-01-02
time=2016-01-01T12:00:00Z,2016-01-01T18:00:00Z
time=2016-01-01T00:00:00,2016-01-01T23:59:59 

sort_key[]=<sort-option>

This is a CMR parameter to control the sort order of the returned results. Sort options are described here in the CMR documentation. 
Example: sort granules by data coverage date in reverse order (newest first): 
sort_key[]=-start_date


Anchor
egiparameters
egiparameters
Useful EGI parameters: (WCS GetCoverage Parameters)

Coverage=/group/sub-group/sub-sub-group/dataset

WCS: Used to specify the coverage to be processed. Specifies the subset data layer or group for Parameter Subsetting. Multiple datasets can be specified separated by commas. 

The dataset value always starts with a slash and the group-subgroup hierarchy are separated with slashes. If only a group or subgroup is specified, all lower level datasets are included in the processing. 

Bbox=<W>,<S>,<E>,<N>

WCS: Bounding Box used for spatial subsetting. Coordinates are in decimal degrees. This is the same order as used for the CMR spatial filter parameter

Time=<start_datetime>,<end_datetime>

WCS Keyword: Specify data datetime range filter for the CMR query. 
Note, the Time keyword is a shared CMR/egi keyword that also invokes temporal subsetting/stitching for applicable data sets.

WCS: Used for Temporal subsetting.
Note: The Time keyword also is used as a CMR temporal filter.

SUBSET=time("<datetime>")
SUBSET=lat(<min>,<max>)
SUBSET=lon(<min>,<max>)
SUBSET=x(<min>,<max>)
SUBSET=y(<min>,<max>)

WCS: SUBSET=time("<datetime>") is used for Temporal subsetting with the datetime as the specified start time.

WCS: Minimum and Maximum values to be used for the Bounding Box for spatial subsetting. Coordinates are in decimal degrees(lat/lon) or in meters(x/y).


Format=<format>

WCS: Optional output file format specifier used for re-formatting. Supported values vary by data type.
[GeoTIFF, HDF-EOS5, NetCDF4-CF, NetCDF-3, ASCII, HDF-EOS, KML, Shapefile] 
If this parameter is not used, then the output format is the same as the input format (no reformatting).

BoundingShape=<GeoJSON content>

Optional, used for polygon subsetting. 
GeoJSON content string. 
Note: When drawing a polygon that crosses Anti-Meridian, need to redefine negative longitude values to > 180, vise versa, positive longitude values to < -180 in order for it to render correctly and for ogre to convert it correctly to geojson format.

shapefile=@<KML/Shapefile>Optional, used for polygon subsetting. 
[KML, Shapefile] 
Note: When drawing a polygon that crosses Anti-Meridian, need to redefine negative longitude values to > 180, vise versa, positive longitude values to < -180 in order for it to render correctly and for ogre to convert it correctly to geojson format.
EMAILSpecifies the email address to which any notifications will be sent. The special values yes and true are used to indicate that the email address should be obtained from the Earthdata Login user profile for the user submitting the request. These special values can only be used if the EGI is configured behind EDL and user credentials are provided OR a CMR token is submitted with the request.


Anchor
adminparameters
adminparameters
Administrative and formatting parameters:

page_size=<n>

This is a CMR parameter to control the number of granules in the page of returned results. 

Discussion: When there are multiple granules returned from the CMR query, they are returned in sets called pages. The default page size is 10 granules. The page_size KVP allows the user to change the page size. Multi-page results can be accessed one page at a time using the page_num KVP described below. 

Note that the system configuration parameter MAX_GRANS_FOR_SYNC_REQUEST limits the size of a request in EGI. Exceeding that number returns an error. Page_size should always be set to less than or equal to this request limit.

See the FAQ for more discussion. 

page_num=<n>

This is a CMR parameter to select the page of results to be processed. The page contains the number of granules selected in the page_size KVP. 

Version=1.0.0

WCS Clients Only: Indicates WCS Version 1.0.0 compatibility (optional) 

Request=GetCoverage

WCS Clients Only: Identifies the type of WCS request (optional) 

token=<token>

Allows the user to provide an Earthdata Login token. This token is used as proof that the user has been authenticated by the Earthdata Login system. It is used for:

  • enabling user access to ACL protected granule rsults in CMR
  • enabling Programmatic Access delivery of results only to authenticated users
  • metrics collection 


Anchor
usageexamples
usageexamples
EGI Programmatic Access Usage Examples

The following examples address the set of scenarios identified in the NSIDC request for the Programmatic Access capability. Note that Programmatic Access can generally be used to search CMR and access any data sets configured for ESI processing at any of the ECS DAACs that use ESI. These examples show the URL that can be used directly with curl, as well as the command line to invoke the sample script. The following code block shows an example curl command that could be used with an example URL.

...

Anchor
scenario1
scenario1
Scenario 1 SPL3SMP Spatial and Parameter Subsetting and Reformatting to GeoTIFF

SPL3SMP Characteristics

  • Name: SMAP L3 Radiometer Global Daily 36 km EASE-Grid Soil Moisture V003
  • Format: HDF-5
  • Spatial Extent: Daily Global Composite, Bounding Rectangle: (85.0445°, -180°, -85.0445°, 180°)
  • Organization: 31 data sets in one group

Service Request Description

  • find SPL3SMP granules from Jun 1-3 2016
  • select soil moisture parameter
  • spatially subset over region of interest
  • reformat into GeoTIFF

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SPL3SMP \
	&version=003 \
    &time=2015-03-30,2015-04-20 \
    &Subset_Data_Layers=/Soil_Moisture_Retrieval_Data/soil_moisture \
    &Bbox=100,-20,140,20 \
    &format=GeoTIFF"
Code Block
pa.sh short_name=spl3smp \
    version=003 \
    time=2015-03-30,2015-04-20 \
    Subset_Data_Layers=/Soil_Moisture_Retrieval_Data/soil_moisture \
    Bbox=100,-20,140,20 \
    format=GeoTIFF


Anchor
scenario2
scenario2
Scenario 2 SPL2SMA Reformatting to geotiff

SPL2SMA Characteristics

  • Name: SMAP L2 Radar Half-Orbit 3 km EASE-Grid Soil Moisture V003
  • Format: HDF-5
  • Spatial Extent: Single Orbit Swath, Bounding Rectangle: (85.0445°, -180°, -85.0445°, 180°)
  • Organization: 80 data sets in four groups

Service Request Description

  • find SLP3SMP granules within a temporal window
  • select the Radar_Data group
  • spatially subset to region of interest
  • reformat to GeoTIFF

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SPL3SMA \  
    &version=003 \  
    &time=2015-04-20,2015-04-27 \  
    &Subset_Data_Layers=/Radar_Data \  
    &Bbox=60,10,100,30 \  
    &format=GeoTIFF"
Code Block
pa.sh short_name=spl3sma \
    version=003 \
    time=2015-04-20,2015-04-27 \
    Subset_Data_Layers=/Radar_Data \
    Bbox=60,10,100,30 \
    Format=GeoTIFF

Anchor
scenario3
scenario3
Scenario 3 GLAH12 Spatial and Parametric Subsetting

GLAH12 Characteristics

  • Name: GLAS/ICESat L2 Global Antarctic and Greenland Ice Sheet Altimetry Data (HDF5) V034
  • Format: HDF5
  • Spatial Coverage: Global Extent Bounding Rectangle: (90.0°, -180.0°, -90.0°, 180.0°)
  • Organization: 173 Datasets in 5 major groups; Each granule contains 14 Orbits 

...

  • find GLAH12 version 034 granules with data date of April 12-13 2007
  • select only the 1HZ data group
  • spatially subset to a lat-lon box

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=GLAH12 \
    &version=034 \  
    &time=2007-04-12T00:00:00,2007-04-14T00:00:00 \  
    &Coverage=/Data_1HZ \  
    &bbox=0,-80,100,80"
Code Block
pa.sh short_name=glah12 \
    version=034 \
    time=2007-04-12T00:00:00,2007-04-14T00:00:00 \
    Coverage=/Data_1HZ \
    Bbox=0,-80,100,80


Anchor
scenario4
scenario4
Scenario 4 MOD10A1 reformatting to GeoTIFF, plus spatial and parameter subsetting

MOD10A1 Characteristics

  • Name: MODIS/Terra Snow Cover Daily L3 Global 500m SIN Grid V006 VERSION 006
  • Format: HDF4
  • Spatial Extent: MODIS Sinusoidal Tile Grid 

Service Request Description

  • Find MOD10A1 version 6 granules from January 1 2011 between 00:00 and 02:00
  • Over Eastern Asia (130,30,140,85)
  • Select the NDSI_Snow_Cover dataset
  • Spatially subset to the desired area (130,30,140,85)
  • Convert to GeoTIFF

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=MOD10A1 \  
    &version=006 \  
    &time=2011-01-01,2011-01-01T02:00:00 \  
    &bounding_box=130,30,140,85 \  
    &Subset_Data_Layers=/MOD_Grid_Snow_500m/NDSI_Snow_Cover \  
    &Bbox=130,30,140,85 \  
    &format=GeoTIFF"

...

Anchor
scenario5
scenario5
Scenario 5 MOD10A1 No Processing, returning raw data with no customization services (subsetting, reformatting or reprojection) applied

MOD10A1 Characteristics

  • Name: MODIS/Terra Snow Cover Daily L3 Global 500m SIN Grid V006 Version 6
  • Format: HDF4
  • Spatial Extent: MODIS Sinusoidal Tile Grid

Service Request Description

  • Find MOD10A1 version 6 granules from December 8, 2002 between 00:00 and 11:00
  • Over Guam (144.0, 13.0, 146.0, 14.0)
  • Return the data without any customization services (subsetting, reformatting or reprojection) applied

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=MOD10A1 \  
    &version=006 \  
    &time=2002-12-08T00:00:00,2002-12-08T11:00:00 \
    &bounding_box=144.0,13.0,146.0,14.0 \
    &AGENT=NO"
Code Block
pa.sh short_name=MOD10A1 \ 
    version=006 \  
    time=2002-12-08T00:00:00,2002-12-08T11:00:00 \  
    bounding_box=144.0,13.0,146.0,14.0 \
    AGENT=NO

Anchor
scenario6
scenario6
Scenario 6 ICESat-2 ATL08 Temporal Subset 

ICESat-2 ATL08 Characteristics

  • Name: ICESat-2 L3A Land and Vegetation Height version 001
  • Format: HDF5
  • Spatial Extent: Along Ground-Track
  • Organization: 6 ground track groups 

Service Request Description

  • Find ATL08 granules from December 21, 2020 between 10:00 and 12:00
  • Select the ground track 1 land_segments group
  • Temporally subset to the desired time frame (2020-12-21T10:00:00 to 2020-12-21T12:00:00)

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL08 \  
    &version=001 \  
    &time=2020-12-21T10:00:00,2020-12-21T12:00:00 \  
    &Subset_Data_Layers=/gt1l/land_segments"
Code Block
pa.sh short_name=ATL08 \ 
    version=001 \  
    time=2020-12-21T10:00:00,2020-12-21T12:00:00 \   
    Subset_Data_Layers=/gt1l/land_segments



Anchor
scenario7
scenario7
Scenario 7 ICESat-2 ATL06 Polygon Subset - KML

ICESat-2 ATL06 Characteristics

  • Name: ICESat-2 L3A Land Ice Height version 001
  • Format: HDF5
  • Spatial Extent: Along Ground-Track
  • Organization: 6 ground track groups 

Service Request Description

  • Find an ATL06 granule,158739
  • Select the ground track 1 land_ice_segments group
  • Using KML passed to subset to the desired area
  • Convert to Shapefile

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    -F "shapefile=@glacier.kml" \
    "https://n5eil02u.ecs.nsidc.org/egi/request?short_name=ATL06 \  
    &version=001 \  
    &Subset_Data_Layers=/gt1l/land_ice_segments \
    &Format=Shapefile \
    &time=2015-10-27T04:59:53,2015-10-27T05:00:07"
Code Block
pa.sh short_name=ATL06 \  
    version=001 \  
    Subset_Data_Layers=/gt1l/land_ice_segments \
    time=2015-10-27T04:59:53,2015-10-27T05:00:07 \
    Format=Shapefile


Anchor
scenario8
scenario8
Scenario 8 ICESat-2 ATL06 Polygon Subset - Shapefile

ICESat-2 ATL06 Characteristics

  • Name: ICESat-2 L3A Land Ice Height version 001
  • Format: HDF5
  • Spatial Extent: Along Ground-Track
  • Organization: 6 ground track groups 

Service Request Description

  • Find an ATL06 granule,158875
  • Select ground track 1 land_ice_height
  • Using Shapefile(.zip) passed to subset to the desired area
  • Convert to Shapefile

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    -F "shapefile=@new_crossing.zip" \
    "https://n5eil02u.ecs.nsidc.org/egi/requestshort_name=ATL06 \
    &version=001 \
    &FILE_IDS=158875 \
    &Subset_Data_Layers=/gt1l/land_ice_height \
    &Format=Shapefile"
Code Block
pa.sh short_name=ATL06 \  
    version=001 \  
    Subset_Data_Layers=/gt1l/land_ice_segments \
    time=2015-10-27T04:59:53,2015-10-27T05:00:07 \
    Format=Shapefile

Anchor
scenario9
scenario9
Scenario 9 ICESat-2 ATL03 Polygon Subset - GeoJSON 

ICESat-2 ATL03 Characteristics

  • Name: ICESat-2 L2A Global Geolocated Photon Data version 001
  • Format: HDF5
  • Spatial Extent: Along Ground-Track
  • Organization: 6 ground track groups 

Service Request Description

  • Find an ATL03 granule
  • Select the ground track 1 left group
  • Using GeoJSON content passed to subset to the desired area
  • Convert to Shapefile

Request Examples

Code Block
curl -s -O -J -w “%{http_code}\n%{url_effective}\n%{redirect_url}\n%{filename_effective}\n” \
    -nL -b ~/.cookies -c ~/.cookies \
    –dump-header response-header.txt \
    "https://n5eil02u.ecs.nsidc.org/egi_DEV06/request? \
    short_name=ATL03 \
    &BoundingShape={"type":"FeatureCollection","crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:OGC:1.3:CRS84"}},"features":[{"type":"Feature","properties":{"Name":null,"description":null,"timestamp":null,"begin":null,"end":null,"altitudeMode":"clampToGround","tessellate":-1,"extrude":0,"visibility":-1,"drawOrder":null,"icon":null},"geometry":{"type":"Polygon","coordinates":[[[-147.25640727423652,52.95206360966622],[-145.9181782282926,53.93605555521322],[-143.83211530373293,54.05413458867886],[-142.9268427138297,52.991423287488104],[-142.5332459356109,51.968071664119215],[-144.18635240412988,48.504420015793755],[-144.3044314375955,46.61515548034351],[-143.63531691462353,45.19820707875582],[-141.43117495659826,43.93869738845565],[-139.3844717098605,42.99406512073053],[-139.58127009896987,41.57711671914284],[-143.43851852551416,39.215536049830035],[-147.25640727423652,52.95206360966622]]]}}]} \
    &version=001 \
    &producer_granule_id=ATL03_20151027T044311_00810302_942_01.h5 \
    &Subset_Data_Layers=/gt1l \
    &format=Shapefile \
    &subagent_id=ICESAT2"
Code Block
pa.sh short_name=ATL03 \  
    version=001 \  
    Subset_Data_Layers=/gt1l \
    BoundingShape={"type":"FeatureCollection","crs":{"type":"name","properties":{"name":"urn:ogc:def:crs:OGC:1.3:CRS84"}},"features":[{"type":"Feature","properties":{"Name":null,"description":null,"timestamp":null,"begin":null,"end":null,"altitudeMode":"clampToGround","tessellate":-1,"extrude":0,"visibility":-1,"drawOrder":null,"icon":null},"geometry":{"type":"Polygon","coordinates":[[[-147.25640727423652,52.95206360966622],[-145.9181782282926,53.93605555521322],[-143.83211530373293,54.05413458867886],[-142.9268427138297,52.991423287488104],[-142.5332459356109,51.968071664119215],[-144.18635240412988,48.504420015793755],[-144.3044314375955,46.61515548034351],[-143.63531691462353,45.19820707875582],[-141.43117495659826,43.93869738845565],[-139.3844717098605,42.99406512073053],[-139.58127009896987,41.57711671914284],[-143.43851852551416,39.215536049830035],[-147.25640727423652,52.95206360966622]]]}}]} \
    Format=Shapefile

Anchor
retrievingrequestresults
retrievingrequestresults
Retrieving Request Results 

After successful completion of a submitted request, the results can be downloaded from the EGI (ESIR) using standard HTTP requests, either via a web browser or through a command line tool such as wget or curl. The request response (emails in the case of an asynchronous request) will list the URLs available to download the results. In the case of a multi-file request, this will include a URL for a ZIP file. It is also possible for a DAAC to configure limits on the maximum number of files or the total file volume permitted for a single zip file. In this case, a request that exceeds the configured limits will be split across multiple ZIP files, and URLs will be provided for each ZIP file necessary to download the complete results set.

...

(don't forget to set the email query parameter). This should take just a minute to complete, and will send an email upon completion of similar to this:

        Status update for ECS data processing request 23002

Your request is currently complete . Your request has completed processing. You may retrieve the results from the download URLs until 2018-12-17 13:28:33.289

Note from Client: To view the status of your request, please see: http://search.sit.earthdata.nasa.gov//data/retrieve/0435908906 

The output of this request can be downloaded from the following URLs:

...

Note that in order to retrieve the data, you must configure wget or curl for URS authentication. When using a web browser, the browser will prompt the user for their credentials if it has not already done so.


Anchor
errorresponses
errorresponses
Error Responses

Two types of errors can be encountered: errors encountered by the operating system when executing the curl program, and errors encountered by the curl program when communicating using the HTTP protocol. To detect either kind of error, the return codes should always be checked.

...

  • HTTP Return Code 303 “See Other”, this indicates a redirect to another url.
  • HTTP Return Code 302 "Moved Temporarily" This response is only generated when the EGI is configured behind Earthdata Login. Your client must be configured to handle the required user authentication. See Earthdata Login Integration.
  • HTTP Return Code 400 “Bad Request”.
  • HTTP Return Code 404 “Not Found”.
    • a CMR-only query
    • a request that matched no granules in the CMR query 
    • Collection not configured for that operation
    • Request exceeds configured limits
    • Parameter not recognized by CMR 
    • Requested processing produced no output data 

Anchor
configurationandhardening
configurationandhardening
EGI Programmatic Access Configuration and Hardening

The Programmatic Access capability is configured by the DAAC staff specifically for each execution environment (mode). 

The Programmatic Access capability also includes features for hardening the interface to mitigate the risk of user activities affecting DAAC operations. 

EGI Programmatic Access System Configuration

Each DAAC operational mode is configured as a distinct data provider to a particular instance of CMR. 
The programmatic Access components are configured so that the query communicates with the correct instance of CMR (identified by the endpoint) and searches the correct provider metadata (identified by the provider= parameter in the url).

...

    EDF DEV02 endpoint: http://f5eil01v.edn.ecs.nasa.gov/dev02/egi/request?

Hardening Considerations

The EGI components that implement Programmatic Access have been hardened to protect the DAAC systems from excessive load. 

...

  1. Transaction Size Limit
    A single request can result in a large number of granules returned from CMR. A configuration parameter allows DAAC operators to set a limit on how many granules can be processed in a single request. An end user who needs to process more than the limit can submit multiple requests, using the CMR page_size and page_num parameters.
  2. Cancel Request 
    If a Programmatic Access request for numerous granules is consuming too many resources (such as CPU or memory), the DAAC operations staff can cancel the request. This takes effect after the current granule finishes processing.

Anchor
references
references
References:

CMR search API
https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html

...