Chapter 2: Getting Started

This chapter will discuss

• creating and managing user accounts
• CMR session management - creating, using, and deleting tokens to provide authorization

If searching for and retrieving publicly available data is the only desired operation, this section can be skipped and the reader can go straight to Chapter 3.

User Accounts

User accounts are used to get access to restricted data, manage privileges, or to interact with other services and tools provided by the CMR or ECHO. User accounts for the CMR system are created and manage by the EARTHDATA Login (URS) system. If you need an account and don't already have one, please click on EARTHDATA Login to create one. Once created you can always go back to EARTHDATA Login to manage it. If you are part of a Data Provider group or other team the team administrator can set up permissions for you to access their restricted data. If you need special privileges you can always contact the CMR operational team at support@earthdata.nasa.gov and they can help you. 

Creating and Managing CMR Sessions

The CMR uses tokens in request messages - the http call to CMR - to validate per request who the requester is and what privileges they have. For most searches, a token is not needed because the metadata records are open to everyone. When certain metadata records are restricted a token is needed so that privileged users can see and access those records.  A Session is nothing more than a series of requests that use the same token meaning that you can use the same token for many requests before you delete it. All tokens expire at the end of a time period; At the time of this writing the duration is 30 days. Because the token is used to track your session, it must be protected by client applications with the same level of security that you use for your login name and password.

To conduct a session the normal steps are:

• Create a token
• Do one or more of the following in any order:
  • Search for records
  • Retrieve records
• Delete the token 

Create a Token

Now that the client partner has created a token, they can search and retrieve records, and conduct other functionality through the CMR or ECHO APIs. This functionality is covered in later chapters of this document. Once finished interacting with the CMR the token can be deleted.

Delete the Token

Chapter 3: Searching for metadata

Client partners can search the CMR for metadata.  Currently clients can search for collection and granule metadata.  In the future clients will also be able to search for metadata describing services, visualizations, parameters (variables), and documents.

CMR Environment URLs

The CMR system has three environments: The Systems Integration Test environment is where the CMR development team tests new functionality. This is the environment that first gets the newest upgrades, but it is the least stable. Once the CMR software has been tested it gets deployed to the User Acceptance Test environment. The environment here is quite stable and it is tested as a system for a couple of weeks before the software is deployed to the operational environment. Client Partners can test their software in either the SIT or UAT environment depending the level of integration and testing. The operational environment is the live system available to users around the world.  All of the examples provided in the rest of the document are using the Systems Integration Test environment.  To run the commands in the other environments just replace the SIT URL with either the UAT or OPS URL to use the API in the respective environments.

CMR Environments

Headers

Headers are a part of HTTP requests and for the CMR they provide information such as the content of the message (Content-Type), tokens to allow increased privileges (Echo-Token), the format of the data that gets returned (accept), etc. Content-Type is a standard HTTP header that specifies the content type of the body of the request for POST method messages. Search and retrieval requests support the following Content-Types. If the Content-Type is not specified, XML is assumed.

Content-Type headers

The Echo-Token allows the CMR to know who is making a request. The Token is in the format of XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.  A token must first be generated as described in the previous section. Once the requester has the token, the token can be placed into the http header for the necessary API calls.

If the caller wishes to control in what format and or specification the data gets returned they can use the Accept header.  The following table lists the valid values.  If this header or an alternative method is not used, the returned outcome will be a reference list of results in XML format.

Accept Headers

For more information about the types please see the CMR API documentation.

Client-Id is another header that allows the client to specify a name.  This helps the CMR operations team monitor query performance per client and it can also make it easier for them to identify your requests if you contact them for assistance.

Following are some examples for using the headers. The purple part of the example will be explained in this section, the rest will be described later:

The following curl command issues a search request with the search parameters contained in a file called searchterms in the current directory. The Content-Type - the specification and format of the searchterms file - is using the json format to specify the search parameters. The accept header states that we want the results as a reference list using the XML format. The Echo-Token header allows the CMR to know who is making the request for authorization purposes. The Client-Id header allows the operations team to monitor performance and allows them to quickly find your requests should you need help.

In the next example a user is issuing a search request using publicly available data and is returned a full metadata record list of results. Notice that only the Accept header is needed.

In the next example a user is issuing a search request using publicly available data and the default result reference list.  Notice that no headers are needed.

As an alternate to using the Accept header are extensions where the client can use the Type Received name in the query to get the same results. Instead of using "Accept: application/opendata+json", "opendata" can be used at the end of the main query before parameters are specified. 

Other examples use the DIF 10 specification and the ISO specification respectively.

To change what is wanted in the request just replace the header as needed per the tables above.

Results

There are a several of types of results that can be returned.  For each of these types different formats are supported.:

1. A reference list of results
   1. XML
2. A list of results with partial metadata records being provided
   
   1. XML
   2. JSON
   3. ATOM
   4. CSV - supported for granules only
   5. KLM
3. A list of results with full metadata records being provided
   
   1. XML
   2. opendata

The following is an example of a reference list of results

The results specify

1. how many metadata records were found by the "hits" tag
2. how long the query took in milliseconds
3. a list of metadata record results specified by the "reference" tag

With in each reference tag a limited information about the metadata is provided.

1. The metadata name which corresponds to the UMM Entry Title
2. The CMR profile or concept ID -  a CMR generated unique ID.  The ID is encoded by a letter of the profile or concept (C for collection, G for granule, S for service), followed by a CMR generated number, followed by a "-" and then followed by the ID of the metadata provider.
3. The exact CMR location to download the metadata
4. The latest revision number of the metadata record.

The following is an example of a full metadata record list of results in the ECHO 10 specification using the XML format.

The results specify

1. how many metadata records were found by the "hits" tag
2. how long the query took in milliseconds
3. a list of metadata record results specified by the "result" tag

Within each result tag three attributes are shown about the metadata record followed by the full metadata record. The attributes display the CMR concept id (profile ID), the specification and format of the metadata, and the revision number of the shown metadata record.  For detailed information about the result specifications and formats available with examples please see the CMR API documentation.

Searching

There are several ways to search the CMR system all using the RESTful principles:

1. Using the API calls and parameters with the GET or POST methods
2. Using a JSON query language with a POST method
3. Using the Alternative Query Language (AQL)

API calls and parameters GET method

The most popular and preferred way is to use the API calls and parameters with the GET or POST methods. For detailed documentation the API documentation is located at https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html.

The CMR uses jetty as its application server and it is currently set to take roughly 500k characters in the URL. Clients using the Search API with query parameters should be careful not to exceed this limit or they will get an HTTP response of 413 FULL HEAD. If a client expects that the query url could be extra long so that it exceeds 500k characters, they should use the POST method for searching instead of the GET method. First we will describe search using the GET method.

The basic search command is shown in the following example 

The query returns the first 10 publicly available collection results in a reference list using the XML format.  As described in the header section to see restricted data, if you have the privileges you will need to use a token.

There are search parameters that can be applied to provide more functionality.  They are listed below

• page_size - The number of results per page - the default is 10. 0 and 2000 are the minimum and maximum values respectively.  For example: page_size=100 shows 100 result records per page if that many results exist.
• page_num - The page number to return. For example: page_num=1 is the first page of results;  page_num=2 is the second page of results; page_num=10 is the 10th page of results.
• sort_key - Indicates one or more elements to sort on. For example: sort_key[]=platform  (the brackets "[" and "]" may need to be escaped by using the \ character)
• pretty - Returns formatted - readable - results if set to true.  For example pretty=true.  For all the returned examples in this document this flag is used.
• token - Specifies the client token. This is an alternative to using the Echo-Token header.
• echo-compatible - This is used by systems requiring ECHO results. To get the best use out of CMR Client Partners shouldn't use this parameter.

These search parameters are for collection requests only.

• include_has_granules - Includes a has-granules tag or attribute in the response so the client knows if the collection encompasses any granules. E.g. include_has_granules=true
• include_granule_counts - Includes a granule-counts tag or attribute in the response with the number of granules represented by the collection. E.g. include_granule_counts=true
• include_facets - Includes a list of facets and their counts at the end of the results. This is mainly used for collection search displays. E.g. include_facets=true
• include_facets with hierarchical_facets - Includes a list of facets preserving the hierarchical order. This is mainly used for collection search displays. E.g include_facets=true&hierarchical_facets=true

 In next example box we will see a set of examples conducting a basic search with using the search parameters just described. The first example shows a client wanting to see 50 metadata references per page. The second example shows 50 metadata references per page using the formatted print.  The third example shows page 2 of 20 results per page showing full records in the echo 10 specification using formatted print. The fourth example issues a request including token and client id headers and does a basic search sorting the results using the platform element.  The request returns page 2 results of 20 results per page showing full records in the ISO 19115 specification using the XML format in a formatted fashion. As one can see the ? character separates the URL from the search parameters and the search parameters are separated by the & character. The parameters can be in any order.

The previous examples all demonstrated searches for collections. The same search parameters apply to granules if it isn't stated that a parameter applies only to collections. To conduct a granule search, just replace collections with granules in the URL.  Following are the same four search requests just listed above but for granules instead of collections.

Now to refine our searches we can use another set of search parameters documented in the table below. These parameters support collection searches and most of these parameters have the brackets next to them and may need to be escaped (\[\]) depending on the language used or how the query is being sent.  All of CMR time search parameters (temporal, updated_since, revision_date, and equator_crossing_date) formats are specified as yyyy-MM-ddTHH:mm:ss.SSSZ format. Where yyyy is year; MM is month; dd is day; T is the date time separator character; HH is the hour; mm is the minute; ss is the second; SSS is the milliseconds (the .SSS can be omitted); and Z specifies Zulu time.  January 2 2000 at 5 seconds and 4 minutes past 3 O'clock in the morning Zulu time is represented as 2000-01-02T03:04:05Z.

keyword=alpha

Collection Search Parameters

Documented in the table below are granule supported search parameters.

Granule Search Parameters

In the following example we wish to find all collection metadata records that contain an AQUA platform and we would like the to see only a formatted reference list of results that contain 20 references.

In the next example we wish to find all collection metadata records that contain an AQUA or an AURA platform and we would like the to see only a formatted reference list of results that contain 20 references.

There are a couple of extra options that certain search parameters have to aid the user.  To use these options the syntax is: options[parameter name][option_key]=value. Parameter name is the name of the search parameter to be affected such as platform. Value is either set to true or false.  Option_key is one of the following:

Extra Options

The following is an example of using options with the platform search parameter.  This example will find any platform that matches A followed by any number of alphanumeric characters and ends with A. This will find both platforms of AQUA and AURA. 

This next example demonstrates a user looking to find records that only contain an instrument that matches "HELLO" in uppercase letters.

Besides science_keywords, if any of the parameters that are searched are repeated, the metadata records that have ANY of the values will match.  The following example demonstrates that the CMR system will match any metadata record containing either value.

For a complete set of examples using all of the search parameters please see the API documentation: https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html

API calls and parameters POST method

The API using the POST method is the same as with the GET method with the exception being the method used is POST instead of GET and the parameters are in the body of the message without a length constraint instead of existing in the URL string.  Using the curl command the following example shows the query.xml file that contains the query we want to execute followed by the curl search request. In the query.xml file the parameters can be left as one long set or formatted to be more easily read - so long as syntax remains the same. Also notice in this example we did not escape the brackets ([ ])

query.xml:

Notice in this specific instance that the Content-Type header is not used.  Don't use it, it will cause an error.

JSON query language with a POST method

For those who understand the JSON format, the CMR provides a JSON RESTful interface. The elements that can be searched are the same as already described above but this interface is for collection searches only. This searching method does provide additional functionality of using conditions (AND, OR, NOT) against the elements to conduct a search.  See the JSON schema https://cmr.sit.earthdata.nasa.gov/search/site/JSONQueryLanguage.json for more details.  The example provided below demonstrates a query with conditions and uses several elements.  

curl -XPOST -H "Content-Type: application/json" https://cmr.sit.earthdata.nasa.gov/search/collections
-d '{"condition": { "and": [{ "not": { "or": [{ "provider": "TEST" },
{ "and": [{ "project": "test-project",
"platform": "mars-satellite" }]}]}},
{ "bounding_box": [-45,15,0,25],
"science_keywords": { "category": "EARTH SCIENCE" }}]}}'

Alternative Query Language (AQL)

The CMR supports the ECHO Alternative Query Language (AQL) if a client wishes to use this capability. While the AQL is supported it is not being enhanced nor modified to take advantage of new CMR features. For a very detailed explanation of AQL with examples of how to use it, please see the ECHO AQL documentation.  

Chapter 4: Retrieving Metadata

There are several ways of retrieving metadata.  The first way consists of getting a result list of full metadata records. This has already been demonstrated, but it is shown again here in the example below.

The response is a result list of full metadata records.  

Another way is to use the concept id to retrieve a record. The syntax is https://cmr.sit.earthdata.nasa.gov/search/concepts/<concept id>. One can also use the concept id/revision number if a specific revision is wanted. The syntax is https://cmr.sit.earthdata.nasa.gov/search/concepts/<concept id>/<revision number>. These examples are shown below.

The CMR supports retrieving metadata records using different standards.  The table below lists these standards.

Supported Standards

Below are several examples using the supported standards mime types.  The first example is retrieving a granule metadata record in the JSON format. The second example retrieves a granule metadata record with a revision of 8 in the ISO specification. The third example retrieves a collection metadata record with a revision of 7 in the DIF 10 specification. The fourth example lists a granule record using the native format with the pretty print option turned on.

Accessing data

After identifying data of interest through catalog queries, you may place a request for that data if the data provider(s) allow it. To create an request, it is critical that you understand the parts that make up a request. 
A request is a collection of request items. Each request item consists of:

• The GUID for the catalog item you wish to request
• The quantity of the item you wish to request

Other options associated with the item:
Many catalog items belonging to many different providers may comprise a single order. Each provider may have its own validation scheme and rules for creating, validating, and submitting particular catalog items. To ensure appropriate validation – as well as to demarcate a larger request for sending to each provider – requests are split up into smaller provider requests, sometimes called sub-requests. Each provider request includes all the request items in a particular request that belong to a specific provider. Each provider request has its own provider request GUID, a unique identifier comprised of the GUID of the provider and the GUID of the request that contains the catalog items for this provider. The following figure shows the structure of a request.

CMR Request Structure 

The CMR is designed to support a variety of providers and their various requesting needs, which brings some complexity to the process of request creation. The following figure illustrates the most commonly used and direct approach for creating and submitting a request.

The most common way to obtain the catalog items needed for creating a request is through the ExecuteQuery operation found in the Catalog Service. You can also specify how the results should be displayed. When doing so, ensure that the CatalogItemID tag is returned. The string value under this tag is the actual catalog item GUID that you can use to request items from the CMR.

Access Options 

Access Options use CMR Forms (refer to  http://www.cmr.nasa.gov/data_partners/data_tools10.shtml for more information) for providers to indicate additional information required from users in order to fulfill their orders, such as the media to put the ordered data on and the specific subset of the data to order. Providers assign Order Option definitions to catalog items. When you add an order item to an order that has required order options, it must contain an option selection that corresponds to one of the assigned option definitions for the catalog item you are ordering. See section 1.3 of the CMR Forms Specification for more details. 
To list the possible options for each catalog item, use the GetCatalogItemOrderInfomation operation on the 
OrderManagementService. The operation returns one object called CatalogItemOrderInformation. This contains a list of catalog items and a list of option definitions. Not every option definition in the list applies to every catalog item returned. Each catalog item is associated with zero or more option definition GUIDs, which will be included in the list returned in CatalogItemOrderInformation. Use only one selection from one of the definitions when you add a catalog item to the order. Use your discretion when choosing the definition to use for the order and the selection from within that definition.

The code listing below shows an example of getting the order options for two granules. 

Code Listing 56: Getting Order Options for a Catalog Item

Creating an Order

Once you have collected the catalog item GUIDs and associated options, you can create an order by calling the CreateOrder operation on the OrderManagementService. Simply pass an array of item IDs to add to the order. 
Common Selection 
If you are adding one or more items to an order at the same time, and their selections are the same, you can choose a common selection for the CreateOrder, AddOrderItems, CreateAndSubmitOrder, and UpdateOrderItems operations that will be applied to all order items passed in without an option selection. This reduces the amount of data transmitted to the CMR and is less resource-intensive because the CMR will need to validate the common selection only once. 
For each item in the array, you should set the following parameters:

Parameters on a Catalog Item

The other attributes of Order Item should be empty when creating an order; the CMR will populate them automatically. The CMR will process the request and return a GUID identifying the newly created order. The code listing below shows an example of creating a simple order. 

Code Listing 57: Creating a Simple Order

Adding Order Items 
Once you have created an order, you may add other items to it using the AddOrderItems operation. This operation uses the GUID of the target order and a list of items to add. As with CreateOrder, the options other than those listed in the table above should be empty; the CMR will process the additions and return a list of GUIDs identifying these additions. The code listing below shows an example of adding items to an existing order. 

Code Listing 58: Adding Items to an Existing Order

Updating Order Items
Once you have added an item to an order, you can modify the options or quantity by calling UpdateOrderItems and passing in the modified items. In this case, you must fill in the GUID of each item since the CMR needs to findthe original item in your order to update it. To ensure you are updating the appropriate item with the correct information, follow these steps:

• Call GetOrderItems to populate the data.
• Modify the desired order items.
• Pass the results to UpdateOrderItems.

The code listing below shows an example of updating an existing item.

Code Listing 59: Updating an Item in an Order

Removing Order Items 

You may remove items from an order using the RemoveOrderItems operation as shown below: 

Code Listing 60: Removing an Item from an Order

Removing Orders and Canceling Orders

To delete a specific provider order(s), use the RemoveProviderOrders operation. You cannot delete orders that you have already submitted. For post-submittal orders, a registered user can use the CancelOrder operation on the OrderManagementService to cancel an open order. 
Guest orders cannot be deleted in the post-submittal phase; the CancelOrder operation will delete the entire order, including all associated orders.

Setting User Information for an Order

Every order must have its own user-specific information attached to it so that a data provider can process the order. Contact information, billing, and shipping addresses are all required as well as user domain and region. Each order is independent of other orders, so user information must be set up for each order created. 
You may set up user information at any point in the order process prior to submission. User information is not required until you validate, quote, or submit the order. If the required information is not present at that time, you will receive an error. You can set up user information for an order by calling the SetUserInformationForOrder operation on the OrderManagementService.

Validating an Order

To ensure that an order is complete, with all the required options and user-associated information for that order filled out, the CMR validates all orders when submitted or quoted. If you attempt to submit an invalid order, you will receive an error message. Currently, the message reflects only the first validation error encountered; you need to fix the reported error and revalidate until the order is error-free. At this point, if you change any aspect of the validated order before finally submitting it, then the order will again require validation. To validate orders, call SubmitOrder or QuoteOrder. You can manually validate orders with the ValidateOrder operation in the 
OrderManagementService.

Keep in mind that running the ValidateOrder operation may return a validation error related to one of the many provider-specific validation rules, which may change over time. You should find the error messages complete enough to help you determine what aspects of the order you need to correct. 

Note that validation errors can occur if the order specified does not exist, or does not belong to you.

Requesting a Quote for an Order

Requesting a quote is an optional step that not all providers support. A quoted order provides a binding price for the order as a whole. The binding quote response from each of the necessary providers may take a day or more to receive. You cannot make changes to an order while it is in the quoting process. If you make changes to the order subsequent to the receipt of a quote, the changes may invalidate the quoted price. You may request a quote from a provider by calling QuoteOrder.

Submitting a Data Request

Once you have fully validated a data request and have made no subsequent changes to the request, submit the request to the system using the SubmitOrder operation in the OrderManagementService. 
Specify your preference for receiving order status information using one of the NotificationLevel choices shown in Table 10: CMR Order Notification Levels

CMR Order Notification Levels

If you do not specify any notification level for a registered user's order, the CMR will use your preference value. If you, as a registered user, have not set a preference value, the default value is INFO. The notification for all guests' orders is also INFO. 
Once an order transmits to the system using the SubmitOrder operation, the CMR sends each provider order contained within your order to the appropriate provider. Each provider will decide whether to accept your provider order and how to process it. 

Tracking and Canceling Orders 
From this point on, you can track the status of an order using the GetOrders operation in the OrderManagementService. At any point before the order actually ships, you may request cancellation of the order through the CancelOrder operation. You can then use the GetOrders operation again to track whether you successfully cancelled the order. 
When the CMR sends a request to a provider to quote, submit or cancel a particular order, both the CMR and that provider must have the same ID for that order. The CMR identifies the specific provider order by the ProviderOrderGUID (which is the combination of the provider GUID and the order GUID). However, the provider may also have its own ID system for tracking a particular provider order. As a solution, the provider order in the CMR also contains a provider-tracking ID. When the provider receives the request from the CMR, the provider has the option to accept the order GUID that the CMR has assigned to that provider order, or to provide the CMR with its own unique ID. If the provider does pass back its own unique ID as the provider-tracking ID, then the CMR will use that ID in addition to the normal CMR ProviderOrderGUID to refer to this provider order. If the provider does not specify a tracking ID, then the provider order will be referenced using only the CMR ProviderOrderGUID.

Order States

An order consists of zero or many provider orders. An order state is calculated based on the state of the individual provider orders that make up the order. The table below shows the order states for orders made up of zero or one provider order or of multiple provider orders having the same provider order state. 

Order States with 0 or 1 Provider Order

Restricted or Deleted Order Items 
Some catalog items you may order only with permission. Each provider sets the restrictions and permissions on catalog item ordering. Providers may also delete catalog items at any time. Restrictions can apply to individual users, groups of users or all users. If a catalog item is not available to you, executing the 
CreateAndSubmitOrder, GetCatalogItemOrderInformation, CreateOrder, or AddOrderItems operation with this catalog item will return an error message indicating that it is not an orderable item. If an item becomes unavailable after you have created an order, you will receive an error message when you invoke the 
UpdateOrderItems, ValidateOrder, QuoteOrder, or SubmitOrder operation on that order.