Chapter 1: Introduction and Scope

The NASA-developed Earth Observing System (EOS) Common Metadata Repository (CMR) is a spatial and temporal metadata registry that stores metadata from a variety of science disciplines and domains —including Climate Variability and Change, Carbon Cycle and Ecosystems, Earth Surface and Interior, Atmospheric Composition, Weather, and Water and Energy Cycles.   The CMR is intended to enable broader use of NASA's EOS data by providing a more uniform view of NASA’s substantial and diverse data holdings.  Its two primary objectives are to: 1) help science communities in need of data from multiple organizations and disciplines more efficiently use search functions and access data and services; and 2) increase the potential for interoperability with new tools and services.  As the potential to exchange and inter-operate increases, the value of these resources increases comparably.

The CMR was designed to accomplish these goals by providing a system with an Application Programming Interface (API).  While the API facilitates the discovery, online access, and delivery of a Data Partner's data holdings; it is the responsibility of the CMR Data Partners to add new metadata, remove old metadata, and modify and control access to existing metadata.  As such, Data Partners retain complete control over what metadata are represented in the CMR at any given time. Client Partners develop client applications that access the CMR API and take advantage of the services made available. These clients, such as Earthdata Search (https://search.earthdata.nasa.gov), CMR open search (https://apicmr.echoearthdata.nasa.gov/opensearch), Reverb (https://reverb.echo.nasa.gov), etc.  etc. allow end users to discover data which has been registered in the CMR's holdings; and can be custom made to meet the needs of a general user audience or a specific science application.

It should be noted that the CMR is a continuously evolving metadata system that merges all existing capabilities and metadata from EOS Clearing House (ECHO) Data Partners and the Global Change Master Directory (GCMD) systems. The CMR elements include all system components, which consist of: CMR itself (formerly the [ECHO]), GCMD, International Data Network (IDN), Earth Science Data and Information System (ESDIS) Metrics System (EMS), all related tools (internal and external), and all Metadata – including the Unified Metadata Model (UMM) concepts, the GCMD Keywords Controlled Vocabulary, and other controlled vocabularies. Thus, this is a living document.  As the CMR matures, updated instructions will be incorporated in later revisions.

NASA's Earth science data has already proven essential to understanding Earth as an integrated system, and other organizations are also providing their Earth science metadata to the CMR for users to search and access.  By simplifying discoverability and accessibility to the CMR’s Earth Science holdings, and fostering interoperability with new tools and services, the user community will enlarge and the pace of scientific discovery and application will accelerate.  For examples of how NASA's Earth science data is helping scientists understand the complexities of our Earth, visit Sensing our Planet and Other Featured Research Articles at https://earthdata.nasa.gov/.

The CMR Concept and Design

NASA's Earth Science Data and Information System (ESDIS) built the CMR based on Extensible Markup Language (XML) and Web Service technologies. The CMR interfaces with clients and users through various APIs. The CMR is an open system with published APIs available to the CMR Development and User community. 

Internally, the CMR specifies APIs and provides middleware components in a layered architecture - including data and service search and access functions. The figure below depicts the CMR system context in relation to its public APIs.

CMR System Concept

Key features of the CMR architecture are:

• Ease of Partner Participation – Designed to be low-cost and minimally intrusive, the CMR offers a set of standard ways for Partners to interface with the system through provided web UIs and a metadata exchange approach that accommodates existing partners and technology.
• Open System / Published APIs – – To accommodate independent CMR clients, CMR uses an open system approach and publishes domain APIs. These APIs are independent of the underlying transport protocols used. CMR communicates using WS-I Basic Profile v1.0 compliant web services for legacy services and RESTful web services for CMR ingest, search, and metadata management.

• Evolutionary Development – The CMR system is being developed incrementally to allow for insight and feedback during the development cycle. Industry trends are followed and the use of commercial, off-the-shelf (COTS) products is optimized.

Security

The CMR system requires Secure Sockets Layer (SSL)-based communication from Client Applications to the CMR API, but does not require, secure communication from CMR to a Client Partner's service. Internally, the CMR system is protected through a layer of software and hardware control mechanisms to preserve the integrity of CMR's holdings. When configuring data access fulfillment, Client Partners are strongly encouraged to utilize SSL communications.

CMR Capability And Functionality

CMR Benefits to Client Partners

The CMR's open system provides earth science data and services to a large, diverse pool of users, enabling scientific community interaction and collaboration. The Client Partners are benefited in the following ways:

Client Partner Skills

Since the CMR uses platform-independent web service definitions for its API, there are no requirements for a client programming language. All examples in this document use curl; however, the code samples provided could be translated to any web service capable language. 

As a CMR Client Partner, you need to be familiar with basic software development and Service Oriented Architecture (SOA) concepts such as:

• XML and XML Schema (XSD)
• Client/Server-based programming (client stubs, remote endpoints, etc.)
• RESTful client and service communication programming
• Service-based Application Programmer's Interface (API)

Client Partner Tasks

As a Client Partner who is beginning to integrate with CMR, you should expect to perform the following tasks,  which are detailed in later sections: 

• Creating and managing user accounts and user access
• Searching for data
• Retrieving data
• Accessing data

CMR System Environments

Three CMR systems are accessible by Client Partners: CMR Operations, CMR UAT and CMR SIT.  Each of these systems is briefly described below.  For additional information, click on the associated link.

• CMR Operations - The CMR Operations system environment is a publically publicly accessible server that houses the production environment.  The Data Holdings within this system include Earth Science data that has been made available to the Earth Science Community by the CMR Data Partners.  This environment is monitored 24/7, updated with enhancements and fixes on a monthly cycle, and experiences virtually no down time.
• CMR UAT (User Acceptance Test) - The UAT environment provides a stable test system to serve the needs of the CMR Data, Client, and Service Partners.The Data Holdings within this system consist of whatever the CMR's Data Partners have made available for their own testing purposes. Any enhancements and fixes that are planned for the Operations Environment are installed in this environment two weeks prior to operations delivery. CMR Partners are encouraged to verify the capabilities when a new release is installed.
• CMR SIT (System Integration Test) The SIT system was established in order to facilitate an exchange of ideas and provide an initial testing ground for upcoming capabilities. There is often very little metadata available in this test environment, but it is fully functional. The next operational version is released into this system approximately 1 month before its schedule Operational release date.  CMR SIT is a closed, private environment only available to CMR Developers.
  

Chapter 2: Getting Started

Creating and Managing User Accounts and Access

This chapter will discuss:

• Creating and managing user accounts
• CMR session management - creating, using, and deleting tokens to provide authorization

*Note:  If searching and retrieving publicly available data is the only desired operation, proceed straight to Chapter 3.

User Accounts

User accounts are employed to enable access to restricted data, manage privileges, and/or to interact with other services and tools provided by the CMR/ECHO. User accounts for the CMR system are created and managed by the Earthdata Login (URS) system. If you need to create an account,  click on Earthdata Login and follow the instructions provided. Once created, you can always return to Earthdata Login to manage your account. If you are part of a Data Provider group or other team, the team administrator can set up permissions for you to access their restricted data. Special privilege requests can be made by contacting the CMR operational team at support@earthdata.nasa.gov. 

Creating and Managing CMR Sessions

Tokens are used by the CMR to validate both the requester and their privileges for each request message (i.e., the http call to CMR) submitted.  For most searches, a token is not needed because the metadata records are unrestricted and accessible by anyone. However, when specific metadata records are restricted, privileged users require a token to see and access those records. 

The same token can be used for multiple requests before being deleted.  A series of requests that use the same token is referred to as a "session."  All tokens expire at the end of a predefined time period - which is currently fixed at 30 days. Because the token is used to track your session, it must be protected by client applications with the same level of security used for your login name and password.

A normal session is conducted via the following steps:

• Create a token
• Perform one or both of the following tasks in any order:
  • Search for records
  • Retrieve records
• Delete the token 

Creating a Token

Information for requesting a Launchpad Token can be found here. More information on getting access to Launchpad tokens can be found on this guide. 

Once a token is created, you can search and retrieve records, as well as conduct other functionality (described in later chapters) through the CMR or ECHO APIs.  When the token is no longer needed, it can be deleted.

Deleting the Token

A Launchpad token will automatically expire after one hour.

Chapter 3: Searching for metadata

Currently clients can search the CMR for collection and granule metadata.  In the future, clients will also be able to search for metadata describing services, visualizations, parameters (variables), and documents.

CMR Environment URLs

The CMR system has three environments:

The Systems Integration Test (SIT) Environment provides Environment provides the CMR development team with a safe place to test new functionality. The newest, untested software and upgrades are are first uploaded here; thus, this environment is the least stable. Once the CMR software has been sufficiently tested, it gets deployed to the User Acceptance Test (UAT) Environment.
  Note that the CMR SIT environment is closed to the public and is intended for internal CMR development only.

The UAT Environment provides a stable environment for The UAT Environment provides a stable environment for testing system software.  The UAT environment is where Open Source developers and Data Partners can deploy new code and/or new metadata for testing before it becomes visible in Operations.  Following a few weeks of successfully testing, the software is deployed to the Operational Environment.

 *Note: Client Partners can test their software in either the SIT or UAT environment depending the level of integration and testingbefore moving that software to Operations.

The Operational Environment is the live system available to users around the world.  The Operational Environment is also known as Production, Operations, or OPS.

*Note:  All of the examples provided in the rest of the document are using the Systems Integration User Acceptance Test environment.  To run the commands in the other environmentsOperational Environment, simply replace the SIT URL with either the UAT or OPS URLreplace all instances of cmr.uat.earthdata.nasa.gov with cmr.earthdata.nasa.gov.

CMR Environments

Headers

Headers are a part of HTTP requests and for the CMR they provide information such as message content content (content type), the format of the data that gets returned (accept), and tokens to allow increased privileges (Echo-EDL Token), etc.  

Content Type

Content-Type is a standard HTTP header that specifies the content type of the body of the request for POST method messages. Search and retrieval requests support the following Content-Types.               

*Note: If the Content-Type is not specified, XML is assumed.

Content-Type headers

Echo-Token

Authorization: Bearer

The Authorization: Bearer header allows a user to specify an EDL token so that CMR knows The Echo-Token allows the CMR to know who is making a request. A token must first be generated as described in the Creating a Token section. The Token is in the format of XXXXXXXX-XXXXOnce the requester has the token, the token can be placed into the http header for the necessary API calls.

Authorization:

The Authorization: header allows a user to specify a Launchpad token so that CMR knows who is making a request. A -XXXX-XXXX-XXXXXXXXXXXX.  A token must first be generated as described in the previous To Create a Token section. Once the requester has the token, the token can be placed into the http header for the necessary API calls.

Accept Headers

Accept Headers are used in cases where the caller wants to control the format and/or specification by which the data gets returned.  The following table lists the valid values.  If this header or an alternative method is not used, XML results will be returned by default.

Accept Headers

For more information about the accepted   accepted header values, please see the CMR API documentation.

Client-Id

Client-Id is an additional header that allows the client to specify their name. Client Partners are strongly encouraged to use this header for the following reasons:

• Helps the CMR operations team monitor query performance per client
• Aids the CMR operations team in identifying clients who are attempting to contact them for assistance with a request.
• Facilitates NASA in collecting information on how much traffic flows through a client provider and what kind of data interests their users.

Below are some examples depicting how to use headers.  The purple part of the example will be explained in this section, while the rest will be addressed later in the document.

As an alternate to using the Accept Header,  the client can use the Type Received name in the query to get the same results. Instead of using "Accept: application/opendata+json", "opendata" can be used at the end of the main query before parameters are specified. 

Other examples use the DIF 10 specification and the ISO specification respectively.

To change the type of request, simply replace the header with the desired information from the table above.

Results

Listed below are the 3 ways results can be returned, as well as the formats supported by each option:

• A reference list of results - XML
• A list of results with partial metadata records - XML, JSON, ATOM, KLM, and CSV (*Note: CSV is supported for granules only)
• A list of results with full metadata records - XML and opendata

The following is an example of a reference list of results

The results specify:

• The number of metadata records found by the "hits" tag
• The duration of the query in milliseconds
• A list of metadata record results specified by the "reference" tag

Within each reference tag, a limited amount of information about the metadata is provided including:

• The metadata name
• The CMR profile or concept ID -  a CMR generated unique ID. The ID is encoded by a letter of the profile or concept (C for collection, G for granule, S for service), followed by a CMR generated number, followed by a "-" and then followed by the ID of the metadata provider.   <letter> <unique-number> "-" <provider-id>
• The exact CMR location from which the metadata can be downloaded
• The latest revision number of the metadata record.

The following is an example of a full metadata record list of results in the ECHO 10 specification using the XML format.

The results specify:

• The number of metadata records found by the "hits" tag
• The duration of the query in milliseconds
• A list of metadata record results specified by the "results" tag

Within each result tag, prior to the full metadata record, the following three attributes are displayed: 

• The CMR concept id (profile ID)
• The specification and format of the metadata
• The revision number of the shown metadata record.  

For detailed information about the result specifications, format options, and examples -  please see the CMR API documentation.

Searching

There are several ways to search the CMR system using the RESTful principles,  which include employing:

• The API calls and parameters with the GET method
• The API calls and parameters with the POST method
• A JSON query language with a POST method
• The Alternative Query Language (AQL)

The most popular and preferred method is to use the API calls and parameters with the GET or POST methods. For more detailed information, see the API documentation located at https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html. 

*Note: The CMR URL character limit is currently set to take roughly 500k characters. Clients using the GET Search API with query parameters should be careful not to exceed this limit or an HTTP response of 413 FULL HEAD will be returned. Clients who expect that the query URL could exceed 500k characters should use the POST method as opposed to the GET method for searching.

API calls and parameters GET method

The following examples demonstrates the basic search command:

The query returns the first 10 publicly available collection results in a reference list using  XML format.  (Note:  As described in the header section above, in order to see restricted data, you must have the correct privileges and will need to use a token).

The search parameters listed below can be applied to provide more functionality:  

The following search parameters are for collection requests only:

• include_has_granules - Includes a has-granules tag or attribute in the response so the client knows if the collection encompasses any granules. E.g. include_has_granules=true
• include_granule_counts - Includes a granule-counts tag or attribute in the response with the number of granules represented by the collection. E.g. include_granule_counts=true
• include_facets - Includes a list of facets and their counts at the end of the results. This is mainly used for collection search displays. E.g. include_facets=true
• include_facets with hierarchical_facets - Includes a list of facets preserving the hierarchical order. This is mainly used for collection search displays. E.g include_facets=true&hierarchical_facets=true

The next box depicts a set of examples conducting a basic search using the search parameters described below.

• The first line requests 50 metadata references per page.
• The second line requests 50 metadata references per page using the formatted print.  
• The third line requests page 2 of 20 results per page showing full records in the echo 10 specification using formatted print.
• The fourth line issues a request including token and client id headers and does a basic search sorting the results using the platform element.  

The request returns page 2 results of 20 results per page showing full records in the ISO 19115 specification using the XML format in a formatted fashion.

*Note: The "?" character separates the URL from the search parameters and the search parameters are separated from each other by the "&" character. The parameters can be in any order.

The previous examples all demonstrate collections searches. The same search parameters apply to granules, unless it is explicitly stated that a parameter applies only to collections. To conduct a granule search, simply replace collections with granules in the URL.  In the box below are the same four search requests that are listed in the box above,  only granules has been substituted for collections.

The search can be refined using the set of search parameters documented in the table below. These parameters support collection searches. Most of these parameters have the brackets next to them and may need to be escaped (\[\]) depending on the language used or the method by which the query is being sent.  All of CMR time search parameters (temporal, updated_since, revision_date, and equator_crossing_date) formats are specified as yyyy-MM-ddTHH:mm:ss.SSSZ format;  Where yyyy is year; MM is month; dd is day; T is the date time separator character; HH is the hour; mm is the minute; ss is the second; SSS is the milliseconds (the .SSS can be omitted); and Z specifies Zulu time.

Documented in the table below are collection supported search parameters:

*Note:  To get the complete and most up to date set of parameters,  visit https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html.

Collection Search Parameters

Documented in the table below are granule supported search parameters.

Granule Search Parameters

The following example demonstrates a request to find collection metadata records that contain an AQUA platform granule.  It further requests that only a formatted reference list of results that contains 20 references be displayed.

The following example demonstrates a request to find collection metadata records that contain an AQUA or an AURA platform granule.  It further requests that only a formatted reference list of results that contains 20 references be displayed.

Certain search parameters have additional options to aid the user.   The syntax to employ them is: options[parameter name][option_key]=value.

• Parameter name is the search parameter to be affected (ex: platform).
• Value is set to "true" or "false".  
• Option_key is one of the following:

Extra Options

The following example demonstrates the use of additional options in conjunction with the platform search parameter.  A user would make the request in the box below to find any platform that begins with "A" followed by any number of alphanumeric characters and ends with "A".   AQUA and AURA would both be located and returned with the results.

This example in the box below depicts a request entered to search for records exclusively contains an instrument matching "HELLO" in uppercase letters.

Besides science_keywords, if any of the parameters that are searched are repeated, the metadata records that have ANY of the values will match.  The following example demonstrates that the CMR system will match any metadata record containing either concept id.

For a complete set of examples using all of the search parameters, visit the API documentation link at: https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html

API calls and parameters POST method

The API using the POST method is the same as with the GET method, with the exception of: 1) The method name used is POST instead of GET;  and 2) the parameters are in the body of the message without a length constraint as opposed to existing in the URL string.  The following example uses the curl command and shows the query.xml file that contains the query request.  The subsequent example depicts the curl search request. As long as syntax remains the same in the query.xml file, the parameters can be one long set or formatted to be more easily read. 

*Note: Notice the brackets ([ ]) were not escaped in this example.

query.xml:

*Note: In this specific instance, the Content-Type header is not used.  This is deliberate as it will cause an error.  

JSON query language with a POST method

The CMR provides a JSON RESTful interface. The elements that can be searched are the same as described above, but this interface is applicable only to collection searches. This search method provides additional functionality in conducting a search by using conditions (AND, OR, NOT) against the elements.  See the JSON schema https://cmr.situat.earthdata.nasa.gov/search/site/JSONQueryLanguage.json for more details.  The example provided below demonstrates a query consisting of conditions and several elements.  

curl -XPOST -H "Content-Type: application/json" -H "Client-Id: GCMD" https://cmr.situat.earthdata.nasa.gov/search/collections
-d '{"condition": { "and": [{ "not": { "or": [{ "provider": "TEST" },
{ "and": [{ "project": "test-project",
"platform": "mars-satellite" }]}]}},
{ "bounding_box": [-45,15,0,25],
"science_keywords": { "category": "EARTH SCIENCE" }}]}}'

Alternative Query Language (AQL)

The CMR supports the ECHO Alternative Query Language (AQL) and is available to clients.  However,  while the AQL is supported, it will not mature or be integrated with new CMR features. For a detailed explanation and examples of AQL, refer to the ECHO AQL documentation.  

Chapter 4: Retrieving Metadata

There are several ways of retrieving metadata:

• Retrieve a result list consisting of full metadata records. While this has been demonstrated above, a example has been replicated here for convenience.

  Tip
  title Example
  curl -v -i "https://cmr.situat.earthdata.nasa.gov/search/collections.native?pretty=true" curl -v -i "https://cmr.situat.earthdata.nasa.gov/search/granules.native?pretty=true"

  
  

• Use the concept id to retrieve a record: The syntax is https://cmr.situat.earthdata.nasa.gov/search/concepts/<concept id>. 
  The concept id/revision number if a specific revision of a metadata record is wanted: The syntax is https://cmr.situat.earthdata.nasa.gov/search/concepts/<concept id>/<revision number>.

  Tip
  title Example
  curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/C1000000803-DEV08" curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/C1000000803-DEV08/7" curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/G23447-ASF/8

  
  

• The CMR supports retrieving metadata records using different specifications and formats, which are listed in the table below:

Supported Standards

Below are several examples using the supported standards mime types:

• Ex 1: Retrieves a granule metadata record in the JSON format. 
  
• Ex 2: Retrieves a granule metadata record with a revision of 8 in the ISO specification.  
• Ex 3: Retrieves a collection metadata record with a revision of 7 in the DIF 10 specification. (*Note: If the record was sent and stored using the ECHO10 specification, that record will be translated to the DIF 10 specification and returned to the caller.)     

• Ex 4: Lists a granule record using the native option with the pretty print option turned on. The native option lists the metadata in the specification it was sent and stored within the CMR.

  Tip
  title Example
  curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/G23447-ASF.json"  curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/G23447-ASF/8.iso" curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/C1000000803-DEV08/7.dif10" curl -v "https://cmr.situat.earthdata.nasa.gov/search/concepts/G23447-ASF.native?pretty=true"

  
  

Chapter 5: Accessing data

There are a variety of ways to access the data of interest from the response of a search query:

1. Access the data provider's site via the landing page in the collection or granule and follow their instructions for data retrieval. Note: This information is not required and may not be present.
    
2.  Order the data through the ECHO system. A detailed explanation of this process is located in the ECHO Client Partners Users Guide Chapter 6: Ordering data through ECHO. Earthdata Search system, found at https://search.earthdata.nasa.gov.