Preface

This document provides a look and guide to the Common Metadata Repository (CMR) from the perspective of the Client Partner.

Conventions

• All references to time are in Universal Time Coordinated (UTC).
• Data Partners are also referred to as Data Providers.
• Client Partners are also referred to as Client Developers.
• Words in bold text are key words or concepts. 
  
  

• Programming examples use a fixed width font, have upper/lower lines separating them 

  from the rest of the text, and are in this color font. 

   

• Comments (denoted by // within examples) 

• Best practices or warnings appear in italicized, boxed text.

Before You Begin

The primary reason for designing the CMR was to increase access to Earth Science data and services by providing a system with a machine-to-machine interface, that is, an Application Programming Interface (API). The CMR functions as a metadata clearinghouse of Earth Science metadata for a wide variety of partners, enabling the science community to exchange information. Data Partners provide the Earth Science community with metadata representing their Earth Science data holdings. CMR technology in turn provides services for Client Partners and Data Partners and supports efficient discovery and access to Earth science data. 
The CMR also functions as an order broker for the data, and offers services applied to that data. The CMR provides a portal on the internet where CMR clients can search the metadata for information they wish to order.   The CMR also functions as an internet portal where CMR clients can search the metadata for information they wish to retrieve.
Client applications can access data holdings via order distribution or online access. Data Partners retain complete control over what metadata are represented in the CMR including inserting new metadata, modifying existing metadata and removing old metadata, and controlling access to their metadata.

Tasks That You Will Perform as a Client Partner

Usually performed in the order shown below:

In Chapter 4

• Logging in and getting started
  • Creating and managing CMR sessions
  • Creating and managing user accounts

In Chapters 5, 6 and 8

• Querying for Earth Science Data
  • Formatting query results
  • Handling large result sets
  • Using subscriptions to automate queries
    • Creating and deleting subscriptions
    • Listing subscriptions

In Chapter 7

• Ordering Retrieving data through CMR
  • Adding order items
  • Updating order items
  • Removing order items
  • Removing orders
  • Setting user information for an order
  • Validating orders
  • Requesting a quote for an order (LBN: I thought this was free?)
  • Submitting an order
  • Tracking or canceling an order

In Chapter 8

• Using Event Notification Service
  • Setting delivery modes
  • Filtering events
  • Creating event subscriptions
  • Removing event subscriptions
  • Renewing event subscriptions

Skills You Will Need as a Client Partner

Since the CMR uses platform-independent web service definitions for its API, there are no requirements for a client programming language. All examples in this document are in snippets of Java code; however, the code samples provided could be translated to any web service capable language. 
As a CMR Data Partner, you need to be familiar with basic software development and Service Oriented Architecture (SOA) concepts such as:

• XML and XML Schema (XSD)
• Client/Server-based programming (client stubs, remote endpoints, etc.)

As a REST-API user, you will need:

• A basic understanding of the REST concept
• HTTP

As a SOAP-API user, you will need:

• Web Service Definition Language (WSDL)
• Service-based Application Programmer's Interface (API)

CMR Concept and Design

NASA's Earth Science Data and Information System (ESDIS) has built the CMR based on Extensible Markup Language (XML) and Web Service technologies. The CMR interfaces with clients and users through its series of Application Program Interfaces (APIs). The CMR is an open system with published APIs available to the CMR Development and User community. 
As the CMR is a middleware application, interacting with it means interacting with the CMR API. There is typically a user-focused client application interacting with CMR's API on behalf of an end user. This client may be a generic, query and order-based client, or may be specific to an end user's research, mission, or general area of interest. The CMR incorporates a Universal Description, Discovery, and Integration (UDDI) registry to facilitate registration, discovery, and invocation of services related to the CMR holdings. 
Internally, the CMR specifies APIs and provides middleware components, including data and service search and access functions, in a layered architecture. The figure below depicts the CMR system context in relation to its public APIs.

All CMR metadata are stored in an Oracle database with spatial extensions. The metadata model is derived primarily from that used by the Earth Observing System Data and Information System (EOSDIS) Core System (ECS). For more details about the CMR model, refer to the Earth Science Metadata Model chapter of the Data Partner Guide
Key features of the CMR architecture are:

• Ease of Partner Participation
  Designed to be low-cost and minimally intrusive, the CMR offers a set of standard ways for partners to interface with the system and a metadata exchange approach that accommodates existing partners and technology.
• Data Model Consistency
  To mitigate the risk of being unable to match all possible partner data models, the CMR has prototyped a Metadata Mapping Tool to translate non-standard formats upon ingest into the CMR.
• Open System/Published APIs
  To accommodate independent clients, the CMR uses an open system approach and publishes domain APIs. These APIs are independent of the underlying transport protocols used. The CMR communicates using WS-I Basic Profile v1.0 compliant web services. This API is located at  api.cmr.nasa.gov/cmr. Interactions with the CMR may involve user interactions in real time or may be machine to machine.
• Evolutionary Development
  The CMR system is being developed incrementally to allow for insight and feedback during the development cycle. Industry trends are followed and the use of commercial, off-the-shelf (COTS) products is optimized.
• Extensibility of Client User Interfaces and Capabilities
  CMR extensibility is ensured by its component architecture, which allows new capabilities and functions to be plugged in, modeling relationships between services/APIs/UIs, and continued prototyping. the CMR's current focus is on the middleware and on enabling many different types of user interfaces via its APIs.

CMR as a Spatially Enabled Metadata Search and Order System

Oracle enables the the CMR system to interact with spatially enabled Earth science metadata by use of spatial extensions into the system and business logic within the system that understands how to interact with that metadata. In addition, a second CMR interface (Ingest) allows metadata updates to go directly into the database, bypassing the message-passing API. The File Transfer Protocol (FTP) server is configured to receive these update files, which are expressed in XML conforming to three schemas, one for granules (or inventory), one for collections (or datasets), and one for browse. 
Note: For ECHO 10.0 and later, these formats are schemas; for legacy ECHO, these formats are DTDs.

The schemas are defined on the ECHO 10.10 Ingest DTD/Schemas page of the ECHO website: http://api.echo.nasa.gov/echo/apis.html
Oracle's spatial capabilities support queries for CMR metadata whose spatial extents are described within the system. A Data Partner can define the spatial extent of a granule or a collection with different spatial constructs (for example: point and polygon). A Client Partner can then construct a search using a point, a line, or a polygon (or multiple polygon) spatial type, and the CMR responds with data whose spatial regions intersect the described region. The CMR provides services for interacting with its Catalog of metadata. Queries can be performed in a number of ways; result formats can be specified, and the resulting data sets can be incrementally accessed so that large return sets can be handled gracefully. The CMR also supports constructing, submitting, and tracking orders for the data that the metadata represents. The CMR supports both an embedding of a Uniform Resource Locator (URL) within the metadata for accessing the data (which the client simply accesses via Hypertext Transfer Protocol [HTTP]), and a more complicated order process in which quotes and order options are accommodated. 
The CMR incorporates the ECS concept of granules and collections and defines separate DTDs for updating each, under the assumption that granules will indicate which collection is considered their ―primary‖ collection. ―Primary collection‖ "primary collection."  "Primary Collection" means the collection that owns the granule. 
A collection is a grouping of granules that all come from the same source, such as a modeling group or institution. Collections have information that is common across all the granules they "own" and a template for describing additional attributes not already part of the metadata model. 
A granule is the smallest aggregation of data that can be independently managed (described, inventoried, and retrieved). Granules have their own metadata model and support values associated with the additional attributes defined by the owning collection. 
A third type of metadata is browse metadata, which provide a high-level view of granule or collection metadata and cross-referencing to other granules or collections.  Browse metadata are not spatially enabled but are still useful.

Security

The CMR system supports Secure Sockets Layer (SSL)-based communication, which a client must use to pass passwords or other sensitive information securely. Internally, the systems are firewalled to prevent unintended access.

Supported Platforms

The CMR system supports clients capable of initiating an HTTP connection from a variety of programming languages

CMR Capability and Functionality

The CMR provides an infrastructure that allows various communities to share tools, services, and metadata. As a metadata clearinghouse, it supports many data access paradigms such as navigation and discovery. As an order broker, the CMR forwards orders for data discovered through the metadata query process to the appropriate Data Partners for order fulfillment. As a service broker, the CMR decentralizes end user functionality and supports interoperability of distributed functions. 
Although this Guide focuses on the needs of Client Partners, the CMR supports the following different, nonexclusive types of Partners:

• Data Partners: Organizations that supply metadata representing their data holdings to the CMR database.
• Client Partners: Organizations that participate by developing software applications to access the Earth science metadata in the CMR database.
• Service Partners: Organizations that participate by advertising their Earth science-related services to the user community via the CMR, which maintains service descriptions in a Service Catalog (either special services, or services that are available as an option on a selected set of granules/collections) and support the user in ordering those services.
• Extended Service Partners: Organizations that participate by providing a central location for registration, classification, and maintenance of Earth science services, interfaces, GUIs, and advertisements. The CMR addresses science user needs through a set of well-defined and open interfaces upon which the user community can build its own client applications. In this way, the CMR supports extendable, flexible user interfaces, allowing industry and the science community to drive the progress of available Earth science applications. For more complete information about client applications, refer to the companion piece to this Guide, the ECHO 10.10
• Data Partner's Guide: The CMR approach allows users to build their own user interfaces to the CMR, rather than being limited to the data search and order system provided by NASA. For Data Partners, the CMR offloads the burden of providing the system resources required for searching and gives users the flexibility to support community-specific services and functionality. the CMR's interoperability features allow all participants to benefit from the distributed development of functions, again reducing dependence on NASA resources.

CMR Benefits to Client Partners

To address the CMR system vision, the CMR has responded to a set of system drivers, that is, reasons for upgrading. These drivers, derived from functional, organizational, and operational concerns expressed by the user community, determined the architectural approach and the types of technical solutions used in building the CMR system.

Ease of Participation

The primary goal of the CMR is to enable organizations to participate in making their resources and capabilities available to the Earth Science community. To facilitate participation by these organizations, the CMR has:

• Minimized the number of requirements that a partner must meet to participate.
• Involved partners in the system's development cycle and requirements definition.
• Selected metadata insert and update mechanisms based on current standard industry practice (for example, XML) that most databases can generate automatically.
• Provided mapping capabilities to convert from one XML representation into another.

Cost to Field

While aggressive in the capabilities it is targeted to support, the CMR minimizes the Cost to Field by continually evaluating performance and functionality against costs, for example, licensing of Commercial Off-the-Shelf (COTS) applications, amount of custom code required, hardware platform requirements, and complexity of networking and installation.

Cost to Operate

Once fielded, the CMR seeks to minimize the cost to operate the system by making it easier to use, thereby minimizing the load on operations staff.

Extensibility

The CMR is being built with long-term extensibility foremost in mind. To enable emerging techniques and strategies for Earth Science research, the CMR has:

• Adopted a 'design for change' as a goal at the beginning of the CMR development.
• Built in the capability to limit the impact of changes to the API on the configuration file, the service interface, and the business logic that implement the function.
• Developed test tools to regression test large portions of functionality automatically overnight, so that when changes are made, undesirable impacts can be discovered quickly.
• Adopted a layered architecture to allow changes to one component of the system without affecting other components.

CMR Systems

There are three CMR Systems that you, as a Client Partner, have access to:

CMR Operations: This is the current operational system for the CMR and is available to all users.

Location: http://api.cmr.nasa.gov/cmr/index.html

CMR Partner Test: This is an operational system used only by the CMR partners where they can test their data and services prior to making the final changes in the operational system.

Location: http://api-test.cmr.nasa.gov/cmr/index.html

CMR Testbed: This is a test system area used by partners and CMR testers to test before changes to the CMR system go operational.

Location: http://testbed.cmr.nasa.gov/cmr/index.html

Logging in, setting up and getting started

This section contains information and examples that are common to most CMR client applications, such as basic Login and Logout and creation and management of user accounts.

CMRTokens & User Contexts

There are two types of users in the CMR: registered users and guests. LBN: Is this still valid or is URS now required? Registered users can have access to restricted data and services but requires the creation of a URS profile. Guest users do not require a URS profile but their access is limited in some areas.

CMR tokens are generated whenever a user interacts with a CMR API. A token is a transient representation of either your URS credentials or your guest status.

In both CMR-REST and CMR-SOAP LBN: Does CMR have a SOAP API? APIs registered access requires the creation of a CMR token and then use of that token throughout your interaction with the CMR.

In the case of CMR-REST, guest token creation is carried out behind the scenes of the API. In the case of CMR-SOAP a client must manually create their token to interact with the API.

User Accounts

Create your user credentials in URS and use them to interact with the CMR using your user name and password.

Creating and Managing CMR Sessions

When using the Web Services API, you need to request a CMR token for all but CMR-REST guest access.

This token acts as your session key and must be passed to all other CMR operations. All clients interfacing with the system are required to pass client identifier information to the CMR. The client identifier is a short description and/or name of the client. Client developers are encouraged to keep this information as concise as possible and to work with the CMR Operations Group to create an appropriate identifier.
A token is obtained when logging into the CMR (see Logging Into the CMR). This is also done when setting user and provider context.
When done working in the CMR, be sure to logoff (see Logging Out of the CMR).
NOTE: If client identifier information is not provided, submitted orders will fail.

Logging Into the CMR

The REST way

Create a token via a POST to the token resource

Creating a CMR token for a user

If you want to perform operations as a provider then you need the provider id and the credentials of a user that has provider privileges

Creating a CMR token for a provider

The CMR token will be returned to you as follows in the response body,

CMR Token response

You will use the value of CMR-TOKEN-ID to interact with the CMR REST API.

The SOAP way

To obtain a token, call the Login operation of the Authentication Service. A new token is created and returned each time Login is invoked. Code Listing 1 is an example of logging in to the CMR and creating a session token for a guest user.
Parameters:

• username - Username of the user logging in.
• password - Password of the user.
• clientInfo - The string identifier of the CMR client used to make this request.
• actAsUserName - name of the user an Admin wants to act as, null for non CMR administrator users.
• behalfOfProvider - provider the user wants to act as, null for guests and registered users with no ProviderRoles.

A Security Token is returned that is used for all subsequent calls to the CMR using the same Token profile.

Code listing 1: Logging in as Guest

Code Listing 2 is an example of logging in to the CMR and creating a session token for a guest user.

Code Listing 2: Logging In as a Registered User

Logging Out of the CMR

The REST way

To logout of your session simply delete your CMR token.

Logging out - deleting a CMR token

The SOAP way

When you are finished with a token, for example, you have finished a session with the CMR; destroy the token using the Logout operation.
Parameter(s):

• token - security token

Code Listing 3: Logging out of the CMR

You do not need to destroy a token after each call; you may reuse a token until it is destroyed with the Logout operation or the token expires. Because the token is used to track your session, it must be protected by client applications with the same level of security that you use for your login name and password.

Interacting with the CMR

The REST way

You can interact with any CMR-REST resource using guest by simply GETing, POSTing, PUTing or DELETEing the resource. Some of these operations will fail if guest does not have the authority to perform them. In cases, where you want to use a registered user you should acquire a CMR token above and attach it as a header to the request you make. For example,

Getting a list of providers with a CMR token

The SOAP way

Interacting with the rest of the CMR API follows the same pattern as logging in and logging out except that it requires that you pass a valid CMR token to each operation. The following example shows logging in, retrieving the version number of the CMR system and logging back out.

Code Listing 4: Getting the CMR Version Number

Querying for earth science metadata

The CMR uses the CMR Alternative Query Language (AQL) for its querying capabilities. (See Chapter 6 for a detailed explanation of the language syntax.) To support potentially large queries, the CMR handles a query expression without regard to the details of whether and how it will return query results to the user.
A CMR query consists of a query expression specified in XML, starting with <query>. All AQL queries must conform to the AQLQueryLanguage.dtd available on the CMR website. Also, refer to Chapter 6, The CMR Alternative Query Language for additional information about AQL.

Table 2: Query Return Result Types

Formatting Your Query Results

You can specify a subset of the information in the result set by using different parameters for the operations
ExecuteQuery and GetQueryResults.
The following elements are used to specify the format and content of a result set:

Table 3: Result Set Content Elements

Metadata results are returned as XML that conforms to one of the following DTDs:

Granule metadata conforms to the Granule Results DTD—refer to Appendix F, Results DTDs (also located at: http://api.echo.nasa.gov/echo/dtd/ECHOGranuleResults.dtd).

Collection metadata conforms to the Collection Results DTD Appendix F, Results DTDs (also located at: http://api.echo.nasa.gov/echo/dtd/ECHOCollectionResults.dtd).
Metadata attributes are made up of two values: the XML metadata attribute name and a primitive type name. The CMR currently ignores the type name. The allowable metadata attribute names are specified in the appropriate DTD (CMRGranuleResults.dtd for granules and CMRCollectionResults.dtd for collections). If you specify a metadata attribute name that has sub-attributes, all of the sub-attributes will be included as well. For example, if you specify Platform, the following elements will be included in the metadata:

• Platform
• PlatformShortName
• Instrument
• InstrumentShortName
• Sensor
• SensorShortName
• SensorCharacteristics
• SensorCharacteristicName
• SensorCharacteristicValue

OperationMode
When you specify a sub-attribute, the CMR will return the ―parent‖ attribute in the hierarchy as well as the sub-attribute. This allows you to ensure that data are correctly scoped. For example, if you specify Sensor, the following elements will be included in the metadata:

• Platform
• PlatformShortName
• Instrument
• InstrumentShortName
• GranuleUR
• GranuleURMetaData

Detailed spatial attributes cannot be used as MetadataAttributes; only their containing element may be specified. For example, you cannot use BoundingBox as a MetadataAttribute, but you can use HorizontalSpatialDomainContainer. The following spatial elements cannot be specified as MetadataAttributes:

• Point
• Circle
• BoundingRectangle
• GPolygon
• Polygon
• PointLongitude
• PointLatitude
• CenterLongitude
• CenterLatitude
• Radius
• WestBoundingCoordinate
• NorthBoundingCoordinate
• EastBoundingCoordinate
• SouthBoundingCoordinate
• Boundary
• ExclusiveZone
• SinglePolygon
• MultiPolygon
• OutRing
• InnerRing

Specifying GranuleURMetaData as a MetadataAttribute is equivalent to not specifying any MetadataAttributes; the result set includes all the elements in the result DTD.
The following code snippet shows how to execute a query for all of the metadata for matching items.

Code Listing 6: Executing a Simple Query

Given the CMR's large store of Earth Science data, it is possible for queries to return very large result sets. The CMR supports retrieving the results from a query in a number of ways. The simplest is to ask the CMR to return the results directly from the ExecuteQuery request by passing RESULTS as the ResultType. However, to prevent a single query from monopolizing CMR resources, the CMR limits the number of results available in response to a query. By default, this limit is 2,000 items. CMR Operations may change this limit depending on CMR usage patterns.
For larger results, the CMR supports a paging mechanism. This allows you to page through the available data in page sizes that you select (up to the CMR Operations configurable limit). For all ResultTypes, the CMR will create and store a result set and return the corresponding GUID. You can page through the result set using the GetQueryResults operation. The arguments to GetQueryResults are similar to ExecuteQuery with the exception that you specify the result set GUID rather than a new query.
Result sets may change after they are created. Providers are continually changing the data they have registered in the CMR. New records may appear or may be removed from a result set. Because of this, you should watch the fields Cursor and CursorAtEnd when paging through a large result set:
Cursor specifies the index of the first record to be returned in the result set. For example, a value of 5 will return results starting from the fifth record. If none is specified, it defaults to 1. If you repeat the same query later, use the same Cursor value.
Use CursorAtEnd to determine when you have reached the end of the result set. This Boolean field is TRUE if the returned results were the last available results in the result set.
The following code illustrates paging through a result set and displaying it to the user.

Code Listing 8: Paging Through Query Results

Visibility of Results

When you execute a query, the query is applied to all the data in the CMR. However, when the results are retrieved, you may not see all of the items. What you can see depends on the rules defined by the Data Partners and the privileges granted to you.

Restricted Items

If a particular item in your result set is restricted for you (i.e., you are not allowed to see it), based on your privileges, it will not be returned.

Deleted Items

It is possible that between the time you execute a query and the time you view the results some of the matched items may have been deleted from the CMR or restricted due to a request from the Data Partner who owns the metadata. In that case, the item will not be returned in your result set. For more information about notification of deleted or restricted order items, refer to section 7.8.1, Restricted or Deleted Order Items.

Querying for Orderable Data

The CMR allows you to exclude from your query data that cannot be ordered. Refer to section 1.1.1.

Searching for Orbit Data

4.4.1 Backtrack Orbit Search Algorithm
Orbit searching is by far the most accurate way to search for level 0-2 orbital swath data. Unfortunately orbital mechanics is a quite difficult field, and the most well known orbit model, the NORAD Propagator, is quite complex. The NORAD Propagator is designed to work with a wide range of possible orbits, from circular to extremely elliptical, and consequently requires quite a bit of information about the orbit to model it well.
To facilitate earth science, the orbits of satellites gathering earth science data are quite restricted compared to the variety of orbits the NORAD Propagator is designed to work with. Generally, the earth science community would like global coverage, with a constant field of view, at the same time every day. For this reason, most earth science satellites are in a sun-synchronous, near-polar orbit. Even missions that are not interested in global coverage, e.g., the Tropical Rainfall Measuring Mission (TRMM), are still interested in having a constant field of view so the coverage of the sensor is at a constant resolution. For this reason, ALL earth science satellites are in circular orbits.
The Backtrack Orbit Search Algorithm, designed and developed by Ross Swick, exploits this fact to simplify the orbit model by modeling an orbit as a great circle under which the Earth rotates. This reduces the number of orbital elements required for the model from 22 to three. Moreover, the NORAD Propagator is designed to predict future orbits based on current status, and consequently must be reinitialized periodically to correct for cumulative error as the model spins forward. As the name implies Backtrack spins the orbit backwards, and in practice spins backwards at most one orbit, so there is no cumulative error.

For more information on Backtrack, please see  http://geospatialmethods.org/bosa/.

Figure 2. Typical Orbit Path Represented on a Globe and the same Path on a Map

Backtrack orbit model

Three parameters to define an orbit:

1. Instrument swath width (in kilometers)
2. Satellite declination or inclination (in degrees)
3. Satellite period (in minutes)

Orbit data representation

Three parameters to represent orbit data:

1. Equatorial crossing longitude
2. Start circular latitude (or start latitude and start direction)
3. End circular latitude (or end latitude and end direction)

How ECHO Searches for Orbit Data

• The user specifies a regular spatial window

Figure 3. Spatial Window

• Backtrack then calculates from both ascending and descending a path for equatorial longitude crossings and start/end circular latitudes according to user's query window.

Sample queries

The following are sample queries that you can execute against the CMR. Note that the provider and the datasets used in these samples are representative only; you should modify the query to suit your needs.

Code Listing 9: Sample Collection Query (Discovery Search)

Code Listing 10: Sample Collection Query from Two Providers (Discovery Search)

Code Listing 11: Sample Collection Query with Temporal and Spatial Constraints (Discovery)

Code Listing 12: Sample Collection Query with Complex Temporal and Spatial Conditions (Discovery)

Code Listing 13: Sample Collection Query Using Provider Specific Attributes (Discovery)

The alternative query language

Accessing data

Metadata subscriptions

Appendices