System overview

Metadata

ECHO is a metadata registry. A number of both NASA and NASA-affiliated data providers export their dataset and granule metadata to the ECHO system on a continuous basis. This metadata is stored for retrieval and indexed for discovery.

Metadata is stored in an oracle database in it's native format in a compressed form.

Metadata is indexed in an elastic search cluster. All pertinent search indices are stored within elastic search with the exception of spatial extent. ECHO replaces that index with a minimum bounding rectangle and stores the actual spatial extent in our oracle database. Oracle's spatial query support provides fine-grain spatial constraint resolution, while elastic provides coarse-grained support.

Ingest

Metadata is obtained from the providers using one of two APIs.

1. A HTTP REST API - providers PUT dataset and granule documents to ECHO
2. An FTP API - providers send files containing collections of dataset and granule documents to ECHO

Access to these APIs is controlled via URS credentials for REST and a white list of ECHO-approved IP addresses for FTP.

The REST API is the preferred mechanism for ingest. All new providers are encouraged to use this over FTP.

Acquisition

Users can discovery Earth Science data via ECHO in a number of ways,

1. Reverb - our user interface web application. This is a graphical user interface that communicates directly with our REST API
2. REST API - a RESTful HTTP API that allows clients to GET information about datasets and granules using the following techniques
   1. GET queries using keyword-value query parameters such as 'bounding_box', 'dataset_id' and 'temporal'
   2. POST queries using an ECHO advanced query language (AQL) construct for more sophisticated queries
3. OpenSearch API - a HTTP GET API following the OpenSearch standard in ATOM format. Note that OpenSearch allows a user to search by 'placename'. This requires an interaction with the external geonames gazeteer for place name convertion.
4. SOAP API - the legacy ECHO search API which uses AQL and the HTTP protocol

Users can retrieve their metadata in a variety of formats including ECHO10, atom and ISO.

Services

ECHO provides ordering and service brokering that connects our users to the provider's services such as ordering, sub-setting etc. Providers can set up definitions of various services through PUMP and assign them to sets of metadata at both the dataset and granule level. When users initiate service invocation through Reverb, for example, ECHO will convey the request to the provider's service. Status of that service invocation is tracked within ECHO and by the provider.

Authentication

Authenticated access is required for some metadata. ECHO uses NASA's User Registration System (URS) to achieve this. ECHO users can register with URS and use those credentials to access controlled data. Consequently, any time a client wishes to authenticate in ECHO (whether it be through Reverb or the API) they must create a profile within URS and use those credentials to login to Reverb or the API.

Providers can control the access to their metadata by assigning Access Control Lists (ACLs) to various sets of inventory. These lists map to groups of users within the URS system.

Components

Catalog REST API

The ECHO Catalog REST API allows clients to search for and retrieve dataset and granule metadata in a fine-grained and RESTful manner. Clients using the Catalog REST API can send and/or receive metadata in a variety of formats including the ECHO10 xml format, ISO, json and ATOM.

Location: https://api.echo.nasa.gov/catalog-rest

ECHO Catalog REST Search Guide

ECHO REST Ingest Guide

ECHO REST API

The ECHO REST API aims to expose business entities (such as orders, option definitions, service definitions, access control lists) within ECHO as resources that can be accessed in a RESTful manner. Clients using the ECHO REST API can send and/or receive messages in XML or JSON format. While the ECHO REST API may not provide for a pure-REST architecture, it does allow for more integration and mashups with browser-based clients through languages such as Javascript.

Currently, only select resources are accessible using the ECHO REST API. Some resources may only be managed via the ECHO SOAP API.

ECHO REST contains an interface to ESDIS's URS (User Registration Service)

Location: https://api.echo.nasa.gov/echo-rest

OpenSearch API

This API provides a standard query interface acting as a facade over catalog-rest. Queries are standard GET keyword-value queries that result in ATOM or HTML responses.

OpenSearch contains a number of interfaces to third party services

1. Open Layers - visual visualization of spatial metadata is provided by this service
2. Geonames gazeteer - for place name resolution

Location: https://api.echo.nasa.gov/opensearch

SOAP API

All services are available to clients using the ECHO SOAP API. Each service is defined with a WSDL, which should can be used when generating client interfaces to ECHO

Endpoints are defined for each service and follow a naming convention based upon the service name and ECHO version. For example, https://api.echo.nasa.gov/echo-v10/AuthenticationServicePortImpl is the publicly available URL to Version 10 of the ECHO Authentication Service.

Location: http://api.echo.nasa.gov/echo/apis.html

FTP Ingest

The FTP Ingest system inserts NASA's Earth Science data into the ECHO database. Metadata is imported, or ingested, from data providers into the ECHO data registry allowing searching and ordering of the new data. Ingest formats and stores new data in the ECHO database directly. The Ingest system interacts with the ECHO kernel through a database table of events that ECHO monitors periodically to notify interested users of new data.

The interface to the legacy ingest system is based on FTP. Providers upload their metadata to a staging area on the ECHO Ingest machine which will then be ingested by ECHO.

All ingested metadata is propagated to the catalog-rest system by legacy ingest.

Reverb

Reverb is a web application that allows users to discover and leverage earth science data and services. In order to do this it needs to communicate with various
ECHO web applications,

• For inventory queries - The Catalog REST API
• For service and order management - The ECHO REST API
• For authentication - URS

Note that the ECHO system is a brokerage service that links science users to providers of earth science data and services. Consequently, services such as ordering and processing are handed off
to the providers via interfaces such as ECHO's order processing api.

Reverb contains a number of interfaces to third party services

1. Google Maps - spatial search constraints can be applied to searches using this graphical user interface
2. Google Analytics - basic usage metrics are obtained using this service
3. Crazy Egg - further usage metrics are obtained from this service
4. Open Layers - visual visualization of spatial metadata is provided by this service
5. NSIDC WMS - 'blue marble' map layer for spatial metadata visualization
6. Geonames gazeteer - for place name resolution
7. ESDIS EMS - for NASA metrics

Location: https://reverb.echo.nasa.gov

Reverb Tutorial

PUMP

PUMP is a web-based interface that provides an easy, screen-driven way for users to perform provider and operational administrative tasks. The main function of PUMP is to give provider operational staff a user interface to manage their data and services.

Location: https://api.echo.nasa.gov/pump

Providers

ECHO houses metadata of the following providers,

• ASF - Alaska Satellite Facility
• CDDIS - Crustal Dynamics Data Information System
• GHRC - Global Hydrology Resource Center
• GSFCS4PA - Goddard Earth Sciences Data and Information Services Center
• LAADS - Level 1 and Atmosphere Archive and Distribution System
• LANCEMODIS - Land Atmosphere Near real-time Capability for EOS
• LARC - NASA Langley Research Center
• LARC_ASDC - NASA Langley Atmospheric Research Center
• LPDAAC_ECS - Land Processes Distributed Active Archive Center
• NSIDC_ECS - NSIDC (National Snow and Ice Data Center) Distributed Active Archive Center
• NSIDCV0 - NSIDC (National Snow and Ice Data Center) Distributed Active Archive Center (legacy V0 system)
• OBPG - Ocean Biology Processing Group
• OMINRT - Ozone Monitoring instrument
• ORNL_DAAC - Oak Ridge National Laboratory Distributed Active Archive Center
• PODAAC - Physical Oceanography Distributed Active Archive Center
• SEDAC - Socioeconomic Data and Applications Center
• USGS_EROS - Earth Resources Observation and Science (EROS) Center at the US Geological Survey