ECHO has been replaced by the Common Metadata Repository (CMR), a high-performance, high-quality, continuously evolving metadata system that catalogs all data and service metadata records for the EOSDIS system and will be the authoritative management system for all EOSDIS metadata.
The ECHO system acts as an order brokering service between end users and Data Partners. Users may submit orders for collections and granules that have been ingested into ECHO and marked as 'orderable.' ECHO will keep basic metrics and information regarding the order including the orderer's information, order contents, and historical order updates.
Through the means described in Section 6, Data Partners can configure access control for their data to determine which users are allowed to order data. All information in presented in this section will assume that Data Partners have configured their data access mechanisms as needed to support their needs. Data Partners generally use Provider User Management Program (PUMP) or the API itself to configure and track orders submitted to their data center. However, a data-center specific tool may be written to use the ECHO API in order to facilitate order management.
For Order Fulfillment API Documentation, Order Fulfillment Types XML Schema, and the Order Fulfillment Service WSDL refer to the Data Partner Tools page on the ECHO website.
Data Partners who would like ECHO to facilitate order brokering services must implement an "Order Fulfillment Service" based on the OrderFulfillmentService API. The API specification and documentation can be found here (http://api.echo.nasa.gov/echo/apis.html) in the "Order Fulfillment" section. In short, an "Order Fulfillment Service" implements the necessary methods in order to receive order quotes, submission, and cancellation requests from ECHO. If the service is correctly written according to the specification made public by ECHO, then the provider will successfully receive order workflow information from ECHO. The data partner may then perform whatever internal activities are needed in order to fulfill the order.
While fulfilling an order, the Data Partner may call methods on the ECHO OrderProcessingService API to update, accept, reject, or close existing ECHO orders which have been submitted. The API specification and documentation can also be found here (http://api.echo.nasa.gov/echo/apis.html).
Each ECHO provider has a set of "provider policies" which specify necessary information in order to facilitate and manage orders placed for data within a Data Partner's holdings. These policies are configurable using PUMP or the ProviderService API. It is important that Data Partners correctly configure these policies in order to ensure orders are properly brokered by ECHO. The following topics discuss the policies that Data Partners are to maintain.
When an order is submitted through ECHO, it is possible that the Data Partner's Order Fulfillment service will be unavailable due to maintenance or other situations. Were this to occur, ECHO can queue any orders which fail submission to be retried later. Each Data Partner may configure their own retry policies. This policy includes the following:
- Retry Attempts - The number of retries ECHO will perform until failing order submission.
- Retry Wait Time - The number of seconds ECHO will wait between retries.
It is important to ensure that the retry policies are sufficient to account for Data Partner regular maintenance and possible extended failure situations. ECHO recommends at a retry policy that attempts order submission every hour for at least 48 hours. As an example, a policy which has a wait time of 3600 seconds and 48 attempts will ensure that this 48 hour retry policy is met.
Supported Order Transactions
ECHO supports the following order actions:
A Data Partner can configure their ECHO provider to support any of these order actions. Order submission must be explicitly chosen in order to allow ECHO to accept orders for a provider's data. Data Partners with data that has an associated cost can support quoting to allow users to request an initial quote outlining the cost of their order prior to submission. Data Partners whose Order Fulfillment service allows for orders to be cancelled after submission can choose to support order cancellation through ECHO.
Endpoint for Receiving Orders
The ECHO provider policies must be configured with the Data Partner's Order Fulfillment service's "end point," a Uniform Resource Identifier (URI), also known as network address, which ECHO will use for all order communication. The network address is usually either an IP or HTTP address.
It is important to make sure that your network will allow ECHO to communicate with this URL. Please contact ECHO Operations to get the originating IP address information.
Order Creation Settings
Data Partners have the capability of configuring the following order settings which ECHO will enforce during order creation:
- DuplicateOrderItemSupport - This indicates whether or not the provider supports separate order items with the same catalog item in the same order. If this is set to false ECHO will not allow an order to contain more than one order item to have a particular catalog item.
- Maximum Items Per Order - This indicates the maximum number of order items which can be added to a single order. If no value is set, then order sizes are unbounded.
Secure Socket Layer (SSL)
In order to support SSL encrypted communication between ECHO and the Data Partner's Order Fulfillment service, the Data Partner's SSL Certificate must be configured in ECHO. Once the public, PEM-encoded key from the certificate has been entered or updated in the provider policies, ECHO Operations must acknowledge and activate the changes. ECHO Operations will be notified via email from the ECHO system when changes are made. Once activated, the new SSL certificate will be used during secure communication. It is important that Data Partners ensure that their server hosting the order fulfillment service is correctly configured to utilize SSL. While troubleshooting SSL communication between ECHO and the Data Partner's Order Fulfillment service, the following items should be verified:
- The configured endpoint in PUMP must specify the https:// protocol
- Firewall access must be granted by the Data Partner allowing ECHO to communicate to the secure port.
- SSL communication must be enabled by the Data Partner in the PUMP "Provider Policies."
- ECHO Operations must activate an updated SSL certificate in PUMP.
- The certificate placed in PUMP should be PEM encoded. An example is shown below.
Data Partners wishing to have additional information provided to their Order Fulfillment service during order communication may configure a custom XML block of information which ECHO will include in every order transaction.
It is an ECHO Client's responsibility to facilitate the creation of an order within ECHO. ECHO will subdivide an order by data center so that each Data Partner will be able to interact with the portion of the order for which they are responsible for fulfilling. The following diagram shows the workflow of an order as it passes into a Data Partner's order Order Fulfillment Service, described in Section 7.1. The entry point to this workflow is when ECHO requests an order quote, submission, or cancellation. Order cancellation and quotes are typically atomic synchronous activities, so their workflows are shown as terminating after the action is accepted or rejected.
Order submissions may be accepted with either an automatic or a manual distinction. An ‗automatic' distinction signifies that the order has been completed by the provider and the order will be closed by ECHO. If the manual distinction is used, this indicates that the provider will notify ECHO through the OrderProcessingService as the order is fulfilled. Shown in the diagram are the most common updates, including the addition of a provider tracking id, status updates, and the final closure.
Note that quote, submissions, and cancellations and quotes may also be postponed, however these workflows are not displayed.
An ECHO Order associated with a single data partner will have on of the following states during its lifecycle:
- Validated - Order has been validated and all information is correct
- Not_Validated - Order has not been validated
- Quoting - Order has been sent to provider for quoting
- Quote_Rejected - Provider does not support quoting
- Quote_Failed - Order failed quoting
- Quoted - Provider has responded with quote
- Submitting - Order has not yet reached the provider
- Submit_Rejected - Order rejected by provider
- Submit_Failed - Order failed to send to provider
- Processing - Provider received order and process
- Cancelling - In the process of Cancelling
- Cancelled - Order has been cancelled (no change from Closed status)
- Closed - Order has been completed
The following sections outline the information included in an ECHO order transmitted to a Data Partner through the Order Fulfillment Service.
ECHO Order vs Provider Order
Within ECHO, there are two distinct items which may be referred to as an "order." These include an "ECHO Order" and a "Provider Order." These two concepts are described below:
- Provider Order - A provider order contains information regarding the items which have been ordered from a Data Partner's holdings, an associated provider tracking ID (discussed later), and other quote and state information. A provider order does not include order information.
- ECHO Order - An ECHO order contains information about the user and client submitting the order, notification level, and a list of provider orders containing the information listed above.
An ECHO Order is identifiable by an OrderGuid, which is a unique identifier for that order within ECHO. The "Provider Order" is identifiable by a combination of the OrderGuid and ProviderGuid, the internal identifier used by ECHO to identify a Data Partner.
Provider Tracking ID
Data Partners often have an internally managed order tracking system which uses a different ID than the ECHO order ID. In order to facilitate improved order tracking capabilities, a Data Partner's Order Fulfillment Service may choose to update the provider's ECHO order with this internal tracking ID. Data Partners should update the corresponding order in ECHO with their unique ID as soon as it is assigned.
Each order contains a listing of order items, which match to records within the Data Partner's ECHO holdings. Each order item included in the order includes the following information:
- Granule UR - The Data Partner's unique granule identifier which will contain a value if the order item is a granule.
- Producer Granule ID - The granule's producer granule identifier which will contain a value if the order item is a granule and the provider has supplied a value for the metadata field.
- Dataset ID - The Data Partner's unique collection identifier to which the granule belongs if the order is for a granule, and the identifier of the collection being ordered if the order is for a collection.
- Catalog Item Guid - The ECHO unique identifier for the catalog item being ordered.
- Quantity - The quantity of this order item being ordered. If the order is for electronic delivery (that is, FTP), then the quantity should always be one. If the order is for the delivery of pre-packaged media, then the quantity can be considered the number of media that are being requested.
- Price - The predetermined price of the item.
- OptionSelection - Additional ordering options for this item. The option selection comes from the client created output of the option definition ECHO Form which the user has completed. The selection will be in an XML format as defined by the ECHO Form specification. If the original item had multiple option definitions, the name of the option selection can be used to determine which set of options the user selected.
Each order contains information about the user who has submitted the order. ECHO will enforce that guest users submit the necessary information as well as registered users. This user information includes the following information:
- User Id - The ECHO User ID (e.g. john.doe)
- Shipping Address - The user's shipping address indicating where an order should be physically shipped.
- Billing Address - The user's billing address identifying who will be paying the bill for the order.
- Contact Address - The user's contact address identifying who is placing the order.
- User Domain - The domain within which the user is associated. (e.g. K12)
- User Region - The region in which the user is located. (e.g. USA)
As shown, the user's information may contain three address fields, however only the Contact Address is required. Each address field contains the same fields, but may contain different information depending on how the user has submitted their order. The following fields are included in each of the three addresses:
- Name - The user's first and last name and role (e.g. research assistant).
- Address - The user's street address. (Optional)
- PhoneNumber(s) - A list of the user's phone numbers. (Optional)
- EmailAddress - The user's email address.
- Organization - The user's organization name.
Some additional information for these fields is given in the following sections.
The address field is a structure containing the following fields:
- Name - The name that the user uses to describe the address (for example: Home, Work, Project, etc.)
- USFormatFlag - Indicates whether a provider can rely on the address looking like a standard US address. The rule is that the system will validate those addresses that have the US Format flag set by checking that Street1, City, State, Zip Code, and Country are all set. If the US Format flag is clear, then only Street1 and Country are required.
- Streets - ECHO allows for up to 5 street address lines, each 32 characters long.
- City - The user's city.
- State - The user's state, if applicable.
- Zip - The user's zip code, if applicable.
- Country - The full name of the country. This should conform to the published list of approved country codes.
The ECHO system allows user to enter a freeform string to define their contact, shipping, or billing address. In cooperation with the EOSDIS User Services Working Group, ECHO has created a list of approved country names that complies with the ISO 3166 standard. This list of names will appear in the ECHO supported PUMP and WIST tools for for user registration and order creation in WIST. ECHO will support all countries on this list and additional user-entered countries. ECHO Data Partners are encouraged to use this list in their applications to provide a consistent list of countries to all ECHO users. The full list is available at the following location: http://www.echo.nasa.gov/documents/common/ApprovedCountryList.xls.
The term "order option" refers to the mechanism by which a Data Partner can request specific information from a user at the time of ordering. For example, an order option may require that users provide ftp-push information, or specific subsetting information. The ECHO Forms Specification defines the structure and content of the XML documents which are used in ECHO to specify the order options assigned to a collection within the ECHO data catalog. The term "ECHO Form" is used to refer to the XML document created using the standards in the ECHO Forms Specification.
It is the responsibility of the ECHO Data Partners to design, upload, and assign order options to their catalog items using the ECHO Forms Specification. If a provider's collection does not allow ordering, then an order option may not be assigned. It is also the responsibility of the Data Partner to verify that their order options correctly define the information needed to complete the order process.
It is the responsibility of an ECHO Client Partner which is offering the service of ordering ECHO data to implement functionality that will display the ECHO Form. It is strongly encouraged that a Client Partner implementing the ordering functionality ensure that the entire ECHO Forms Specification can be represented. ECHO Data Partners may choose to utilize any and all parts of the specification. If ordering is not a feature which is included in the scope of a Client Partner's work, then they need not implement the ECHO Forms Specification.
Not all catalog items have associated definitions - in which case they do not need option selections when ordering. However, if there are any options that are required by that catalog item, they will have to be filled out before an order can be validated and/or submitted (or else an error will be returned by ECHO).
ECHO Forms Specification
As mentioned previously, an ECHO Form is an XML document outlining the information which is to be requested from a user during the order process and how an ECHO client should facilitate the information gathering. The ECHO Forms Specification, outlines the complete set of syntax and features utilized by ECHO Data Partners when creating ECHO Forms. The specification also describes the workflow and user interface elements which are used as form controls.
For further information regarding ECHO forms, including the specification and sample forms, please visit the "Data Partners >> Fulfilling Orders" portion of the ECHO Website.
Data Partners may manage their option definitions through PUMP or the DataManagementService API methods. An ECHO option definition contains the following information:
- Name - The identifying name for the form which must be unique amongst all option definitions created by a Data Partner. This value will be displayed to the users to identify an option definition.
- Scope - The scope of the option definition. This may be system or provider
- ECHOForm - The XML ECHO Form outlining the information which is to be requested from a user during the order process and how an ECHO client should facilitate the information gathering.
- Description - A textual description of the form which will be displayed to the user to give additional information regarding the form. This is useful when multiple option definitions are assigned to the same collection. The description can be used to help a user know which definition they want to use.
- SortKey - This optional field may be used to define a sort order for definitions. This is also useful when multiple option definitions are assigned to the same collection. The values in this field will be sorted based on a standard text sort order.
Provider vs. System Forms
The ECHO Forms that are created and uploaded by Data Partner are referred to as "provider level forms." This designates the fact that they are visible only to the Data Partner who uploaded the form. There are also ECHO Forms that are created and uploaded by the ECHO Team, and they are referred to as "system level forms." These forms are visible to all providers. The purpose of system level forms is to centralize common option definition elements which can be shared by all providers in order to reduce duplication of form components.
An option definition cannot be edited once it has been created. This is due to the fact that an order may have been created against the option definition. Were the definition to be changed before the order was submitted, then the option selection within that order would be invalid for all items it had been applied to. For that reason, forms can only be deprecated or deleted.
If a change is needed, a new form should be created, assigned to the appropriate collection(s) and the previous option definition should be deprecated. Deprecation will allow the option definition to persist within ECHO. Orders which have been created or submitted against this option definition will continue to process successfully. New orders will not be able to use this deprecated definition.
When an option definition is deleted, all orders that have been created and validated with that form, but not submitted, will no longer be valid. All orders that have been submitted with the deleted form will not be affected.
When deleting an option definition, ECHO must perform internal bookkeeping to disassociate order items with the deleted item. Due to the potentially large number of order items, this activity may take a significant amount of time. Providers are encouraged to utilize the deprecation capability and wait at least 60 days before deleting the option definition. ECHO has a 60 day retention period, so waiting beyond this time will eliminate the potential performance issue
Within ECHO, an option assignment must be created between an option definition and collection in order to signify that order options exist during the ordering process. Multiple definitions may be assigned to a single collection, and a definition can be assigned to multiple collections. An option assignment may also include an XPath statement which will filter on a granule's metadata to determine whether the granule should use the assigned option definition, or not. For instance, the following XPath expression may be used as part of an Option Assignment to include a specific option definition if the SWIR_ObservationMode and VNIR1_ObservationMode additional attributes have a value of "ON". As you can see, the XPath starts with the
/results/provider/result/path. The remaining path, beginning with the GranuleURMetadataelement, is evaluated against the Granule Results DTD, found here: http://api.echo.nasa.gov/echo/dtd/ECHOGranuleResults.dtd. Any combination of expressions and metadata fields from the granule results may be used as a part of an Option Assignment XPath statement.