The aim of this page is to provide guide to meeting the core requirements for new LANCE Near Real-Time (NRT) elements.
For full details please refer to the LANCE Requirements document: 423-RQMT-002 Available from https://ops1-cm.ems.eosdis.nasa.gov/cm2/
In the LANCE Requirements document, the requirements are broken down in to the following categories:
- System Requirements
- Ingest Requirements
- Data Processing Requirements
- Data Distribution Requirements
- User Services and Support Requirements
- Algorithm and Product Validation Requirements
A summary of the requirements can be found here: Requirements Checklist for LANCE Elements. In addition the LANCE element will help develop an Appendix specific to each LANCE element with specific requirements and/or exceptions to the core requirements. This Appendix should be modeled after the current LANCE provider appendices (e.g. 423-RQMT-002_D_OMI Requirements) available from https://ops1-cm.ems.eosdis.nasa.gov/cm2/.
A. Summary of key requirements
A.1. As appropriate, provide L0, L1 and L2 data products within 3 hours of observation (L3 products will vary).
A.2. Short Name should have the _NRT suffix e.g. AIRIBRAD_NRT.005
A.3. Provide HTTPS interfaces for data distribution.
A.4. Implement redundant processing and distribution servers to ensure operational availability.
A.5. Register metadata in CMR.[1]
A.6. Ensure metadata provided to CMR includes a “CollectionDataType” of “NEAR_REAL_TIME” to identify/discover NRT collections.
A.7. Provide metrics to EMS
A.8. Provide full-resolution browse imagery to GIBS.
A.9. Provide content for the Earthdata website
A.10. Participate in the LANCE Element bi-weekly telecons and bi-annual User Working Group meetings.
A.11. Create Digital Object Identifiers (DOI) for NRT products.
A.12. Provide user support via Kayako
[1] The EOSDIS Common Metadata Repository (CMR) will provide a single source of unified metadata to better serve end users and application developers. With the introduction of CMR, new metadata fields will be required by May 2015. For further information, the UMM documents that are currently under ESDIS Standards Office Review: https://earthdata.nasa.gov/eso/cmr-reviews. Please email eso-staff@lists.nasa.gov if you would like to provide input for this document.
B. System requirements
Servers, software and hardware should:
B.1. Be capable of ingesting data at the rate provided by data provider[1].
B.2. Process and distribute data to users within 3 hours of observation.
B.3. Implement redundancy to ensure a goal of 100% operational availability.
[1] Existing data providers are: EOSDIS Data and Operations System (EDOS) for AIRS, MLS, MISR, MODIS, OMI, OMPS and VIIRS, and the Japan Aerospace Exploration Agency (JAXA) for AMSR2.
C. Ingest requirements
Servers, software and hardware should:
B.1. Be capable of ingesting data at the rate provided by data provider[1].
B.2. Process and distribute data to users within 3 hours of observation.
B.3. Implement redundancy to ensure a goal of 100% operational availability.
[1] Existing data providers are: EOSDIS Data and Operations System (EDOS) for AIRS, MLS, MISR, MODIS, OMI, OMPS and VIIRS, and the Japan Aerospace Exploration Agency (JAXA) for AMSR2.
D. Data Processing requirements
Servers, software and hardware should:
B.1. Be capable of processing....
[1] Existing data providers are: EOSDIS Data and Operations System (EDOS) for AIRS, MLS, MISR, MODIS, OMI, OMPS and VIIRS, and the Japan Aerospace Exploration Agency (JAXA) for AMSR2.
C. Data Distribution
C.1 Provide basic data subscription services to all users.
C.2 Maintain a minimum of a 7-day rolling archive and a maximum of 14 days. Deviations to this will be reflected in respective Appendices.
C.3 Integrate with EOSDIS Earthdata Login (formally known as URS) and restrict data access to users with a valid Earthdata login account.
D. Metrics and Metadata Requirements
D.1 Provide metrics to ESDIS Metric System (EMS) for Ingest, Archive and Distribution as described in the ICD between EMS and the Data Providers[1].
D.2 Short name should have the _NRT suffix e.g. AIRIBRAD_NRT.005
D.3 Create Digital Object Identifiers (DOIs) for collection level products[2]
D.4 For all LANCE collections, include CMR element “CollectionDataType” set to “NEAR_REAL_TIME” to identify/discover NRT collections e.g.
<CollectionDataType>NEAR_REAL_TIME</CollectionDataType>
D.5 Register data granules in CMR within an average of 15 minutes after products are available for download from Element.
D.6 The following metadata elements will be required by May 2015:
◦ Metadata Standard Name
◦ Metadata Standard Version
◦ Science Keywords
◦ Temporal Extent Information
◦ Spatial Extent Information
◦ Platform
◦ Instrument
◦ Campaign[3]
Note: Metadata Standard Name and Metadata Standard Version have recently been added to the ECHO 10 xml schema. The latest ECHO10 schema can be found at: https://git.earthdata.nasa.gov/projects/EMFD/repos/echo-schemas/browse/schemas/10.0/Collection.xsd
[3] Example for campaign entry: <Campaign><ShortName>LANCE</ShortName><LongName>Land, Atmosphere Near real-time Capability for EOS</LongName></Campaign>
E. LANCE User Services and Support Requirements
E.1 Provide web content and links to data, FAQs Help, Algorithm Theoretical Basis Document (ATBD) or equivalent, User Guide and System status for the Earthdata LANCE website.
E.2 Utilize the ESDIS (Kayako) User Support Tool (UST) to provide user services and direct users to DAACs or other user service teams as appropriate for user questions.
E.3 Inform users of planned down time and gaps in NRT data .
E.4 Notify users of any quality issues that arise unexpectedly due to events such as data gaps, anomalies, processing failures, instrument or spacecraft events and/or issues that they are aware of.
E.5 Nominate a point of contact to participate in bi-weekly LANCE telecons.
E.6 Provide updates for and participate in LANCE UWG meetings, which are held twice a year.
F. LANCE Algorithm and Product Validation
F.1 Coordinate with the Science Team to make available LANCE products using the latest algorithms.
F.2 Provide products from both existing and new / latest algorithms (for the period specified in the LANCE Element Appendix) to give users sufficient time to transition to the latest algorithms. Provide information on the differences between the algorithms on the Earthdata LANCE website.
F.3 Work with the science team to compare NRT products to standard products and identify and document the differences.
F.4 Provide information on quality differences between NRT and standard science data products and provide descriptions of the differences on earthdata.nasa.gov
G. GIBS Imagery Requirements
G.1 Develop and baseline the GIBS Imagery as outlined in the ICD between the GIBS and Imagery Providers (423-ICD-009 obtained from https://ops1-cm.ems.eosdis.nasa.gov).
G.2 Provide full-resolution daily composite and/or granule-based visualizations of science data parameters.
G.3 Provide imagery products to GIBS within 45 minutes, on a best effort basis, after products are available for download from Element.
G.4 Provide imagery products to GIBS after metadata has been registered with ECHO/CMR.
STEPS REQUIRED TO REGISTER METADATA IN CMR
PROVIDING METRICS TO EMS
EMS Help address is: ems-help@lists.nasa.gov
To send metrics to EMS a new LANCE element needs:
- An EMS "Data Provider" account via NAMS
- A connection to file server 1 (fs1) to be able to send files (EMS will help set this up). The element needs to know what type of log files they'll be sending in, a Datafile Manifest and they'll need the ICD (currently being updated in COMET) that has info format of the files, etc… Once the Datafile manifest is set up they can send in test files.
- Files will be processed once schema is set up.
- EMS will need the file server IP, Once the acct is created, EMS will log in, set up the keys, etc…
If there are any missing or incorrect files with missing search expressions etc then the LANCE element will be sent an email and will be responsible for fixing any problem. …if there’s a failed validation an email is sent right away. Missing files can be sent to the temp directory to be reprocessed.
EMS Analysts will generate the metrics for ESDIS from the data they receive. The 4 week report is generated weekly on Wednesdays. If there is missing data they will make a note in the report. But for things like search expression or product metadata error and there isn’t timely follow up there can be issues. That’s why the emails are sent. ESDIS approval is required for reprocessing more than 90 days of data. There is a GUI interface called APEX that lets Data Providers look at metrics as well as reports that can be run and emailed. Lalit generates metrics from EMS for ESDIS.
Sample schema/log file, Datafile Manifest Template and ICD
