Versions Compared

Old Version 158

changes.mady.by.user Doug Newman

Saved on

compared with

New Version Current

changes.mady.by.user Scott Ritz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Page Tree Search

Panel
borderColor
#9933FF
Yellow
titleColorWhite
titleBGColor
#9933FFtitleGeneral Information ExpandtitleWhat is the
Yellow

NASA's Common Metadata Repository (CMR)

? The Common Metadata Repository (CMR)

is a high-performance, high-quality, continuously evolving metadata system that catalogs all data and service metadata records for

the EOSDIS system

NASA's Earth Science Data and Information System (ESDIS) and will be the authoritative management system for all EOSDIS metadata.

 These

These metadata records are registered, modified, discovered, and accessed through programmatic interfaces leveraging standard protocols and APIs.

Expand
titleWhat are the objectives of CMR?

CMR is designed to:

  • serve as a middleware replacement for the ECHO and GCMD’s backend. Note: Users of GCMD should see no impact - as the GCMD frontend will remain as is.
  • handle metadata at the Concept level; including Collections, Granules, Visualizations, Parameters, Documentation, Services, and more.
  • manage hundreds of millions of metadata records; making them available through high performance, standards compliant, temporal, spatial, and faceted search.
  • incorporate both human and machine metadata assessment features that work to ensure the highest quality metadata possible.
  • support multiple metadata standards using an evolvable Unified Metadata Model (UMM).
Expand
titleWhat is the UMM?

The UMM is an extensible metadata model which provides a ‘Rosetta stone’ or cross-walk for mapping between CMR-supported metadata standards. Information detailing what the UMM is and how it works is available on Earthdata.gov

Expand
titleCMR FAQ
Expand
titleHow does CMR work?

The CMR system is composed of:

  • CMR itself (formerly the [ECHO])
  • GCMD
  • International Data Network (IDN)
  • Earth Science Data and Information System (ESDIS) Metrics System (EMS)
  • All related tools (internal and external)
  • All metadata — including UMM concepts, the GCMD Keywords Controlled Vocabulary, and other controlled vocabularies.                                                         

                       Image Removed

                                                     

                                                   High Level Architecture Diagram
                

The CMR is designed to handle metadata at the Concept level. Collections and Granules are common metadata concepts, but this can be extended out to Visualizations, Variables, Documentation, Services, and more. The CMR provides a flexible ingest system with pluggable adapters that can handle multiple metadata record formats, multiple metadata record concepts, and relationships and validations between them. As new formats are introduced, new ingest adapters can be written for the CMR to provide ingest, validation and search support and response adapters provide format conversions for backward compatibility.                                                       

                      Image Removed

                                                         Metadata at the Concept Level  

For more information on the CMR, please see the CMR Earthdata Homepage.



Panel
borderColor#6654BA
titleColorWhite
titleBGColor#668AF0
titleCMR Search

CMR Search API Documentation

CMR Search Site Documentation

CMR Client Partner User Guide

For more information, see https://cmr.earthdata.nasa.gov/search


Panel
borderColor#6654BA
titleColorWhite
titleBGColor#9045B3
titleUnified Metadata Models (UMM)

UMM - Collections (UMM-C)

UMM - Granules (UMM-G)

UMM - Services (UMM-S)

UMM - Variables (UMM-Var)

UMM - Tools (UMM-T)

For more information, see UMM Documents


Panel
borderColor#6654BA
titleColorWhite
titleBGColorOrange
titleCMR Support

Questions, comments, technical issues, and change requests should be sent to: support@earthdata.nasa.gov




Panel
borderColor#6654BA
titleColorWhite
titleBGColor#45B39D
titleCMR Ingest

CMR Ingest API Documentation

CMR Data Partner User User Guide

CMR Harvesting Best Practices

Onboarding Process for a New CMR Provider

For more information, see https://cmr.earthdata.nasa.gov/ingest/


Panel
borderColor#6654BA
titleColorWhite
titleBGColor#B3A945
titleOther Documentation/Related Work

CMR Git Repository

CMR Requirements Authoring Guide

CMR Life-Cycle

CMR Workload Specification

EED-2 Open Sourcing Guide

ESO-CMR Reviews 

Earthdata Search Client

GCMD Keyword Landing Page


Panel
borderColor#6654BA
titleColorWhite
titleBGColorGreen
titleInformation About Working With This Wiki


Panel
borderColorMaroon
titleColorBlack
titleBGColorWhite
titleInformation about working with this wiki:


Expand

Purpose of the Wiki

This wiki is intended strictly for the use of sharing information about NASA Earth Science Data and Information System (ESDIS) topics. For more information please click here .

We welcome your participation and comments within Earthdata Wiki.

Rules of Conduct

Earthdata Wiki is governed by the general rules of respectful civil discussion. ESDIS may monitor any user-generated content as it chooses and reserves the right to remove content for any reason whatsoever, without consent. The following rules of conduct are to be followed when using the Earthdata Wiki. Breach of these rules will result in users loses privileges or being banned permanently.

Participation requires a valid Earthdata Login. Only approved NASA users and affiliates will be allowed to upload data. If a wiki space becomes dormant for a period of more than 2 years, ESDIS reserves the right to delete/remove it.

In order to maintain an environment where everyone is welcome and discussion flows freely, certain types of conduct are prohibited on Earthdata Wiki. Specifically restricted conduct includes trolling, flaming, ranting, personal attacks, harassment, threats, doxing, racism, discrimination, hate speech, sexism, spamming, off-topic posting, pyramid quoting, rumor mongering, impersonation, advertising (including HTML links and pictures), and linking to applications or utilities of uncertain intent.

In addition, the Earthdata Wiki is not for discussion of real-life current affairs, news, politics, or religion. Discussion should revolve around ESDIS topics. Specific content prohibited on Earthdata Wiki includes pornography, profanity, and real money trading (RMT).


NASA makes no representations or warranties regarding the information posted.

Expand
titleWhat are the benefits of CMR?

Enhanced Performance

Modern Earth Science applications strive to provide end users with nearly immediate access and interactivity across massive stores of Earth Science data. That data is discovered, navigated, and often interrogated through science metadata. As the range of applications grow and more and more information moves from the underlying science data to metadata, the challenges of navigating even just the metadata increases. CMR is designed to handle hundreds of millions of metadata records; making them available through high-speed performance, standards compliant, temporal, spatial, and faceted search.

The CMR builds on the work done by ECHO and the GCMD to provide a unified, authoritative repository for NASA's Earth Science metadata.

                         Image Removed  

                                                        End-User Interface Diagram

Quality Assurance

High performance access to metadata is only part of the problem. To be useful to a broad range of Earth Science applications, the metadata must be of high quality, complete, and consistent. The CMR incorporates both human and machine metadata assessment features that work to ensure the highest quality metadata possible. During ingest, automated metadata scoring rubrics are applied giving data providers insight into how to make their data more discoverable or usable by end users. Science Coordinators and review teams can review metadata that fails verification or lacks required information to help providers make their metadata more consistent and complete.

 

              Image Removed            

                                                        Metadata Quality Assurance Diagram

                                               

Consistent Metadata Representation

  1. The CMR's ingest adapter framework supports pluggable adapters which validate distinct metadata formats such as ECHO10, GCMD DIF, and ISO19115 against a common set of core metadata requirements, the Unified Metadata Model (UMM).
  2. The CMR will be able to take multiple metadata records associated with a common core concept, such as a Collection, and merge the disparate information into a robust and standards compliant ISO19115 representation for interested clients.
  3. As additional metadata concepts are introduced to the CMR, new ingest adapters will provide verification and search indexing capabilities across diverse metadata such as visualization and parameter information.

                                                                 

Image Removed

 

Expand
titleHow does CMR search and ingest work during transition?

All search and ingest that are done via ECHO now will continue to be serviced. The ECHO API will work seamlessly with the CMR and will be backwards compatible. However, there is a period of transition from returning ECHO results to returning CMR results. During this transition period, ECHO will fan-out ingest and search requests to the CMR system.

Panel
borderColor#FFE0C2
titleColorBlack
titleBGColor#FFE0C2
titleThe diagram below outlines the high-level interaction between these systems:

Image Removed

Info

For more information, please email support@earthdata.nasa.gov and include 'CMR' in the email subject.

Expand
titleCMR and reconcilliation

Reconciliation making sure your local provider metadata inventory matches CMR's metadata inventory

ECHO provided long and short form reconciliation functionality to providers. This was a vital piece of functionality for ECHO, prior to catalog-rest, for the following reasons,

  1. ECHO ingest was asynchronous. Once you FTP'd your files at some point in the future it was ingested and it may fail. How do I know what failed?
  2. ECHO ingest was a batch process. If a batch failed it was tricky trying to figure out what part of that batch failed.
  3. ECHO normalized the metadata in a database. It deconstructed what the provider put in and reconstructed it when it was searched for. That was an error-prone process

Consequently, providers needed to verify that the ECHO inventory matched what they had locally using separate processes. Short form verification allowed you to match inventory at a coarse level using id matching. Long form allowed you to determine discrepancies in content of metadata.

What is important to a provider is the following,

  1. The same collection and granule metadata is represented in both systems - short form reconciliation
  2. The content of the above is the same - long form reconciliation

How can we do that in CMR?

  1. When you ingest metadata into CMR it is a synchronous process. It is also searchable within CMR within a few milliseconds. After a successful ingest, CMR will return a concept id of that piece of metadata. You can search for that piece of metadata using that concept id and/or the native id. This way you can do long-form reconciliation on a metadata-by-metadata basis.
  2. You can get the number of collections or granules in a provider using the holdings endpoint. This, in conjunction, with step 1, gives you short-form reconciliation.

If step 1 fails for a piece of inventory you can trouble-shoot and repeat the exercise. If step 2 fails them additional triage is required form the CMR Operations group.

Panel
borderColorBlue
titleColorWhite
titleBGColorBlue
titleCMR/UMM Links
Column
width60

CMR Links

Column
width40

UMM Links  

 

Documentation Links

Panel
borderColorGreen
titleColorWhite
borderWidth2
titleBGColorGreen
titleCMR Service Desk
Tip

Questions, comments, technical issues, and change requests should be sent to: support@earthdata.nasa.gov

 

 

 

 

 

 

Panel
borderColormaroon
titleColorBlack
titleBGColorPeriwinkle
titleRelated Links

CMR Overview

CMR Project Home

ESO-CMR Reviews  

Panel
borderColorOrange
titleColorBlack
titleBGColorPeriwinkle
titleRelated Work

NASA ISO for EOSDIS

ECHO Home and Technical Documentation

GCMD Home

CMR Quality Software Wiki (GCMD wiki)

Global Imagery Browse Services - GIBS





Panel
borderColor#CCCC00
titleColorBlack
title
JIRA and Jama Links
Restricted Access

CMR

JIRA Inbox borderColorblacktitleColorBlacktitleBGColorPeriwinkletitleRestricted Access

Operations

CMR JIRA Project

Jama

Panel

Metrics

Internal Team Documents

CMR Quality JIRA Project

CMR Git Repository