We have a collection in the NSIDC_TS1 provider, AA_L2A.001.  When we validate it we get a warning:

curl -i -XPOST -H "Content-type: application/iso19115+xml" -H "Echo-Token:  XXXX" "https://cmr.uat.earthdata.nasa.gov/ingest/providers/NSIDC_TS1/validate/collection/AA_L2A.001" -d @AA_L2A.001.xml
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Date: Thu, 29 Sep 2016 21:19:03 GMT
Content-Type: application/xml; charset=ISO-8859-1
CMR-Request-Id: 2349bc2b-8ef7-4f11-a553-710e60b4a772
Content-Length: 164
Server: Jetty(9.2.z-SNAPSHOT)

<?xml version="1.0" encoding="UTF-8"?><result><warnings>object has missing required properties (["DataCenters","ProcessingLevel","RelatedUrls"])</warnings></result>

This implies that we should not be able to export it and have CMR validate it.  However, when we try it:

curl -i -XPUT -H "Content-type: application/iso19115+xml" -H "Echo-Token: XXXX" "https://cmr.uat.earthdata.nasa.gov/ingest/providers/NSIDC_TS1/collections/AMSR%2FADEOS-II%20L2A%20Global%20Swath%20Spatially-Resampled%20Brightness%20Temperatures%20V001" -d @AA_L2A.001.xml
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Date: Thu, 29 Sep 2016 21:22:52 GMT
Content-Type: application/xml; charset=ISO-8859-1
CMR-Request-Id: 6b4d17b3-7bb7-4cd8-924c-75bf4f872ff5
Content-Length: 238
Server: Jetty(9.2.z-SNAPSHOT)

<?xml version="1.0" encoding="UTF-8"?><result><concept-id>C1000000709-NSIDC_TS1</concept-id><revision-id>6</revision-id><warnings>object has missing required properties (["DataCenters","ProcessingLevel","RelatedUrls"])</warnings></result>

I just ran the command a few moments ago (9/29 15:22 local time). You can see that the record did just successfully ingest when I ran it:

n5oml01{cfowler}[95]->curl -i -g -H "Echo-Token: XXXX" "https://cmr.uat.earthdata.nasa.gov/search/collections.umm-json?pretty=true&short_name=AA_L2A&provider-id=NSIDC_TS1"
HTTP/1.1 200 OK
Date: Thu, 29 Sep 2016 21:24:56 GMT
Content-Type: application/vnd.nasa.cmr.legacy_umm_results+json; charset=UTF-8
Access-Control-Expose-Headers: CMR-Hits, CMR-Request-Id
Access-Control-Allow-Origin: *
CMR-Hits: 1
CMR-Took: 8
CMR-Request-Id: bc31f278-3361-4e65-824e-fa0dc78849b1
Content-Length: 678
Server: Jetty(9.2.z-SNAPSHOT)

{
  "hits" : 1,
  "took" : 8,
  "items" : [ {
    "meta" : {
      "native-id" : "AMSR/ADEOS-II L2A Global Swath Spatially-Resampled Brightness Temperatures V001",
      "provider-id" : "NSIDC_TS1",
      "concept-type" : "collection",
      "concept-id" : "C1000000709-NSIDC_TS1",
      "revision-date" : "2016-09-29T21:22:53Z",
      "user-id" : "cathyf",
      "deleted" : false,
      "revision-id" : 6,
      "format" : "application/iso19115+xml"
    },
    "umm" : {
      "entry-title" : "AMSR/ADEOS-II L2A Global Swath Spatially-Resampled Brightness Temperatures V001",
      "entry-id" : "AA_L2A_1",
      "short-name" : "AA_L2A",
      "version-id" : "1"
    }
  } ]

So the question is, why are records that are missing "required properties" still ingesting? 

  • No labels

1 Comment

  1. user-7b92a

    Our warning messages aren't very clear here. We're going to be updating them to be more clear about what's going on and we're also currently updating our Client Developer's Guide about this as well.

    We map all metadata into a common domain model called UMM (Unified Metadata Model) in the CMR. We have a schema defined for UMM-C (C = collection) that defines required fields, maximum lengths and field types. We've been working over the past year to incorporate a new version of UMM that's more restrictive in what it requires. Among many other things it requires fields like spatial which are currently optional in the CMR. The more restrictive version is part of an effort to improve metadata quality which will help make data more discoverable by end users. We've recently started validating metadata against this schema in the CMR but we're not enforcing it yet. We return warnings instead of errors. At some point in the future (with plenty of notice for providers) we would shift these warnings to errors and prevent ingest of this data. 

    The message "object has missing required properties (["DataCenters","ProcessingLevel","RelatedUrls"])" isn't descriptive enough as there's no context. We're going to add more information in that error message to indicate this is part of a UMM-C validation that are temporarily warnings.