Hi,

I'm working on the CMR client for dataset level metadata at NSIDC, we are in the process of testing our records using the REST API (UAT) and we ran into a weird issue now submitted as a bug here:  CMR-2927 - Getting issue details... STATUS

Basically we had a document that validated against the schema and yet we got an error when we tried to validate it using the API, the apparent reason was that there are some extra items being used, in this case the <gmd:referenceSystemInfo> tag, which is not mandatory according to the schema. My question would be how can we get a better idea of the actual API requirements? is the schema enough? maybe the API should return the full error stacktrace and/or the documentation needs to be extended?

thanks,

Luis.

  • No labels

3 Comments

  1. user-7b92a

    A 500 error on the CMR API indicates we've got a bug on our end or something misconfigured. This one is definitely a bug in the CMR. We've pulled this issue into our current sprint to fix it. If you ever see a 500 error from the CMR you should let us know about it though we do have logging and alerts to detect these kinds of problems.

    Validation follows these steps:

    1. XML Schema Validation
    2. Parse into UMM data structures.
    3. UMM Validation (Spatial is valid area, temporal is valid)
    4. Ingest business rule validation (There's not another item with the same id, etc.)

    Normally when validation is working correctly we will give you all the errors we can from which ever stage failed along with a 400 status code. You can see that just being valid against the XML schema isn't enough. There are further validations like spatial and temporal and business rule validation. 

  2. Luis Lopez

    That was fast, thanks for the quick response!

    XML validation is the most straightforward, we can also integrate UMM validation (spatial, temporal). However the ingest business validation is a bit obscure to me, i.e. I got this for an update operation I was testing.

    validation error
    <?xml version=\"1.0\" encoding=\"UTF-8\"?>
<errors>
 <error>The Entry Title [PALS Soil Moisture Experiment (1999-2008) 800 m Earth-Fixed Grid Brightness Temperature and Backscatter Match-Up V001] must be unique. The following concepts with the same entry title were found: [C1216079684-NSIDCV0TST].</error>
</errors>

    Is there a wiki for these rules? It would be very useful to us to have a list of what will happen under different conditions, like if a collection is already there and we don't change anything we'll get a 409 if we try to update it, etc.

    1. user-7b92a

      Our Data Partner User Guide does document most of the validation rules but it's not explicit about it. I filed a new issue,  CMR-2935 - Getting issue details... STATUS , to improve that. Take a look at that guide and I can help you with specific issues. 

       

      That error you received says that there's another collection, identified by C1216079684-NSIDCV0TST, with the same entry title. Entry title is one of the unique identifiers of a collection. If C1216079684-NSIDCV0TST is the same collection and you're trying to update it then you're referring to it using a different "native id". That's the id used in the URL to identify a collection that you're ingesting. 

      You can find the collections native id using a curl like the following:

      curl "https://cmr.uat.earthdata.nasa.gov/search/collections.umm-json?pretty=true&concept_id=C1216079684-NSIDCV0TST&all_revisions=true&token=XXX"

      {
        "hits" : 2,
        "took" : 224,
        "items" : [ {
          "meta" : {
            "native-id" : "nsidc0666test", <--------------
            "provider-id" : "NSIDCV0TST",
            "concept-type" : "collection",
            "concept-id" : "C1216079684-NSIDCV0TST",
            "revision-date" : "2016-05-09T14:58:14Z",
            "user-id" : "cmr_nsidc_uat",
            "deleted" : true,
            "revision-id" : 25,
            "format" : "application/iso19115+xml"
          },
          "umm" : {
            "entry-title" : "PALS Soil Moisture Experiment (1999-2008) 800 m Earth-Fixed Grid Brightness Temperature and Backscatter Match-Up V001",
            "entry-id" : "NSIDC-0666-TEST_1",
            "short-name" : "NSIDC-0666-TEST",
            "version-id" : "1"
          }
        }, {
          "meta" : {
            "native-id" : "nsidc0666test",
            "provider-id" : "NSIDCV0TST",
            "concept-type" : "collection",
            "concept-id" : "C1216079684-NSIDCV0TST",
            "revision-date" : "2016-05-09T14:51:36Z",
            "user-id" : "cmr_nsidc_uat",
            "deleted" : true,
            "revision-id" : 23,
            "format" : "application/iso19115+xml"
          },
          "umm" : {
            "entry-title" : "PALS Soil Moisture Experiment (1999-2008) 800 m Earth-Fixed Grid Brightness Temperature and Backscatter Match-Up V001",
            "entry-id" : "NSIDC-0666-TEST_1",
            "short-name" : "NSIDC-0666-TEST",
            "version-id" : "1"
          }
        } ]
      }

      You can see the collection was deleted in the CMR and used the native id nsidc0666test. Did you just get that error yesterday? If so that might be a bug on our part. We should let you recreate a collection with a different native id if the previous version was deleted.

       

Accessibility