...

Check You Have Correct Permissions to Your Chosen Collection

In order to To subscribe to a collection's granules, you need to make sure you can update and read itthe collection. Use the following command to make sure you have permissions.

...

Check you have correct permissions

curl --location 'https://localhost:8080/access-control/permissions?user_id=<your-user-id>&concept_id=C3261894785-JM_PROV1'

Response:

{"C3261894785-JM_PROV1":["read","order"]}

If you do not have permission use the following to create one if one does not exist, but only if you are the provider. Otherwise you will have to ask the provider to give you permission.

Creating permissions for first time

curl --location 'https://localhost:8080/access-control/acls' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "group_permissions": [acls' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "group_permissions": [
        {
            "permissions": [
                "read",
        {
            "updatepermissions",: [
                "deleteread",
                "orderupdate",
            ],
    "delete",
        "user_type": "guest"
       "order"
            ],
            "user_type": "guest"
        },
        {
            "permissions": [
                "read",
                "update",
                "delete",
                "order"
            ],
            "user_type": "registered"
        }
    ],
    "catalog_item_identity": {
        "name": "Guest and Registered Users Collections",
        "provider_id": "JM_PROV1",
        "granule_applicable": true,
        "collection_applicable": true,
        "collection_identifier": {
            "concept_ids": [
                "C3261894786-JM_PROV1", "C3261894785-JM_PROV1", "C3261906797-JM_PROV1"
            ],
            "entry_titles": [
                "Collection for CMR-10162_1 v1.18.1", "Collection for CMR-10162_2 v1.18.1", "Collection for CMR-10162_3 v1.18.1"
            ]
        }
    }
}'

...

{
    "revision_id": 1,
    "concept_id": "ACL3261894787-CMR"
}

Updating permissions for existing ACL

If a permission does exist, but it needs to be updated then use the following command, assuming you are the provider.  Otherwise the provider will be to be contacted to provide permissions.

Updating permissions for existing ACL

curl 

curl --location --request PUT 'https://localhost:8080/access-control/acls/ACL3261894787-CMR' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "group_permissions": [
        {
        "user_type": "registered",
        "permissions": [
                "read",
                "update",
                "delete",
                "order"
            ]
        }, 
        {
        "user_type": "guest",
        "permissions": [
                "read",
                "update",
                "delete",
                "order"
            ]
        }],
    "catalog_item_identity": {
        "name": "All Granules",
        "provider_id": "JM_PROV1",
        "granule_applicable": true,
        "collection_applicable": true,
        "collection_identifier": {
            "concept_ids": ["C3261894786-JM_PROV1", "C3261894785-JM_PROV1", "C3261906797-JM_PROV1"],
            "entry_titles":["Collection for CMR-10162_1 v1.18.1", "Collection for CMR-10162_2 v1.18.1", "Collection for CMR-10162_3 v1.18.1"]
        }
    }
}
'

...

{
    "revision_id": 5,
    "concept_id": "ACL3261894787-CMR"
}

Check you have correct permissions

curl --location 'https://localhost:8080/access-control/permissions?user_id=jmaeng&concept_id=C3261894785-JM_PROV1'

Response:

{"C3261894785-JM_PROV1":["read","order"]}

Create a Subscription with the CMR

Once your queue is set up then you need to create a new subscription with the CMR

To create a new subscription you will need to use the CMR API and send a subscription request.  In the request you will need to send a subscription record that follows the UMM-Sub schema version 1.1.1

Following is an example UMM-Sub record 

Create a Subscription with the CMR

Once your queue is set up then you need to create a new subscription with the CMR

To create a new subscription you will need to use the CMR API and send a subscription request.  In the request you will need to send a subscription record that follows the UMM-Sub schema version 1.1.1

Following is an example UMM-Sub record 

{"Name": "<The name of your subscription>",
 "Type": "granule",
 "SubscriberId": "<Your Earthdata Login Name>",
 "CollectionConceptId": "C1200463968-CMR_ONLY",
 "EndPoint": "arn:aws:sqs:<region>:<account>:<queue-name>",
 "Mode": ["New", "Update", "Delete"],
 "Method": "ingest",
 "MetadataSpecification": {
   "URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
   "Name": "UMM-Sub",
   "Version": "1.1.1"
 }
}

{"Name": "<The name of your subscription>",
 "Type": "granule",
 "SubscriberId": "<Your Earthdata Login Name>",
 "CollectionConceptId": "C1200463968-CMR_ONLY",
 "EndPoint": "arn:aws:sqs:<region>:<account>:<queue-name>",
 "Mode": ["New", "Update", "Delete"],
 "Method": "ingest",
 "MetadataSpecification": {
   "URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
   "Name": "UMM-Sub",
   "Version": "1.1.1"
 }
}

The above subscription record that gets sent as the payload with the subscription create/update request is described below.

...

The Method element signifies if this is an search type of subscription or an ingest type of subscription. Valid values are either Ingest or Search. Use ingest, if the desire is to be notified when a granule is ingested, updated, or deleted.  Use Search otherwise.

Copy the rest of the elements as they are in the example.

Use the CMR API to create the subscription

https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#subscription

Example CURL command:

curl --location 'https://localhost:8080/ingest/subscriptions/jyna_ingest_subscription_3' \
--header 'Content-Type: application/vnd.nasa.cmr.umm+json' \
--header 'Authorization: ••••••' \
--data-raw '{
    "Name": "Ingest-Subscription-Test",
    "Type": "granule",
    "SubscriberId": "mysubscriberid",
    "EmailAddress": "myemail",
    "CollectionConceptId": "C3261906797-JM_PROV1",
    "EndPoint": "arn:aws:sqs:<region>:<account-id>:<my-queue-name>",
    "Mode": ["New","Update"],
    "Method":"ingest",
    "MetadataSpecification": {
        "URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
        "Name": "UMM-Sub", 
        "Version": "1.1.1"
    } 
}

Response:

<?xml version="1.0" encoding="UTF-8"?>
<result>
    <concept-id>SUB3261906801-JM_PROV1</concept-id>
    <native-id>jyna_ingest_subscription_3</native-id>
    <revision-id>1</revision-id>
</result>

Confirm the Subscription

Once the subscription request has been successfully ingested into the CMR look in your queue either programmatically or through the AWS Console to find a subscription confirmation message.  If a subscription message is not received a CMR operator will have to look at the subscription logs to see what the errors are.  Almost all of the errors are because the queue access policy isn't sufficient from step 1.

An example of a subscription confirmation message is shown below:

{
  "Type" : "SubscriptionConfirmation",
  "MessageId" : "cd7f9a70-e89d-44b0-aa6c-d3be3f3b0316",
  "Token" : "2336412f37...5a7640ed",
  "TopicArn" : "<CMR Topic ARN",
  "Message" : "You have chosen to subscribe to the topic <CMR Topic ARN>.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
  "SubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=ConfirmSubscription&TopicArn=<CMR Topic ARN>&Token=2336412f37...5a7640ed",
  "Timestamp" : "2024-11-14T22:31:23.352Z",
  "SignatureVersion" : "1",
  "Signature" : "s5e1851nqyJb...JeoG64ifQFHnw==",
  "SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c64...1136.pem"
}

The message type is "SubscriptionConfirmation" and the message also contains an element called SubscribeURL.  The client either manually or programmatically needs to go to the SubscribeURL site. AWS will then confirm the subscription of your queue to the CMR's topic. The SubscribeURL element only exists in the subscription confirmation message when you need to confirm the subscription.  Any message after that will not contain this element.

If you are doing this manually, the easiest way to look at your queue is to use the AWS SQS console.

This is the AWS response you will get back and will be subscribed to the CMR:

<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<ConfirmSubscriptionResult>
<SubscriptionArn>arn:aws:sns:<CMR AWS region>:<CMR AWS account>:<CMR Topic Name>:<UUID></SubscriptionArn>
</ConfirmSubscriptionResult>
<ResponseMetadata>
<RequestId>53574d28-08be-589e-8fa9-129888f2bbb5</RequestId>
</ResponseMetadata>
</ConfirmSubscriptionResponse>

Once you have acted on this subscription confirmation message you can delete it from your queue.

Now your queue is ready to receive messages from the CMR. 

After a granule has been ingested into the CMR that meets the subscription criteria (collection-concept-id and mode) your queue will receive a message that looks like the following:

{
  "Type" : "Notification",
  "MessageId" : "fbdb230e-befe-56a6-99ff-7cbe3f7e74ec",
  "TopicArn" : "<CMR Topic ARN>",
  "Subject" : "Update Notification",
  "Message" : "{\"concept-id\": \"G1200463969-CMR_ONLY\", \"granule-ur\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01\", \"producer-granule-id\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01.nc\", \"location\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1200463969-CMR_ONLY/16\"}",
  "Timestamp" : "2024-11-14T22:52:48.010Z",
  "SignatureVersion" : "1",
  "Signature" : "VElbKqyRuWNDgI/GB...rjTP+yhjyzdWLomsGA==",
  "SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c6465fa...1136.pem",
  "UnsubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=<Subscription ARN>",
  "MessageAttributes" : {
    "collection-concept-id" : {"Type":"String","Value":"C1200463968-CMR_ONLY"},
    "mode" : {"Type":"String","Value":"Update"}
  }
}

The values for Subject are "New Notification", "Update Notification", or "Delete Notification" and the subject corresponds to a new granule ingest, updated granule, or a deleted granule.

The message element contains a string json with the granules concept id, granule-ur, the producer-granule-id, and the URL to retrieve the concept from the CMR. 

Note that this notification comes after this record has been saved to CMR's database and the location URL will retrieve it from there.  Once CMR has saved the granule (it is the same for any concept ingested) it is put on a queue for so the CMR indexers can index the record to our Elastic Search Engine. If the CMR is being used heavily, there can be a delay getting the concept into our search engine and therefore these records may not be immediately found when using the search/granules API.  It is guaranteed that the record can be retrieved using the URL that is provided in the location element. 

The message attributes are used to route the message to the correct subscription. In the future the subscriber ID will also show up in the message attributes. 

...

subscription. Valid values are either Ingest or Search. Use ingest, if the desire is to be notified when a granule is ingested, updated, or deleted.  Use Search otherwise.

Copy the rest of the elements as they are in the example.

Use the CMR API to create the subscription

https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#subscription

Example CURL command:

curl --location 'https://localhost:8080/ingest/subscriptions/jyna_ingest_subscription_3' \
--header 'Content-Type: application/vnd.nasa.cmr.umm+json' \
--header 'Authorization: ••••••' \
--data-raw '{
    "Name": "Ingest-Subscription-Test",
    "Type": "granule",
    "SubscriberId": "mysubscriberid",
    "CollectionConceptId": "C3261906797-JM_PROV1",
    "EndPoint": "arn:aws:sqs:<region>:<account-id>:<my-queue-name>",
    "Mode": ["New","Update"],
    "Method":"ingest",
    "MetadataSpecification": {
        "URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
        "Name": "UMM-Sub", 
        "Version": "1.1.1"
    } 
}

Response:

<?xml version="1.0" encoding="UTF-8"?>
<result>
    <concept-id>SUB3261906801-JM_PROV1</concept-id>
    <native-id>jyna_ingest_subscription_3</native-id>
    <revision-id>1</revision-id>
</result>

Confirm the Subscription

Once the subscription request has been successfully ingested into the CMR look in your queue either programmatically or through the AWS Console to find a subscription confirmation message.  If a subscription message is not received a CMR operator will have to look at the subscription logs to see what the errors are.  Almost all of the errors are because the queue access policy isn't sufficient from step 1.

An example of a subscription confirmation message is shown below:

{
  "Type" : "SubscriptionConfirmation",
  "MessageId" : "cd7f9a70-e89d-44b0-aa6c-d3be3f3b0316",
  "Token" : "2336412f37...5a7640ed",
  "TopicArn" : "<CMR Topic ARN>",
  "Message" : "You have chosen to subscribe to the topic <CMR Topic ARN>.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
  "SubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=ConfirmSubscription&TopicArn=<CMR Topic ARN>&Token=2336412f37...5a7640ed",
  "Timestamp" : "2024-11-14T22:31:23.352Z",
  "SignatureVersion" : "1",
  "Signature" : "s5e1851nqyJb...JeoG64ifQFHnw==",
  "SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c64...1136.pem"
}

The message type is "SubscriptionConfirmation" and the message also contains an element called SubscribeURL.  The client either manually or programmatically needs to go to the SubscribeURL site. AWS will then confirm the subscription of your queue to the CMR's topic. The SubscribeURL element only exists in the subscription confirmation message when you need to confirm the subscription.  Any message after that will not contain this element.

If you are doing this manually, the easiest way to look at your queue is to use the AWS SQS console.

This is the AWS response you will get back and will be subscribed to the CMR:

<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<ConfirmSubscriptionResult>
<SubscriptionArn>arn:aws:sns:<CMR AWS region>:<CMR AWS account>:<CMR Topic Name>:<UUID></SubscriptionArn>
</ConfirmSubscriptionResult>
<ResponseMetadata>
<RequestId>53574d28-08be-589e-8fa9-129888f2bbb5</RequestId>
</ResponseMetadata>
</ConfirmSubscriptionResponse>

Once you have acted on this subscription confirmation message you can delete it from your queue.

Now your queue is ready to receive messages from the CMR. 

After a granule has been ingested into the CMR that meets the subscription criteria (collection-concept-id and mode) your queue will receive a message that looks like the following:

{
  "Type" : "Notification",
  "MessageId" : "fbdb230e-befe-56a6-99ff-7cbe3f7e74ec",
  "TopicArn" : "<CMR Topic ARN>",
  "Subject" : "Update Notification",
  "Message" : "{\"concept-id\": \"G1200463969-CMR_ONLY\", \"granule-ur\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01\", \"producer-granule-id\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01.nc\", \"location\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1200463969-CMR_ONLY/16\"}",
  "Timestamp" : "2024-11-14T22:52:48.010Z",
  "SignatureVersion" : "1",
  "Signature" : "VElbKqyRuWNDgI/GB...rjTP+yhjyzdWLomsGA==",
  "SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c6465fa...1136.pem",
  "UnsubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=<Subscription ARN>",
  "MessageAttributes" : {
    "collection-concept-id" : {"Type":"String","Value":"C1200463968-CMR_ONLY"},
    "mode" : {"Type":"String","Value":"Update"}
  }
}

The values for Subject are "New Notification", "Update Notification", or "Delete Notification" and the subject corresponds to a new granule ingest, updated granule, or a deleted granule.

The message element contains a string json with the granules concept id, granule-ur, the producer-granule-id, and the URL to retrieve the concept from the CMR. 

Note that this notification comes after this record has been saved to CMR's database and the location URL will retrieve it from there.  Once CMR has saved the granule (it is the same for any concept ingested) it is put on a queue for so the CMR indexers can index the record to our Elastic Search Engine. If the CMR is being used heavily, there can be a delay getting the concept into our search engine and therefore these records may not be immediately found when using the search/granules API.  It is guaranteed that the record can be retrieved using the URL that is provided in the location element. 

The message attributes are used to route the message to the correct subscription. In the future the subscriber ID will also show up in the message attributes. 

Once the message has been processed, it should be deleted to keep your queue clean.

While rare, it is possible for a message to show up twice because of AWS SQS at-least-once delivery. Be sure to design your applications to be idempotent (the software should not be adversely affected when processing the same message more than once).    

Resources

• AWS Documentation using the Console
  https://docs.aws.amazon.com/sns/latest/dg/sns-send-message-to-sqs-cross-account.html
• AWS has great documentation and code examples in many different languages.
  https://github.com/awsdocs/aws-doc-sdk-examples
• UMM-Sub schema can be found: 
  https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/subscription/v1.1.1/umm-sub-json-schema.json
  https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1/umm-sub-json-schema.json
• CMR API documention: 
  https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#subscription