Steps to creating an ingest subscription:
Setup an AWS SQS queue
Setup an AWS SQS queue to your specifications but you will need to add to your Queue Access Policy to allow CMR to send messages to your queue.
When Clients create their own queues make sure to add the following to your Queue Access Policy:
{
"Sid": "<Your Unique ID for this Access Policy>",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": <Your Queue ARN>
}
When you create a queue using the AWS Console: The default access policy is Basic with the Only the queue owner who can send messages to the queue and only the queue owner can receive messages from the queue.
The default JSON policy is
{
"Version": "2012-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": "<Your Account>"
},
"Action": [
"SQS:*"
],
"Resource": <your queue ARN>
}
]
}
Click on Advanced and just before the end bracket put in the figure 1 above so that the following is your access policy
{
"Version": "2012-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": "<Your Account>"
},
"Action": [
"SQS:*"
],
"Resource": <Your Queue ARN>
},
{
"Sid": "<Your Unique ID for this Access Policy",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": <Your Queue ARN>
}
]
}
Setup Collection to Subscribe To
Before you can create an ingest subscription, you will want to choose a pre-existing collection or create a new collection. This is the collection you will subscribe to so that if a granule is created, updated, or deleted you will see it in your queue.
Check You Have Correct Permissions to Your Chosen Collection
In order to subscribe to a collection, you need to make sure you can update and read it. Use the following command to make sure you have permissions.
Creating permissions for first time
curl --location 'https://localhost:8080/access-control/acls' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
"group_permissions": [
{
"permissions": [
"read",
"update",
"delete",
"order"
],
"user_type": "guest"
},
{
"permissions": [
"read",
"update",
"delete",
"order"
],
"user_type": "registered"
}
],
"catalog_item_identity": {
"name": "Guest and Registered Users Collections",
"provider_id": "JM_PROV1",
"granule_applicable": true,
"collection_applicable": true,
"collection_identifier": {
"concept_ids": [
"C3261894786-JM_PROV1", "C3261894785-JM_PROV1", "C3261906797-JM_PROV1"
],
"entry_titles": [
"Collection for CMR-10162_1 v1.18.1", "Collection for CMR-10162_2 v1.18.1", "Collection for CMR-10162_3 v1.18.1"
]
}
}
}'
Response
{
"revision_id": 1,
"concept_id": "ACL3261894787-CMR"
}
Updating permissions for existing ACL
curl --location --request PUT 'https://localhost:8080/access-control/acls/ACL3261894787-CMR' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
"group_permissions": [
{
"user_type": "registered",
"permissions": [
"read",
"update",
"delete",
"order"
]
},
{
"user_type": "guest",
"permissions": [
"read",
"update",
"delete",
"order"
]
}],
"catalog_item_identity": {
"name": "All Granules",
"provider_id": "JM_PROV1",
"granule_applicable": true,
"collection_applicable": true,
"collection_identifier": {
"concept_ids": ["C3261894786-JM_PROV1", "C3261894785-JM_PROV1", "C3261906797-JM_PROV1"],
"entry_titles":["Collection for CMR-10162_1 v1.18.1", "Collection for CMR-10162_2 v1.18.1", "Collection for CMR-10162_3 v1.18.1"]
}
}
}
'
Response:
{
"revision_id": 5,
"concept_id": "ACL3261894787-CMR"
}
Check you have correct permissions
curl --location 'https://localhost:8080/access-control/permissions?user_id=<your-user-id>&concept_id=C3261894785-JM_PROV1'
Response:
{"C3261894785-JM_PROV1":["read","order"]}
Create a Subscription with the CMR
Once your queue is set up then you need to create a new subscription with the CMR
To create a new subscription you will need to use the CMR API and send a subscription request. In the request you will need to send a subscription record that follows the UMM-Sub schema version 1.1.1
Following is an example UMM-Sub record
{"Name": "<The name of your subscription>",
"Type": "granule",
"SubscriberId": "<Your Earthdata Login Name>",
"CollectionConceptId": "C1200463968-CMR_ONLY",
"EndPoint": "arn:aws:sqs:<region>:<account>:<queue-name>",
"Mode": ["New", "Update", "Delete"],
"Method": "ingest",
"MetadataSpecification": {
"URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
"Name": "UMM-Sub",
"Version": "1.1.1"
}
}
The above subscription record that gets sent as the payload with the subscription create/update request is described below.
The subscriber ID must be your Earthdata Login user name.
The collection concept id is the CMR concept id of the collection for which granule ingest notifications are wanted. Make sure that you have read permission to the collection, otherwise the subscription creation will fail. Permission can be checked with the following command: curl "https://cmr.sit.earthdata.nasa.gov/access-control/permissions?user_id=<Earthdata Login Name>&concept_id=<CMR Collection Concept ID>".
The EndPoint is the ARN for the SQS Queue that was created in step 1.
The Mode element describes of what kind of granule ingest notification is wanted.
- Notifications will be sent if
- New - A granule is ingested into the CMR for the first time.
- Update - A granule is ingested into the CMR as an update, meaning it already exists in the CMR, but another ingest of that granule occurs.
- Delete - A granule is deleted from the CMR.
- One, two, or all three options can be used.
The Method element signifies if this is an search type of subscription or an ingest type of subscription. Valid values are either Ingest or Search. Use ingest, if the desire is to be notified when a granule is ingested, updated, or deleted. Use Search otherwise.
Copy the rest of the elements as they are in the example.
Use the CMR API to create the subscription
https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#subscription
Example CURL command:
curl --location 'https://localhost:8080/ingest/subscriptions/jyna_ingest_subscription_3' \
--header 'Content-Type: application/vnd.nasa.cmr.umm+json' \
--header 'Authorization: ••••••' \
--data-raw '{
"Name": "Ingest-Subscription-Test",
"Type": "granule",
"SubscriberId": "mysubscriberid",
"EmailAddress": "myemail",
"CollectionConceptId": "C3261906797-JM_PROV1",
"EndPoint": "arn:aws:sqs:<region>:<account-id>:<my-queue-name>",
"Mode": ["New","Update"],
"Method":"ingest",
"MetadataSpecification": {
"URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
"Name": "UMM-Sub",
"Version": "1.1.1"
}
}
Response:
<?xml version="1.0" encoding="UTF-8"?>
<result>
<concept-id>SUB3261906801-JM_PROV1</concept-id>
<native-id>jyna_ingest_subscription_3</native-id>
<revision-id>1</revision-id>
</result>
Confirm the Subscription
Once the subscription request has been successfully ingested into the CMR look in your queue either programmatically or through the AWS Console to find a subscription confirmation message. If a subscription message is not received a CMR operator will have to look at the subscription logs to see what the errors are. Almost all of the errors are because the queue access policy isn't sufficient from step 1.
An example of a subscription confirmation message is shown below:
{
"Type" : "SubscriptionConfirmation",
"MessageId" : "cd7f9a70-e89d-44b0-aa6c-d3be3f3b0316",
"Token" : "2336412f37...5a7640ed",
"TopicArn" : "<CMR Topic ARN>",
"Message" : "You have chosen to subscribe to the topic <CMR Topic ARN>.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=ConfirmSubscription&TopicArn=<CMR Topic ARN>&Token=2336412f37...5a7640ed",
"Timestamp" : "2024-11-14T22:31:23.352Z",
"SignatureVersion" : "1",
"Signature" : "s5e1851nqyJb...JeoG64ifQFHnw==",
"SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c64...1136.pem"
}
The message type is "SubscriptionConfirmation" and the message also contains an element called SubscribeURL. The client either manually or programmatically needs to go to the SubscribeURL site. AWS will then confirm the subscription of your queue to the CMR's topic. The SubscribeURL element only exists in the subscription confirmation message when you need to confirm the subscription. Any message after that will not contain this element.
If you are doing this manually, the easiest way to look at your queue is to use the AWS SQS console.
This is the AWS response you will get back and will be subscribed to the CMR:
<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/"> <ConfirmSubscriptionResult> <SubscriptionArn>arn:aws:sns:<CMR AWS region>:<CMR AWS account>:<CMR Topic Name>:<UUID></SubscriptionArn> </ConfirmSubscriptionResult> <ResponseMetadata> <RequestId>53574d28-08be-589e-8fa9-129888f2bbb5</RequestId> </ResponseMetadata> </ConfirmSubscriptionResponse>
Once you have acted on this subscription confirmation message you can delete it from your queue.
Now your queue is ready to receive messages from the CMR.
After a granule has been ingested into the CMR that meets the subscription criteria (collection-concept-id and mode) your queue will receive a message that looks like the following:
{
"Type" : "Notification",
"MessageId" : "fbdb230e-befe-56a6-99ff-7cbe3f7e74ec",
"TopicArn" : "<CMR Topic ARN>",
"Subject" : "Update Notification",
"Message" : "{\"concept-id\": \"G1200463969-CMR_ONLY\", \"granule-ur\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01\", \"producer-granule-id\": \"SWOT_L2_HR_PIXC_578_020_221R_20230710T223456_20230710T223506_PIA1_01.nc\", \"location\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1200463969-CMR_ONLY/16\"}",
"Timestamp" : "2024-11-14T22:52:48.010Z",
"SignatureVersion" : "1",
"Signature" : "VElbKqyRuWNDgI/GB...rjTP+yhjyzdWLomsGA==",
"SigningCertURL" : "https://sns.<region>.amazonaws.com/SimpleNotificationService-9c6465fa...1136.pem",
"UnsubscribeURL" : "https://sns.<region>.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=<Subscription ARN>",
"MessageAttributes" : {
"collection-concept-id" : {"Type":"String","Value":"C1200463968-CMR_ONLY"},
"mode" : {"Type":"String","Value":"Update"}
}
}
The values for Subject are "New Notification", "Update Notification", or "Delete Notification" and the subject corresponds to a new granule ingest, updated granule, or a deleted granule.
The message element contains a string json with the granules concept id, granule-ur, the producer-granule-id, and the URL to retrieve the concept from the CMR.
Note that this notification comes after this record has been saved to CMR's database and the location URL will retrieve it from there. Once CMR has saved the granule (it is the same for any concept ingested) it is put on a queue for so the CMR indexers can index the record to our Elastic Search Engine. If the CMR is being used heavily, there can be a delay getting the concept into our search engine and therefore these records may not be immediately found when using the search/granules API. It is guaranteed that the record can be retrieved using the URL that is provided in the location element.
The message attributes are used to route the message to the correct subscription. In the future the subscriber ID will also show up in the message attributes.
Once the message has been processed, it should be deleted to keep your queue clean.