Page History

Element Description

The direct distribution information main element allows data providers to provide users information on getting direct access to data products that are stored in the Amazon Web Service (AWS) S3 buckets when they are initially looking at a collection. The end users get information such as the S3 credentials end point, a credential documentation URL, as well as bucket prefix names, and an AWS region.

Best Practices

There are multiple sub-elements that comprise Archive and Direct Distribution Information for Collections:

FormatRegion: Defines a single format for an archival artifact (e.g., binary, jpeg, HDF5, HDF-EOS5, GeoTIFF). It is strongly recommended that the format be selected from the GCMD Granule Data Format vocabulary list in order to ensure consistent use of data formats across the CMR.  

Format Type: Allows the provider to define if the archival artifact's format is in its native format or another supported format

Format Description: Allows the data provider to provide added information about the provided Format.

Average File Size: An approximate average of the size of the the archivable item, which gives users an idea of the magnitude of each archivable file

Average File Size Unit: Unit of measure for the average file size (e.g., KB, MB, TB)

Total Collection File Size: An approximate total size of the archivable items, which gives users an idea of the magnitude of all the archivable files combined

Total Collection File Size Unit: Unit of measure for the total collection file size (e.g., KB, MB, TB)

Total Collection File Size Begin Date: The date that data collection began for the collection, which should be in the following format: yyyy-MM-ddTHH:mm:ssZ

Description: Enables the provider to provide more information about the archivable item 

Media: Defines how the distributable material can be obtained by an end user (e.g., CD-ROM, hard drives, online, etc.)

Fees: Provide the price an end user will need to pay in order to obtain the distributable material

Each of these elements are available under 'FileArchiveInformation' and 'FileDistributionInformation'. FileArchiveInformation describes the data in its archived format. FileDistributionInformation describes the data as distributed by a data provider; it can be the original format in which the data are archived, or can be used to specify when the data is distributed in a format other that its original format.    

Examples:

Format: "HDF-EOS5"

Format Type: "Native"

Format Description: "This is an EOSDIS specific format and it is also available in ASCII."

Average File Size: "250"

Average File Size Unit: "KB"

Total Collection File Size: "25"

Total Collect File Size Unit: "GB"

Total Collection File Size Begin Date: "2018-04-01T00:00:00Z"

ArchiveAndDistributionInformation/ FileArchiveInformation/

Native

Supported

ArchiveAndDistributionInformation/ FileArchiveInformation/ AverageFileSizeUnit

KB

MB

GB

TB

PB

NA

n/a

ArchiveAndDistributionInformation/ FileArchiveInformation/ TotalCollectionFileSize

ArchiveAndDistributionInformation/ FileArchiveInformation/ TotalCollectionFileSizeUnit

KB

MB

GB

TB

PB

NA

ArchiveAndDistributionInformation/ FileArchiveInformation/ TotalCollectionFileSizeBeginDate

ArchiveAndDistributionInformation/ FileArchiveInformation/ Description

1 - 1024 characters

ArchiveAndDistributionInformation/

FileDistributionInformation/Format

ArchiveAndDistributionInformation/

FileDistributionInformation/ FormatType

Native

Supported

ArchiveAndDistributionInformation/

FileDistributionInformation/ FormatDescription

ArchiveAndDistributionInformation/ FileDistributionInformation/

1 - 80 characters

ArchiveAndDistributionInformation/ FileDistributionInformation/ AverageFileSizeUnit

KB

MB

GB

TB

PB

NA

ArchiveAndDistributionInformation/ FileDistributionInformation/ TotalCollectionFileSize

ArchiveAndDistributionInformation/ FileDistributionInformation/ TotalCollectionFileSizeUnit

KB

MB

GB

TB

PB

NA

ArchiveAndDistributionInformation/ FileDistributionInformation/ TotalCollectionFileSizeBeginDate

ArchiveAndDistributionInformation/ FileDistributionInformation/ Description

1 - 1024 characters

Fees

1 - 255 characters

Metadata Validation and QA/QC

All metadata entering the CMR goes through the below process to ensure metadata quality requirements are met. All records undergo CMR validation before entering the system. The process of QA/QC is slightly different for NASA and non-NASA data providers. Non-NASA providers include interagency and international data providers and are referred to as the International Directory Network (IDN).

Please see the expandable sections below for flowchart details.

ARC Priority Matrix

This element is categorized as highest priority when:

• No data format is provided
• The data format provided is incorrect

This element is categorized as medium priority when:

• The data format provided is correct, but could be more specific 
  • E.g. Providing 'NetCDF' when the format should be 'NetCDF-4'
  • E.g. Providing 'ASCII' when the data format should be 'CSV'

This element is categorized as low priority when:

• The data format provided is correct, however, it does not exactly match its entry in the GCMD Granule Data Format vocabulary list. 
  • E.g. providing 'netcdf4' versus 'NetCDF-4'
  • E.g. providing 'geo-tiff' versus 'GeoTIFF'

ARC Automated Checks

• TBD

Dialect Mappings

the possible values for the Amazon Web Service US Regions where the data product resides.  The current valid values are: us-east-1,us-east-2,us-west-1, and us-west-2.

S3BucketAndObjectPrefixNames: Defines the possible values for the Amazon Web Service US S3 bucket and/or object prefix names.

S3CredentialsAPIEndpoint: Defines the URL where the credentials are stored.

S3CredentialsAPIDocumentationURL: Defines the URL where the credential documentation are stored.

Each of these elements describes the information that is necessary to pull out data products that are stored in the AWS cloud using a s3 URL that is located inside each granule's  (data product file's) metadata.

Examples:

Region: "us-east-1"

S3BucketAndObjectPrefixNames: "Bucket 1"

S3CredentialsAPIEndpoint: "https://some-data-product-credentials-api.com/api.html"

S3CredentialsAPIDocumentationURL: "https://some-data-product-credentials-api.com/docs.html"

Metadata Validation and QA/QC

All metadata entering the CMR goes through the below process to ensure metadata quality requirements are met. All records undergo CMR validation before entering the system. The process of QA/QC is slightly different for NASA and non-NASA data providers. Non-NASA providers include interagency and international data providers and are referred to as the International Directory Network (IDN).

Please see the expandable sections below for flowchart details.

Dialect Mappings

<gmi:MI_Metadata>
  ...
 <gmd:distributionInfo>
                <gmd:MD_Distribution>
                    <gmd:distributor xlink:href="DirectDistributionInformation">
                        <gmd:MD_Distributor>
                            <gmd:distributorContact gco:nilReason="inapplicable"/>
                            <gmd:distributionOrderProcess>
                     

<Distribution>
  <Distribution_Media>Online Archive></Distribution_Media>
  <Average_Granule_Size>231</Average_Granule_Size> **Note: In DIF 10.2 this would appear as: <Distribution_Size>231 MB per file</Distribution_Size>
  <Average_Granule_Size_Unit>MB</Average_Granule_Size_Unit>
  <Distribution_Format>HDF-EOS5</Distribution_Format>
  <Distribution_Format_Type>Native</Distribution_Format_Type>  **Note: The Distribution_Format_Type field is not available in DIF 10.2
  <Fees>Free</Fees>
</Distribution>

"ArchiveAndDistributionInformation": {
    "FileDistributionInformation": [
      {
        "FormatType": "Native",
        "AverageFileSize": 231,
        "Fees": "Free",
        "Format": "HDF-EOS",
        "AverageFileSizeUnit": "MB",
        "Media": [
          "Online Archive" <gmd:MD_StandardOrderProcess>
        ]
            }
    ]
  },

ECHO 10

Only 2 UMM-C Archive and Distribution Information sub-elements map to ECHO 10, Data Format and Fees, and neither element is required. 

N/A

N/A

<DataFormat>HDF-EOS5</DataFormat>
<Price>0.0</Price>

"ArchiveAndDistributionInformation": {
  "FileDistributionInformation": [
    {
      "FormatType": "Native",
      "Fees": "0.0",            <gmd:orderingInstructions>
                                    <gco:CharacterString>Region: us-east-2 S3BucketAndObjectPrefixNames: TestBucketOrObjectPrefix</gco:CharacterString>
                                    </gmd:orderingInstructions>
                                </gmd:MD_StandardOrderProcess>
                            </gmd:distributionOrderProcess>
      "Format": "HDF-EOS5"
    }
  ]
},                 <gmd:distributorTransferOptions xlink:href="DirectDistributionInformation_S3CredentialsAPIEndpoint">
                                <gmd:MD_DigitalTransferOptions>
                                    <gmd:onLine>
                                    <gmd:CI_OnlineResource>
                                    <gmd:linkage>
                                    <gmd:URL>https://DAACCredentialEndpoint.org</gmd:URL>
                                    </gmd:linkage>
                                    <gmd:description>
                                    <gco:CharacterString>The S3 credentials API endpoint.</gco:CharacterString>
                                    </gmd:description>
                                    </gmd:CI_OnlineResource>
                                    </gmd:onLine>
                                </gmd:MD_DigitalTransferOptions>
                            </gmd:distributorTransferOptions>
                            <gmd:distributorTransferOptions xlink:href="DirectDistributionInformation_S3CredentialsAPIDocumentationURL">
                                <gmd:MD_DigitalTransferOptions>
                                    <gmd:onLine>
                                    <gmd:CI_OnlineResource>
                                    <gmd:linkage>
                                    <gmd:URL>https://DAACCredentialDocumentation.org</gmd:URL>
                                    </gmd:linkage>
                                    <gmd:description>
                                    <gco:CharacterString>The S3 credentials API Documentation URL.</gco:CharacterString>
                                    </gmd:description>
                                    </gmd:CI_OnlineResource>
                                    </gmd:onLine>
                                </gmd:MD_DigitalTransferOptions>
                            </gmd:distributorTransferOptions>
                        </gmd:MD_Distributor>
                    </gmd:distributor>
                  ...

"DirectDistributionInformation" : {
  "Region" : "us-east-2",
  "S3BucketAndObjectPrefixNames" : [ "TestBucketOrObjectPrefix" ],
  "S3CredentialsAPIEndpoint" : "https://DAACCredentialEndpoint.org",
  "S3CredentialsAPIDocumentationURL" : "https://DAACCredentialDocumentation.org"
},          

<gmd:DS_Series>
  <gmd:seriesMetadata>
    <gmi:MI_Metadata>
      ...
      <gmd:distributionInfo>
                <gmd:MD_Distribution>
                    <gmd:distributor xlink:href="DirectDistributionInformation">
    

<gmi:MI_Metadata>
  ...
    <gmd:distributionFormat xlink:href="FileDistributionInformation_<Block Number>"/>
        <gmd:MD_Format>
            <gmd:specification>
                <gco:CharacterString>TotalCollectionFileSizeBeginDate</gco:CharacterString>
              </gmd:code>
              <gmd:codeSpace>
                <gco:CharacterString>2018-04-01T00:00:00Z</gco:CharacterString>
              </gmd:codeSpace>
            <gmd:specification>
          <gmd:MD_Format>
      <gmd:distributionFormat xlink:href="FileDistributionInformation_<Block Number>"/>  
                

ArchiveandDistributionInformation: [ 
  { 
     AverageFileSize: "250", 
  },
  { 
     AverageFileSizeUnit: "KB", 
  },
  { 
     TotalCollectionFileSizeBeginDate: "2018-04-01T00:00:00Z", 
  } 
],
                

ISO 19115-2 SMAP

Archive and Distribution Information does not currently map to ISO 19115-2 SMAP.

N/A

N/A

UMM Migration

None

UMM Migration

None

History

UMM Versioning

History

UMM Versioning

ARC Documentation