Child pages
  • OPeNDAP, OpenSearch, & the CMR
Skip to end of metadata
Go to start of metadata

Overview

One of the items on the BEDI list to increase data access involved exposing more products via OPeNDAP. We are working on documenting current usage of these access URLs in Discovery Metadata and developing recommendations for exposing it in our search APIs (CMR and OpenSearch flavors) and interpreting in our web clients (Earthdata SearchGCMDReverb)

 

This is the first attempt at documenting the needs, the existing usage and recommendations, we want your feedback, so please feel free to comment below.

OpenSearch

Exposing OPeNDAP links in OpenSearch

Currently, CMR OpenSearch has no provision for exposing OPeNDAP links to the user.

We have been asked by GSFC to expose this information  CWIC-205 - Getting issue details... STATUS

SDPS are working on an OPeNDAP solution with NCR 8052122

In order to facilitate end user access to data it would be convenient to export OpenDAP URLs to ECHO so that the end user would have a choice of download not only the whole granule, but an OpenDAP specified subset of it.  We plan to deploy OpenDAP at the DAACs before 8.3.3, so it would be useful if we could give the DAACs the ability to export OpenDAP URLs to ECHO so they could make use of it.

Drivers

  1. CMR OpenSearch should present OPeNDAP urls compliant to ESIP Best Practices (see below)
  2. CMR search results in ATOM format need to facilitate that
  3. Providers need to supply enough information to do (1) and (2)
    1. SDPS
    2. Others
  4. Other formats?

Discussion topics

  1. How are SDPS exposing OPeNDAP url information to ECHO. Answer: SDPS are able to present the following, utilizing the OnlineResourceURL element -> url + type=OPeNDAP + mime-type of the originating resource (probably HDF)
  2. Is the collection and granule schema able to support the three elements required to comply with ESIP Best Practices?  Not unless we export a url for each explicit format.
  3. How do we expose that in search for
    1. ECHO10 - OnlineResourceURL
    2. ISO - unknown
    3. ATOM - link element with some discriminator saying 'this is an OPeNDAP url'
    4. ATOM OpenSearch - see Best Practices

The ESIP Best Practice

The ESIP OpenSearch Best Practices documents a means to expose these URLs in a meaningful way. It is detailed below:

7.2.7 Characterizing OPeNDAP Link Elements (Optional)

In addition to these basic link types, some data providers may offer a variety of data access points to the same data.  An example of this is data providers that offer data via the OPeNDAP standard, which itself has several different return types:  specific forms of metadata or data in varying formats.  In order to distinguish OPeNDAP data access points from simple FTP or HTTP URLs, as well as from each other, an xlink-based scheme has been adapted to identify the type of OPeNDAP access point.  An OPeNDAP service link is identified by a <link> attribute of the type “xlink:role”.  In this case, specifically, the xlink role and type attributes are given as:

xlink:role="http://xml.opendap.org/dap/dap2.xsd" xlink:type="simple"

In addition, the combination of the rel attribute, mime-type (i.e., type) attribute and xlink:arcrole attribute are used to distinguish the different types of OPeNDAP response:

 

Server Response Type

rel

type

xlink:arcrole

HTML

describedby

text/html

#html

info

describedby

text/html

#info

DDX

describedby

application/xml

#ddx

RDF

describedby

application/rdf+xml

#rdf

ASCII

enclosure

text/plain

#ascii

NetCDF

enclosure

application/x-netcdf

#netcdf

Note that a search engine is not required to return all of these responses when identifying OPeNDAP links, just the ones that it (a) supports and (b) wishes to advertise to clients.

It is understood that inclusion of these xlink elements can cause a Discovery response to fail validation by some Atom feed validators, but this has been traded off for the extra information this construct provides to client programs about the options for obtaining data and metadata from the provider.

The drivers behind this, at first glance, verbose means of describing an OPeNDAP service are as follows:

  • OpenSearch wishes to expose all possible formats to a client that doesn't understand OPeNDAP.

An OPeNDAP-savvy client could take a URL to a resource and, knowing that it was an OPeNDAP URL, infer that appending various extension would yield meaningful results.

A client that doesn't understand OPeNDAP would not be able to exploit a net-cdf version of this resource without being explicitly told that.

So ESIP recommends that all formats of the resource be explicitly called out. For example,

the OPeNDAP resource

http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz

Would be represented in an OpenSearch result as:

<entry>
...
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.html" 
    hreflang="en-US" title="(OPeNDAP)" rel="describedby" type='text/html' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#html' />
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.info" 
    hreflang="en-US" title="(OPeNDAP)" rel="describedby" type='text/html' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#info' />
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.ddx" 
    hreflang="en-US" title="(OPeNDAP)" rel="describedby" type='application/xml' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#ddx' />
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.rdf" 
    hreflang="en-US" title="(OPeNDAP)" rel="describedby" type='application/rdf+xml' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#rdf' />
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.cdf" 
    hreflang="en-US" title="(OPeNDAP)" rel="enclosure" type='application/x-netcdf' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#netcdf' />
<link href="http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.ascii" 
    hreflang="en-US" title="(OPeNDAP)" rel="enclosure" type='text/plain' 
    xmlns:xlink="http://www.w3.org/1999/xlink" xlink:role="http://xml.opendap.org/dap/dap2.xsd" 
    xlink:type="simple" xlink:arcrole='#ascii' />
...
</entry>
  • OpenSearch wishes to convey, explicitly, the capabilities of an OPeNDAP service since not all versions of OPeNDAP support all formats

Meaning the above result represents an up-to-date OPeNDAP server. Other servers may not support net-cdf fo

OpenSearch and CMR Metadata

CMR OpenSearch decorates a CMR ATOM response, at the collection and granule levels, with OpenSearch specific elements and augmentations.

In order to expose OPeNDAP url information as per the ESIP recommendations we need to know the following,

  • that a resource falls in the OPeNDAP category
  • What mime type it is
  • what subset, if any, of OPeNDAP formats it supports

Currently, the ECHO10 result format will identify a url of type 'OPeNDAP' as follows,

curl https://cmr.earthdata.nasa.gov:443/search/concepts/G550016-GHRC

<OnlineResource>
	<URL>http://ghrc.nsstc.nasa.gov/opendap/ssmi/f14/monthly/</URL>
	<Type>OPeNDAP</Type>
</OnlineResource>

That same resource in ATOM gives (note CMR is still using the 'old' relations to describe a link)

curl https://cmr.earthdata.nasa.gov:443/search/concepts/G550016-GHRC.atom

<link href="http://ghrc.nsstc.nasa.gov/opendap/ssmi/f14/monthly/" hreflang="en-US" title="(OPeNDAP)" rel="http://esipfed.org/ns/fedsearch/1.1/data#"></link> 

In order for CMR OpenSearch to expose this as an OPeNDAP resource we would need from CMR, at a minimum, some means of identifying it as an OPeNDAP url.

 

If we wanted to convey what formats that OPeNDAP implementation supports we have a bigger problem. That information would need to be obtained from the provider.

CMR

Metadata URL Support

UMM-C

In the Unified Metadata Model for Collections, all URLs are mapped into the RelatedURL element. This element looks almost identical to the existing DIF Related_URL.

Here is a sample mapping from UMM-C to the DIF and ECHO 10 elements:

UMM-C Element SpecificationDIF MappingECHO 10 Online AccessECHO 10 Online Resource

RelatedUrl/ContentType/Type

RelatedUrl/ContentType/Type-OnlineResource/Type*
RelatedUrl/ContentType/SubtypeRelatedUrl/ContentType/Subtype-OnlineResource/Type* 
RelatedUrl/URLRelatedUrl/URLOnlineAccessURL/URLOnlineResource/URL
RelatedUrl/DescriptionRelatedUrl/DescriptionOnlineAccessURL/Description OnlineResource/Description
RelatedUrl/MimeTypeRelatedUrl/MimeTypeOnlineAccessURL/MimeTypeOnlineResource/MimeType

* The ECHO OnlineResource/Type element is used by some DAACs to provide Subtype via a colon delimited string (e.g. 'GET DATA : OPENDAP DATA (DODS)').

ECHO

via Online Access URL

 

via Online Resource

 

DIF 9.9

via Related_URL

Current Usage

Granule

ECHO 10 Format

GHRC Implementaion
(https://cmr.earthdata.nasa.gov:443/search/concepts/G550016-GHRC)
<OnlineResource>
	<URL>http://ghrc.nsstc.nasa.gov/opendap/ssmi/f14/monthly/</URL>
	<Type>OPeNDAP</Type>
</OnlineResource>
PO.DAAC Implementaion
(https://cmr.earthdata.nasa.gov:443/search/concepts/G2921045-PODAAC)
<OnlineAccessURL>
	<URL>http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz.html</URL>	
	<URLDescription>The OPENDAP location for the granule.</URLDescription>
</OnlineAccessURL>
SDPS Implementaion
(http://testbed.echo.nasa.gov/reverb/granules/G1000021178-EDF_OPS)
<OnlineAccessURL>
  <URL>http://f5eil01v.edn.ecs.nasa.gov:24336/opendap//FS1/MOST/MOD29P1D.005/2011.01.01/MOD29P1D.A2011001.h13v27.005.2011002082448.hdf.html</URL>
  <MimeType>text/html</MimeType>
</OnlineAccessURL>

Collection

ECHO 10 Format

GES DISC Implementaion (Collection Level)
(https://cmr.earthdata.nasa.gov:443/search/concepts/C186901238-GSFCS4PA)
<AdditionalAttributes>
  <AdditionalAttribute>
    <Name>OPeNDAPServer</Name>
    <DataType>STRING</DataType>
    <Description>Access the AIRS L2 Cloud-Cleared Infrared (IR) Radiances Product through OPeNDAP.</Description>      
    <Value>http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRI2CCF.005/contents.html</Value>
  </AdditionalAttribute>
</AdditionalAttributes>
Alternate GES DISC Implementaion (Collection Level)
(https://cmr.earthdata.nasa.gov:443/search/concepts/C114367117-GSFCS4PA)
<OnlineResource>
  <URL>http://disc2.nascom.nasa.gov/opendap/TRMM_L3/TRMM_CSH/</URL>
  <Description>OPeNDAP Access dataset TRMM Monthly 0.5 degree x 0.5 degree Convective/Stratiform Heating</Description>
  <Type>GET DATA : OPENDAP DATA (DODS)</Type>
</OnlineResource>

DIF Format

GES DISC Implementation
(http://gcmd.gsfc.nasa.gov/KeywordSearch/Metadata.do?Portal=GCMD&MetadataType=0&MetadataView=Full&KeywordPath=&EntryId=GES_DISC_TRMM_CSH_V6)
<Related_URL>
    <URL_Content_Type uuid="2f1f3f44-61d5-48db-a8ac-06e3f777bf35">
        <Type>GET DATA</Type>
        <Subtype>OPENDAP DATA (DODS)</Subtype>
    </URL_Content_Type>
    <URL>http://disc2.nascom.nasa.gov/opendap/TRMM_L3/TRMM_CSH/</URL>
    <Description> 
		OPeNDAP Access dataset TRMM Monthly 0.5 degree x 0.5 degree Convective/Stratiform Heating
	</Description>
</Related_URL>

Recommendations

Granule 

ECHO 10 Format

Providers will describe their OPeNDAP urls, at a granule level, as follows

GHRC Implementaion
<OnlineResource>
	<URL>http://podaac-opendap.jpl.nasa.gov/opendap/allData/coastal_alt/preview/L4/OSU_COAS/weekly/osu_cioss_weekly_msla_geovel_1992_v1.nc.gz</URL>
	<Type>GET DATA : OPENDAP DATA (DODS)</Type>
    <MimeType>application/x-gzip</MimeType> 
</OnlineResource>

Advantages

Given this information a client could, understanding it was an OPeNDAP resource, invoke services on that granule by changing the extension and/or query string to the url. For example,

 

Collection

ECHO 10 Format

Providers may describe collections of granules with the same OPeNDAP capabilities at the collection level as follows,

GES DISC Implementaion (Collection Level)
<OnlineResource>
    <URL>http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRI2CCF.005/contents</URL>
	<Type>GET DATA : OPENDAP DATA (DODS)</Type>
</OnlineResource>

 

DIF Format

Current usage by GES DISC is good. We recommend sticking with this.

 

GES DISC Implementation
(http://gcmd.gsfc.nasa.gov/KeywordSearch/Metadata.do?Portal=GCMD&MetadataType=0&MetadataView=Full&KeywordPath=&EntryId=GES_DISC_TRMM_CSH_V6)
<Related_URL>
    <URL_Content_Type uuid="2f1f3f44-61d5-48db-a8ac-06e3f777bf35">
        <Type>GET DATA</Type>
        <Subtype>OPENDAP DATA (DODS)</Subtype>
    </URL_Content_Type>
    <URL>http://disc2.nascom.nasa.gov/opendap/TRMM_L3/TRMM_CSH/</URL>
    <Description> 
		OPeNDAP Access dataset TRMM Monthly 0.5 degree x 0.5 degree Convective/Stratiform Heating
	</Description>
</Related_URL>


Advantages

Given this information a client could, understanding it was an OPeNDAP resource, invoke services on that granule by changing the extension and/or query string to the url. For example,

Using this technique a client could infer the properties of many granule resources at a time without having to grab a ddx file for each granule.

  • No labels

1 Comment

    • Besides the online url links, SDPS also exposes Opendap through their egi/esi interface and supports all teh formats opendap supports
    • SDPS also uses opendap as a post processing tool for netcdf-3 and ascii formats after heg or other subsetter processing