On-Prem To In-Cloud Services Migration

Vision for Legacy HDF (.he4), SMAP and ICESat-2 migration to the Cloud

Feature Summary:

• L1/L2 - Trajectory Subsetter - SMAP special case - Gridded outputs
  • Enable in existing code base for Cloud deployment
  • Gridded outputs to .h5 (Cropped grid, Revive un-merged branch?  Not-GeoTIFF as is current)
  • GeoTIFF formatter (COG, HGA) - chaining
  • Reprojection special case? (HGA?, chaining)
    
    
• L1/L2 - Trajectory Subsetter + general chaining for formats (ICESat-2, SMAP, other?) √
  • Projection - use Swath-Projector (minor updates likely)
    
    • (Potential use cases for Oblique Equal-Area, Oblique Mercator, or Oblique EquiDistance projections)
    • (Not an existing on-prem feature)
  • Reformat: Vector Data: Shapefile, KML, NetCDF, multi-file: ASCII
    - as micro services, lifted mostly from on-prem implementations (or port/rewrite)
    • (Note size constraints, e.g., ICESat-2 files)
    • NetCDF: CF, metadata upgrade from .h5 
    • ShapeFile, KML format output (.shp, .kml, .kmz) 
    • ASCII format output? - rewrite to Python, chaining - single file per variable
      • CSV
        • Choice of delimiters? (Comma, tab)
        • “Row of vars” per time-stamp? (see Tabular ASCII)
        • “Time, Name, Value” format
          or “Time, Value” per variable (name)?
        • GDAL standard?
      • “Tabular-ASCII” - lifted from on-prem, or rewritten to Python
        • Textual layout
        • “Row of vars” per time-stamp
        • Blocked per group and/or frequency (count)
      • OPeNDAP / GDAL (HGA): various ASCII format choices ?
        
        
• L3/L4 - HOSS + chaining (ICESat-2, SMAP other?) √ (some reconfig, perhaps tweaks to code?)
  (Note will be .nc4 output not originally scoped .h5, but hopefully not significant)
  • • ICESat-2 has some notorious quirkiness - Lower-Left orientation, Edge-Aligned dimension scales
    • Possible support for missing dimension-scales - use CF-ish config entry for grid-parameters (SMAP)
      - see: GeoTagging for a Geo-Located Grid Array
  • Reprojection - TBD (I suspect not HGA, rather PyResample based, Data Reprojection Wizard)
  • Reformat - GeoTIFF/COG (HGA?), other standard Cloud formats, Zarr

Some preliminary thoughts:

• Separate support for Projection (Swath-Projection-Gridding) ?
  • Vs. Regridding (same projection)
  • Vs. Reprojection (Equal-Area to Conformal, Cartesian to Polar, others)
• Feature expansion in Cloud Services to match on-prem
  • Less any no-longer-relevant featues (H5-EOS)
  • “Main Services”: HOSS for L3/L4, Trajectory Subsetter for L1/L2
    • For Variable and Spatial/Temporal Subsetting 
    • BBox and Shape/Polygon Region
    • HDF-5, NetCDF-4
  • + Formats, + Projection/Reprojection
  • Alternative for Legacy (HDF4, HDF-EOS-2)?
    • or updates to OPeNDAP ?
• Back-Port to on-prem - Harmony-on-prem within SDPS ?
  • To potentially retire legacy code base for data not yet migrated
  • Open port on EIL for a localized Harmony access?
  • EDSC/CMR reconfiguration

SMAP L1/L2 - 

• Trajectory Subsetter √ (done)
• GeoTIFF special case - minor adapter enablement
  • + some coding refinement to enable GeoTIFF outputs
  • -or- Refactor as separate micro service (SMAP-to-GeoTIFF)?
  • -or- Gridding as an option within HDF-5 format (SMAP-to-Grid)
    • We have this functionality in our archives if not immediately accessible.
    • + GeoTIFF formatter (HGA) √ (preferred, I think)
• Reprojection (chaining): GeoTIFF/COG => HGA, other?
• .he5 ,   NetCDF format output (.nc4) ? 
  • micro services - lifted from on-prem is possible
  • from GeoTIFF, w/ aggregation 
  • -or- from HDF-5 w/ Grid & Trajectory handling?
    • again, available from on-prem if useful.
• ShapeFile format output (.shp) - as a micro service, 
  • lifted from on-prem is an option
• ASCII format output? - rewrite to Python, chaining
  • CSV
    • Choice of delimiters? (Comma, tab)
    • “Row of vars” per time-stamp? (see Tabular ASCII)
    • “Time, Name, Value” format
      or “Time, Value” per variable?
    • GDAL standard?
  • “Tabular-ASCII” - lifted from on-prem, or rewritten
    • Textual layout
    • “Row of vars” per time-stamp
    • Blocked per group and/or frequency (count)
  • OPeNDAP / GDAL (HGA) various ASCII format choices ?

Re. ICESat-2 L1/L2 - multiple divergent data products

L1/L2 (Trajectory) -

• Trajectory Subsetter √ (done)
• No Gridding, Reprojection nor NetCDF format requirements
• ShapeFile (.shp) and ASCII formats (see above), but with size constraints

SMAP L3/L4 - including 3D / Multi-Dimension (?)

• HOSS Updates for projected-grid data. √ (done)
• HOSS - with configuration file entries, minor updates?
• SMAP is notoriously lacking in CF content
• NetCDF format output (.nc4)
• default for HOSS (common .h5 to .nc4 “fixer”/Formatter - built-in to OPeNDAP!)
• An issue if HDF-5 is desired - close compliancy with source data, possibly just a file extension change?
• GeoTIFF / COG format - HGA chaining? (I think so)
• Regridding/Reprojection feature (TBD)
• ASCII format? (which ASCII)
• OPeNDAP has an option, I believe
• Other choices, HGA, chaining?

ICESat-2 - ATLAS - L3/L4 (Gridded, mix of projections, some subtle grid issues) -

• HOSS - with configuration file entries, minor updates?
• ICESat-2 has some notorious quirkiness - Lower-Left orientation, Edge-Aligned dimension scales
• NetCDF - default for HOSS (common .h5 to .nc4 “fixer”/Formatter - built-in to OPeNDAP!)
• To GeoTIFF / COG - HGA chaining?
• Regridding/Reprojection feature (TBD)

HDF-EOS-4 legacy data - multiple slightly divergent data products

• HOSS - with configuration file entries, minor updates?
• Assuming OPeNDAP updates - not sure how feasible
• HDF4 - NetCDF pre-processor.
• Note .nc4 is default output for HOSS (common .h5 to .nc4 “fixer”/Formatter - built-in to OPeNDAP!)
  so processed-outputs compliancy with source data may be an issue.
• ?? HEG rewrite ?? (Unlikely, except possibly - to capture unique reprojection/interpolation handling)
• Regridding/Reprojection feature (TBD)
• GeoTIFF - HGA