Versions Compared

Old Version 8

changes.mady.by.user Hyokyung Lee

Saved on

compared with

New Version Current

changes.mady.by.user Hyokyung Lee

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Attendees

Discussion items

  • Review Past meeting Action Items

  • BAH AWS Account Status
  • LocalStack Updates
  • Process Driven Strategy Diagram
  • Data Review Process

Past Action items

  •   


New Action items

  •  Hyokyung Lee to move code to SDT repo.
  •  Matthew Tisdale Who is managing the NetCDf OPeNDAP Handler?
  •  How best to advertise/inform users on challenges they will have when the download the datasets


Meeting Notes

Past Actions Items:

  • Matt to send DPG link.

BAH AWS Account Status:

  • Jason gave updates.

LocalStack Updates:

  • Checked in code for Geoserver for GEE, for ARCGIS project. Geolambda Modification which is necessary to build the lambda package.
  • See some updating the lambda itself - lambda.py that calls the GDAL Open
  • Dec. 18 - can does this call - file can be located anywhere in http or https - lambda function can read HDF5 for NetCDF using the call and process it.
  • Has not tried the Virtual interface for S3 - going to test it to day
  • Geolambda works great - python code can read the data out of HTTP - good out of shape to be part fo cumulus
  • Have not converted to TIFF but can read the data - 
  • Noticed Geolambda and lambda need to be up to date - should be the same code checked in
    • latest one that works with virtual server interface that can read HDF5 correctly - lates GDAL release
    • Going to put more GEE code into 2.4 release.

Process Driven Strategy Diagram

  • A bit broader than our A42 work but for the broader enterprise
  • Agree on when we do what transformation and where - to get some consistency among the DAACs, etc.
  • From a community perspective where should we be doing some of these transformation?
  • Where could the DAAC be writing these rules?
  • OPeNDAP is not flexible
    • Users do not have many choices once fixes are made there
  • Great thing about Lambda and GEE based transformation
    • People can write their own functions and ship to their systems
  • Bypass OPeNDAP with GEE capability
  • Overall architecture
  • Seen groups like ArcGIS, Panoply, etc. starting to use OPeNDAP
    • Usually comes down to CF issues at the end of the day
  • The more you do for the clean up on the OPenDAP side, the better.
  • We need to find the overall community to make the OpeNDAP and GDAL handler the same - if one is corrected the other one is fixed?
    • Is this even feasible to keep them aligned?
  • Every product could be a CF compliant - on GEE side just need to maintain the GEE NetCDF Driver
  • Cleaning everything up through OPeNDAP makes it easer for future processing
  • Need to come up with some examples for both options
  • HDF Group modifies the HDF handler in Hyrax OPeNDAP
    • Allow for saving as netCDF-3 or netCDF-4 after subsetting through fileout_netcdf handler.
    • But  HDF Group does not manage the NetCDF handler.
  • OpenDAP Team manages the netCDF Handler - not that active though
  • Workflows and variables - noting if it CF compliant, fit in the workflows

Data Review Process:

CERES SYN1DEG All Sky Surface Shortwave Down (Radiation) - Hourly Data:

https://opendap.larc.nasa.gov/opendap/hyrax/CERES/SYN1deg-1Hour/Terra-Aqua-MODIS_Edition4A/2018/06/CER_SYN1deg-1Hour_Terra-Aqua-MODIS_Edition4A_407406.20180601.hdf?adj_all_sw_dn[0:1:5][0:1:23][0:1:179][0:1:359]

  • Error with grouping trying to import into Panoply
  • Raytheon funding can fix fundamental issue with HDF4 handler
  • If any problem with  handler and document for the community