Meeting Notes
Past Actions Items:
- Matt to send DPG link.
BAH AWS Account Status:
- Jason gave updates.
LocalStack Updates:
- Checked in code for Geoserver for GEE, for ARCGIS project. Geolambda Modification which is necessary to build the lambda package.
- See some updating the lambda itself - lambda.py that calls the GDAL Open
- Dec. 18 - can does this call - file can be located anywhere in http or https - lambda function can read HDF5 for NetCDF using the call and process it.
- Has not tried the Virtual interface for S3 - going to test it to day
- Geolambda works great - python code can read the data out of HTTP - good out of shape to be part fo cumulus
- Have not converted to TIFF but can read the data -
- Noticed Geolambda and lambda need to be up to date - should be the same code checked in
- latest one that works with virtual server interface that can read HDF5 correctly - lates GDAL release
- Going to put more GEE code into 2.4 release.
Process Driven Strategy Diagram
- A bit broader than our A42 work but for the broader enterprise
- Agree on when we do what transformation and where - to get some consistency among the DAACs, etc.
- From a community perspective where should we be doing some of these transformation?
- Where could the DAAC be writing these rules?
- OPeNDAP is not flexible
- Users do not have many choices once fixes are made there
- Great thing about Lambda and GEE based transformation
- People can write their own functions and ship to their systems
- Bypass OPeNDAP with GEE capability
- Overall architecture
- Seen groups like ArcGIS, Panoply, etc. starting to use OPeNDAP
- Usually comes down to CF issues at the end of the day
- The more you do for the clean up on the OPenDAP side, the better.
- We need to find the overall community to make the OpeNDAP and GDAL handler the same - if one is corrected the other one is fixed?
- Is this even feasible to keep them aligned?
- Every product could be a CF compliant - on GEE side just need to maintain the GEE NetCDF Driver
- Cleaning everything up through OPeNDAP makes it easer for future processing
- Need to come up with some examples for both options
- HDF Group modifies the HDF handler in Hyrax OPeNDAP
- Allow for saving as netCDF-3 or netCDF-4 after subsetting through fileout_netcdf handler.
- But HDF Group does not manage the NetCDF handler.
- OpenDAP Team manages the netCDF Handler - not that active though
- Workflows and variables - noting if it CF compliant, fit in the workflows
Data Review Process:
CERES SYN1DEG All Sky Surface Shortwave Down (Radiation) - Hourly Data:
- Error with grouping trying to import into Panoply
- Raytheon funding can fix fundamental issue with HDF4 handler
- If any problem with handler and document for the community
- HDF4 handler page (https://hdfeos.org/zoo/hdf4_handler/index.php) - ASDC products - CERES, MISR, etc. products - if something breaks it is unusual
2 Comments
Matthew Tisdale
I was able to add it using a single variable: https://opendap.larc.nasa.gov/opendap/hyrax/CERES/SYN1deg-1Hour/Terra-Aqua-MODIS_Edition4A/2018/06/CER_SYN1deg-1Hour_Terra-Aqua-MODIS_Edition4A_407406.20180601.hdf?adj_all_sw_dn
It looks like their is a group issue: Error when adding the dataset as a whole:
There was an error opening the dataset. Variable name (cloud layer) must be unique within Group.
Hyokyung Lee
I don't see issue.
https://opendap.larc.nasa.gov/opendap/hyrax/CERES/SYN1deg-1Hour/Terra-Aqua-MODIS_Edition4A/2018/06/CER_SYN1deg-1Hour_Terra-Aqua-MODIS_Edition4A_407406.20180601.hdf.ascii?adj_all_sw_dn[0:1:0][0:1:0][0:1:0][0:1:0]
Try to add .ascii or .dods after .hdf file name.