Highlights

  Through BEDI, GDAL Enhancement for ESDIS (GEE) project identified an issue in handling multi-dimensional dataset and made patches. When the GEE team made a pull request after clearing NOSA, GDAL community reviewed the requests and created a new RFC 75 to generalize our patch work further. The RFC was discussed, approved, and fully implemented in GDAL 3.1.

  The large GDAL community now can easily access and transform arbitrary N-dimensional dataset that can be found in netCDF and HDF. The new GDAL 3.1 multidimensional APIs and tools also supports group hierarchy so users can unambiguously extract and subset data.

  Any NASA HDF data product that has transposed X and Y dimension can benefit from the new GDAL 3.1 capability.  This new capability is already tested through SDT project and proven to work in cloud as well. For example, creating GeoTIFF image from MOPITT 5D dataset in a large (~30G) TERRA FUSION granule on AWS S3 was possible through the new gdalmdimtranslate command line tool.

Recommendation

  Data transformation can be done independently from arcpy or any other Esri software. Using the latest GDAL python or CLI is recommended for transformation if arcpy lacks what the latest GDAL can provide.

  Some arcpy functions need Portal Signin. Therefore, installing ArcGIS Portal and federating it with ArcGIS Server is recommended.

  In AWS environment, web proxy installation can be skipped and native AWS web service front-end can be used.

  If  input data source is not on S3 through CUMULUS, consider using OPeNDAP to subset data. Use NcML to modify some attributes and overwrite variable to make data CF-compliant.

  If ArcGIS Pro on Windows will be used for creating mosaic dataset, working with a large input table is fine.

  If arcpy and MDCS on Linux Windows Server will be used, there's a limit (99 files) in processing input table for creating mosaic dataset.

  If MDCS fails, export the input table into CSV and modify header to standardize field names - StdZ, StdTime. Then, try ArcGIS Pro to create a large mosaic dataset using the CSV file.

  If AWS Lambda will be used for data transformation, make sure that data size is small and transfer & transformation doesn't take longer than 15 minutes.

  Creating input table on AWS RDS is recommended for parallel / asynchronous data transformation through Lambda. Make sure that table has unique key to avoid duplicate entries.

  Convert any mosaic dataset into CRF to optimize service performance especially on cloud. It will cost extra for storage but use it to improve user experience.