Introduction

  Esri ArcGIS Enterprise is an amazing all-inclusive turn key solution for NASA Earthdata on cloud.

Highlights

  Through BEDI, GDAL Enhancement for ESDIS (GEE) project identified an issue in handling multi-dimensional dataset and made patches. When the GEE team made a pull request after clearing NOSA, GDAL community reviewed the requests and created a new RFC 75 to generalize our patch work further. The RFC was discussed, approved, and fully implemented in GDAL 3.1.

  The large GDAL community now can easily access and transform arbitrary N-dimensional dataset that can be found in netCDF and HDF. The new GDAL 3.1 multidimensional APIs and tools also supports group hierarchy so users can unambiguously extract and subset data.

  Any NASA HDF data product that has transposed X and Y dimension can benefit from the new GDAL 3.1 capability.  This new capability is already tested through SDT project and proven to work in cloud as well. For example, creating GeoTIFF image from MOPITT 5D dataset in a large (~30G) TERRA FUSION granule on AWS S3 was possible through the new gdalmdimtranslate command line tool.

Recommendation

  Data transformation can be done independently from arcpy or any other Esri software. Using the latest GDAL python or CLI is recommended for transformation if arcpy lacks what the latest GDAL can provide.

  Some arcpy functions need Portal Signin. Therefore, installing ArcGIS Portal and federating it with ArcGIS Server is recommended.

  In AWS environment, web proxy installation can be skipped and native AWS web service front-end can be used.

  If  input data source is not on S3 through CUMULUS, consider using OPeNDAP to subset data. Use NcML to modify some attributes and overwrite variable to make data CF-compliant.

  If ArcGIS Pro on Windows will be used for creating mosaic dataset, working with a large input table is fine.

  If arcpy and MDCS on Linux Windows Server will be used, there's a limit (99 rows) in processing input table for creating mosaic dataset. Input source doesn't matter - CSV or netCDF.

  If MDCS fails, export the input table into CSV and modify header to standardize field names - StdZ, StdTime. Then, try ArcGIS Pro to create a large mosaic dataset using the CSV file.

  If AWS Lambda will be used for data transformation, make sure that data size is small and transfer & transformation doesn't take longer than 15 minutes.

  Creating input table on AWS RDS is recommended for parallel / asynchronous data transformation through Lambda. Make sure that table has unique key to avoid duplicate entries.

  Convert any mosaic dataset into CRF to optimize service performance especially on cloud. It will cost extra for storage but use it to improve user experience.

Web Mapping Service Recommendation

• When using grouped WMS services it is recommended to view an individual sublayer of the WMS to improve performance and minimize requests being sent to the WMS server.
• When viewing multiple WMS services it is recommended to turn off legend for layers not being used to minimize server requests.
• WMS should be used for strictly data visualization. WMS has limited capability when needing to run any analytical processes such as querying, extracting, or binning. Selecting points of interest and popup configuration is not supported with WMS services.
• WMS layers can support styles which can be used as custom parameters for layer requests. Custom parameters are applied to all layers that you specify to be added to the map. It is recommended that if user wants to append different custom parameters to specific layers only, user must add that layer seperately.
• It is recommened when configuring the image service to enable caching on demand so that users can navigate to areas of interest which will then generate cached tiles and increase performance and image quality of the WMS service.
• If WMS service allows to clients to cache tiles locally it is recommended that the clients must manually clear their cache to get any newer tiles from the server. If you will be updating your cache frequently it is best not to use this option.
• Identifying features is the only querying task you can perform on WMS layers, identify will only work on features from the topmost layer in the WMS services drawing hierarchy, it is also recommended to set your identify tolerance levels based on user requirements. This can be done when configuring the service on the WMS server
• OGC WMS coordinate system needs to be the same coordinate system for the basemap in order to display correctly. When creating the image service it is recommended to use the coordinate system required by users as they are unable to change the coordinate system of the WMS

Web Coverage Services Recommendation 

• To configure your WCS service to use external capabilities files, update the WCS service to reference the URL of your capabilities files. You can do this in ArcGIS Server Manager

Future Work

  The following sections list new features that are desirable in near future.

GDAL

• Add input file option for gdalmdimtranslate so that netCDF-4 compatible file with .h5 extension can be opened by netCDF driver instead of HDF5 driver.

ArcGIS

• Support OPeNDAP as source in MRF Proxy.
• Remove errors that only occur on Linux wine but don't appear onAprcGIS Pro in Windows Server.
• Adjust time slider on Portal image layer viewer for "Has Live" data when new data arrives. User needs to configure time slider setting manually to see a new data.