The scope of this wiki page is to capture developer documentation for the Harmony GDAL Adapter (HGA). This page should help us understand the features of HGA, including the viable input and output types, along with available transformations. This page could also include known issues and anticipated development efforts for the service.
HGA is a Harmony backend service offering a number of transformation features, which leverage the common geospatial tool GDAL. This service has been managed and developed by several teams since its original inception, including the Harmony core team, the Alaska Satellite Facility (ASF), the Physical Oceanography DAAC (PO.DAAC) and now the Data Services team. The source code currently exists in an open-source repository, within the NASA GitHub organisation.
Refer to regression tests for a good start here, the bullet point list for
The steps below are a rough description of the path taken through transform.py, the primary source of service code for HGA.
General note: GDAL commands are mostly constructed as string that would be used in a shell or other terminal and then executed using Python's subprocess.check_output
function (see HarmonyAdapter.cmd
). It would be great to use the Python GDAL packages that allow these transformations to be written in Python-native code. (Some operations, such as masking for a polygon spatial subset, do adopt this Python-based approach)
HarmonyBaseAdapter
from harmony-service-lib-py.HarmonyAdapter.process_item
method.HarmonyAdapter.process_geotiff
, HarmonyAdapter.process_netcdf
or HarmonyAdapter.process_zip
is called.HarmonyAdapter.process_zip
extracts files from the zip file and then calls either HarmonyAdapter.process_geotiff
or HarmonyAdapter.process_netcdf
.HarmonyAdapter.process_geotiff
calls HarmonyAdapter.combin_transfer
, which in turn calls class methods (in this order): HarmonyAdapter.subset
, HarmonyAdapter.
reproject, HarmonyAdapter.resize
, HarmonyAdapter.recolor
HarmonyAdapter.process_netcdf
uses HarmonyAdapter.nc2tiff
to convert the contents of the NetCDF-4 into GeoTIFF files using gdal_translate
. Then it calls HarmonyAdapter.combin_transfer
.HarmonyAdapter.combin_transfer
in turn calls four processing methods that perform various subsetting, projection, gridding and colouring operations on the input:HarmonyAdapter.subset
:HarmonyAdapter.subset2
, which uses: gdal_translate
to extract a bounding box, or the minimally encompassing bounding box for a shape file. Masking is then applied in the case of a shape file.HarmonyAdapter.varsubset
, and again uses gdal_translate
.HarmonyAdapter.reproject
:gdalwarp
command that uses request parameters: crs, interpolation, scaleSize, scaleExtent.HarmonyAdapter.resize
:gdal_translate
command that uses request parameters: interpolation (defaults to "bilinear"), width and height.HarmonyAdapter.recolor
:gdaldem
command.The HarmonyAdapter.add_to_results
function will compile the output files, and stack if appropriate. The stacking occurs using HarmonyAdapter.stack_multi_file_with_metadata
(the logic in these method might relate to issues with DAS-1479).HarmonyAdapter.process_item
, the output STAC item is updated and the outputs are staged to the Harmony S3 bucket.subprocess
architecture.recolor
gets called for all variables (via combin_transfer
from this loop or this loop). But... there's only ever one call to stage the output (here). There's a separate call to stage the .wld
file, but nothing to loop through multiple outputs."