Page History
| Table of Contents |
|---|
Introduction
Geolambda is a docker-based lambda function package builder.
...
FROM lambdalinux/baseimage-amzn:2017.03-004
Problem
docker-compose hangs during geos package compilation on Mojave:
...
| Code Block |
|---|
Runtime failure is likely ! ERROR 4: `/vsicurl/https://gamma.hdfgroup.org/ftp/pub/outgoing/NASAHDF/CER_ES4_TRMM-PFM_Edition2_019018.19808.hdf' not recognized as a supported file format. |
HDF5 Open Failure for GDAL version 2.3.1
GDAL 2.3.1 can't open MOP03T. It seems that GDAL 2.4.0 is necessary for vsicurl.
| Code Block |
|---|
Error detected in HDF5 (1.10.4) thread 0:\n #0\ 00: H5F.c line 509 in H5Fopen(): unable to open file\n major: File accessibilty\\ n minor: Unable to open file\n #001: H5Fint.c line 1400 in H5F__open(): unable \ to open file\n major: File accessibilty\n minor: Unable to open file\n #002:\ H5Fint.c line 1546 in H5F_open(): unable to open file: time = Fri Jan 18 02:59:01 \ 2019\n, name = '/vsicurl/https://gamma.hdfgroup.org/ftp/pub/outgoing/NASAHDF/MOP03T\ -20131129-L3V5.2.1.he5', tent_flags = 0\n major: File accessibilty\n minor: U\ nable to open file\n #003: H5FD.c line 734 in H5FD_open(): open failed\n major:\ Virtual File Layer\n minor: Unable to initialize object\n #004: H5FDsec2.c lin\ e 346 in H5FD_sec2_open(): unable to open file: name = '/vsicurl/https://gamma.hdfg\ roup.org/ftp/pub/outgoing/NASAHDF/MOP03T-20131129-L3V5.2.1.he5' |
Identifying Band from HDF5
| Code Block |
|---|
Warning 1: GDAL was built against curl 7.53.1, but is running against 7.51.0. Runti\
me failure is likely !
'NoneType' object has no attribute 'GetStatistics': AttributeError
Traceback (most recent call last):
File "/var/task/lambda.py", line 26, in handler
stats = band.GetStatistics(0, 1)
AttributeError: 'NoneType' object has no attribute 'GetStatistics'
|
/vsis3 looks up s3.amazonaws.com
If HDF5:%22%2Fvsis3%2Ftest%2FMOP03T-20131129-L3V5.2.1.he5%22:%2F%2FHDFEOS%2FGRIDS%2FMOP03%2FData_Fields%2FRetrievedSurfaceTemperatureDay is submitted, /vsis3 looks up amazonaws.com, not LocalStack S3.
...
The above behavior can be corrected by setting AWS_S3_ENDPOINT but GDAL always uses https:// for data retrieval. https:// cannot be used with LocalStack by default. Define USE_SSL=true when you start LocalStack. However, SSL is not supported by mockit [5].
Python gdal.Open(opendap_url)
/vsimem is invoked on downloaded file.
...
This is due to DODS driver is missing [6].
Background
docker-compose.yml specifies two services
...
| Code Block | ||
|---|---|---|
| ||
$aws sts get-session-token --duration-seconds 129600 |
Solution
- Modify Dockerfile: HDF5 version to 1.10.4. GDAL version to 2.4.0.
- Modify lambda-package-base.sh: libhdf5.so.101 to 103
- docker-compose build (--no-cahce)
- docker-compose run core
- sftp lambda-deploy.zip to host system (e.g. #sftp hyoklee@nene) to save the lambda function.
- Unzip and add your lambda.py. Zip it again.
- Deploy to local stack.
- Check lambda functions executed: $docker ps -a
- Check output logs: $docker logs [container_id]
Experiment
Geolambda's GDAL image could successfully run on LocalStack for a GEE sample GeoTIFF file https://gamma.hdfgroup.org/ftp/pub/outgoing/joe/gee/MOP03T.45.tif. /viscurl worked fine with gdal.Open().
| Code Block |
|---|
Warning 1: GDAL was built against curl 7.53.1, but is running against 7.51.0. Runtime failure is likely !
[-9999.0, 444.1781311035156, -9000.95712966157, 3016.201309916306]
[DEBUG] 2019-01-17T03:38:47.852Z 228987c5-4707-4171-b0c5-da5e55b6f4fd [-9999.0, 444.1781311035156, -9000.95712966157, 3016.201309916306]
END RequestId: 228987c5-4707-4171-b0c5-da5e55b6f4fd
REPORT RequestId: 228987c5-4707-4171-b0c5-da5e55b6f4fd Duration: 529 ms Billed Duration: 600 ms Memory Size: 1536 MB Max Memory Used: 41 MB
{"body": [-9999.0, 444.1781311035156, -9000.95712966157, 3016.201309916306], "statusCode": 200}
|
Performance
The baseline is Goelambda Python on Mac OS X Docker.
| Filename | size | urllib3 | nc4 subset | direct meta |
|---|---|---|---|---|
| https://opendap.larc.nasa.gov/opendap/hyrax/CERES/SYN1deg-1Hour/Terra-Aqua-MODIS_Edition4A/2018/06/CER_SYN1deg-1Hour_Terra-Aqua-MODIS_Edition4A_407406.20180601.hdf | 720M | 32 minutes |
References
- https://lists.osgeo.org/pipermail/gdal-dev/2013-May/036359.html
- https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_drivers (Notable exceptions are netCDF/HDF4/HDF5)
- https://github.com/OSGeo/gdal/pull/786 (Recent support of VSI on netCDF/HDF5)
- https://github.com/koordinates/gdal-vsis3-tests/blob/master/test_gdal_s3.py
- https://github.com/localstack/localstack/issues/55
- http://osgeo-org.1560.x6.nabble.com/gdal-dev-optimal-vsicurl-settings-for-merging-range-requests-td5389484.html
- https://hub.docker.com/r/cumuluss/cumulus-geolambda/
...
Overview
Content Tools