...
Product | From | To | Time | Size | Note | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CERES | Mosaic Dataset | CRF | 2h 45m | 14GB | ArcGIS Pro / 52,673 files
| ||||||||||
CERES | netCDF-4 on S3 | MD on local DB | 6h | 200 rows | ArcGIS Pro 2.6 / 8,760 = (24 * 365) rows. | ||||||||||
CERES | netCDF-4 local | MD on local DB | 1h 4m 50s | 2.58GB | ArcGIS Pro 2.6 / 8,760 = (24 * 365) rows. | ||||||||||
CERES | AWS RDS Table | MD on local DB | 1h 42m 20s | ArcGIS Pro 2.6 Input table points to 8,760 CoG on S3 Parallel processing parameter = 4 | |||||||||||
CERES | AWS RDS Table | MD on local DB | 46m 46s | ArcGIS Pro 2.6 Input table points to 8,760 netCDF-4 on S3 Parallel processing parameter = 4 | |||||||||||
MERRA2 | 365 file list text file | Table | 38s | 24 * 365 rows | Miniconda 3.7 python and ls2rds.py script (sqlalchemy) | ||||||||||
MERRA2 | 1 file list text file | Table | 0.35s | 24 rows | Miniconda 3.7 python and ls2rds.py script (sqlalchemy) | ||||||||||
MERRA2 | AWS RDS Table | MD on RDS | 3m 14s | 2 rows (Cache size: 230K/file) | 2 NetCDF subset on S3 MDCS script on ArcGIS Server Linux 10.7.1 | ||||||||||
MERRA2 | AWS RDS Table | MD on RDS | 1m 40s ~ 2m 30s | 24 rows | 24 NetCDF subset on S3 MDCS script on ArcGIS Server Linux 10.7.1 | ||||||||||
MERRA2 | AWS RDS Table | MD on RDS | 3m 54s | 24 * 4 rows | 4 NetCDF files on S3 MDCS Script on ArcGIS Server Linux 10.7.1 | ||||||||||
MERRA2 | AWS RDS Table | MD on RDS | 3m 6s | 24 * 365 rows | 365 NetCDF files on S3 ArcGIS Pro on Windows Server (Adding Rasters only; no pixel cache; no multidim geoprocessing) | ||||||||||
MERRA2 | MD on RDS | MD on RDS | 1hr | 24 * 365 rows | 365 NetCDF files on S3 MDCS Calculate Statistics script on ArcGIS Server Linux 10.7.1 BMI is very quick. | ||||||||||
MERRA2 | MD on RDS | CRF on local disk | 25m 48s | 19.4GB | Using ArcGIS Pro 2.5.1 with the following options
| ||||||||||
MISR | netCDF-4 | netCDF-4 | 7m 30s | 454M | Subsetting 365 netCDF-4 files and merging with ncks from the downloaded files onArcGIS Server Linux 10.8. | ||||||||||
MISR | netCDF-4 | Mosaic on local disk | 19m 18s |
| |||||||||||
MISR | Mosaic on local disk | CRF on local disk | 10hr 10m 53s | 2.4GB | Using ArcGIS Pro 2.6 with the following options
| ||||||||||
MOP03J | HDF-EOS5 | MRF | 0.065 | 24M | C++ (gdal_translate) / 1 file | ||||||||||
MOP03J | HDF-EOS5 | MRF | 0.365 | 24M | Python (mrf.py) / 1 file | ||||||||||
MOP03J | AWS RDS Table | MD on RDS | 2m 30s | 3 rows | 3 MRF files on S3 | ||||||||||
Sentinel02Sentinel-2 | AWS RDS Table | MD on RDS | 17m 50s | 789 rows |
| ||||||||||
TerraFusion | netCDF-4/HDF5 | OPeNDAP HTML form | 4m | Source file: 32.8G | AWS Open Data S3 on us-west-2 | ||||||||||
TerraFusion | netCDF-4/HDF5 S3 | OPeNDAP DDS only | 2m17s | Source file: 32.8G DDS: 199.78k | AWS Open Data S3 on us-west-2 wget | ||||||||||
TerraFusion | netCDF-4/HDF5 S3 | GeoTIFF on local disk | 109s | 25G | EC2 miniconda 3.7 gdal + pyresample & S3 on us-east-1 MODIS 2030x1354 subset | ||||||||||
TerraFusion | netCDF-4/HDF5 S3 | GeoTIFF on local disk | 157s | 32.8G | EC2 miniconda 3.8 gdal 3.0.2 + pyresample & S3 on us-west-2 MODIS 2030x1354 subset | ||||||||||
TerraFusion | netCDF-4/HDF5 local | GeoTIFF on local disk | 0.9s | 32.8G | EC2 miniconda 3.8 gdal 3.0.2 + pyresample MODIS 2030x1354 subset | ||||||||||
TerraFusion | netCDF-4/HDF5 S3 | GeoTIFF on local disk | 14.9s | 32.8G | EC2 miniconda 3.8 gdal 3.1.2 + pyresample & S3 on us-west-2 MOPITT 436x29 subset | ||||||||||
TerraFusion | THREDDS S3 | GeoTIFF on local disk | 604s | 32.8G | EC2 miniconda 3.8 pydap + pyresample & S3 on us-west-2 | ||||||||||
TerraFusion | THEDDS Local | GeoTIFF on local disk | 1s | 32.8G | TerraFusion file was copied into local drive. | ||||||||||
TerraFusion | THEDDS Local | netCDF4 on local disk | 30.7s | 32.8G | TerraFusion file was copied into local drive. | ||||||||||
TerraFusion | THEDDS Local | netCDF4 on local disk | 10s | 32.8G | TerraFusion file was copied into local drive. |
...