ArcGIS Server Performance Recommendations: https://enterprise.arcgis.com/en/server/latest/administer/linux/tuning-your-arcgis-server-site.htm

Modifications Made:

  • CERES Service:
    • Changed Maximum number of instance per machine to: 4

Response from ImageService Performance

We use arcgis-server-8.gisdemo.net server. The command is time wget HTML query URL. The parameter Max Mosaic Image Count controls the number of values returned. In CRF, it is set to null. In MRF, it is set to 50. In CoG/netCDF-4 it is set to 20.

ProductFormatStorageTime (s)Note
CERESCRFS34.2

1 URL call / Sampling 1,000 points
SDT-79 - Getting issue details... STATUS

CERESCRFEBS (=Local)3.8
CERESCRFS35.0returns 8,760 points even if input parameter is 20.
CERESCRFLocal5.1returns 8,760 points even if input parameter is 20.
CERESMRFS32.4Output has much less points (=50) than CRF (=8,760).
CERESCoGS33.6Output has 20 points. 8,760 CoG files.
CERESnetCDF-4S31.24Output has 20 points. 8,760 netCDF-4 files.

Transformation Performance

Please adjust lambda timeout parameter in serverless.yml for MDCS script or data conversion. Timeout can generate incomplete Mosaic Dataset (MD) or MRF file.

ProductFromTo

Time

SizeNote
CERESMosaic DatasetCRF2h 45m14GB

ArcGIS Pro / 52,673 files
SDT-74 - Getting issue details... STATUS

CERESnetCDF-4 on S3MD on local DB6h200 rowsArcGIS Pro 2.6 / 8,760 = (24 * 365) rows.
CERESnetCDF-4 localMD on local DB1h 4m 50s2.58GBArcGIS Pro 2.6 / 8,760 = (24 * 365) rows.
CERESAWS RDS TableMD on local DB1h 42m 20s
ArcGIS Pro 2.6
Input table points to 8,760 CoG on S3
Parallel processing parameter = 4
CERESAWS RDS TableMD on local DB46m 46s

ArcGIS Pro 2.6

Input table points to 8,760 netCDF-4 on S3

Parallel processing parameter = 4

MERRA2365 file list text fileTable38s24 * 365 rowsMiniconda 3.7 python and ls2rds.py script (sqlalchemy)
MERRA21 file list text fileTable0.35s24  rowsMiniconda 3.7 python and ls2rds.py script (sqlalchemy)
MERRA2AWS RDS TableMD on RDS3m 14s2 rows
(Cache size: 230K/file)		2 NetCDF subset on S3
MDCS script on ArcGIS Server Linux 10.7.1
MERRA2AWS RDS TableMD on RDS1m 40s ~
2m 30s		24 rows24 NetCDF subset on S3
MDCS script on ArcGIS Server Linux 10.7.1
MERRA2AWS RDS TableMD on RDS3m 54s24 * 4 rows4 NetCDF files on S3
MDCS Script on ArcGIS Server Linux 10.7.1
MERRA2AWS RDS TableMD on RDS3m 6s24 * 365 rows365 NetCDF files on S3
ArcGIS Pro on Windows Server (Adding Rasters only; no pixel cache; no multidim geoprocessing)
MERRA2MD on RDSMD on RDS1hr24 * 365 rows365 NetCDF files on S3
MDCS Calculate Statistics script on ArcGIS Server Linux 10.7.1
BMI is very quick.
MERRA2MD on RDSCRF on local disk25m 48s

19.4GB
17,597 files
26,284 folders
24 * 365 rows

Using ArcGIS Pro 2.5.1 with the following options
  • Transpose on
  • Process as Multidim
MISRnetCDF-4netCDF-47m 30s454MSubsetting 365 netCDF-4 files and merging with ncks from the downloaded files onArcGIS Server Linux 10.8.
MISRnetCDF-4Mosaic on local disk19m 18s
  • 2019 merged netCDF-4 (time=365)
  • ArcGIS Pro 2.6 in Windows Server
MISRMosaic on local diskCRF on local disk10hr 10m 53s2.4GBUsing ArcGIS Pro 2.6 with the following options
  • Transpose on
  • Process as Multidim
MOP03JHDF-EOS5MRF0.06524MC++ (gdal_translate)  / 1 file
MOP03JHDF-EOS5MRF0.36524MPython (mrf.py) / 1 file
MOP03JAWS RDS TableMD on RDS2m 30s3 rows

3 MRF files on S3
MDCS Script on ArcGIS Server Linux 10.7.1

Sentinel-2AWS RDS TableMD on RDS17m 50s789 rows
  • ArcGIS Pro 2.6 in Windows Server
  • CoG files on us-west-2.
TerraFusionnetCDF-4/HDF5OPeNDAP HTML form4mSource file: 32.8GAWS Open Data S3 on us-west-2

TerraFusionnetCDF-4/HDF5 S3OPeNDAP DDS only2m17sSource file: 32.8G
DDS: 199.78k		AWS Open Data S3 on us-west-2
wget
TerraFusionnetCDF-4/HDF5 S3GeoTIFF on local disk109s25GEC2 miniconda 3.7 gdal + pyresample & S3 on us-east-1
MODIS 2030x1354 subset
TerraFusionnetCDF-4/HDF5 S3GeoTIFF on local disk157s32.8GEC2 miniconda 3.8 gdal 3.0.2 + pyresample & S3 on us-west-2
MODIS 2030x1354 subset
TerraFusionnetCDF-4/HDF5 localGeoTIFF on local disk0.9s32.8GEC2 miniconda 3.8 gdal 3.0.2 + pyresample
MODIS 2030x1354 subset
TerraFusionnetCDF-4/HDF5 S3GeoTIFF on local disk14.9s32.8GEC2 miniconda 3.8 gdal 3.1.2 + pyresample & S3 on us-west-2
MOPITT 436x29 subset
TerraFusionTHREDDS S3GeoTIFF on local disk604s32.8G

EC2 miniconda 3.8 pydap + pyresample & S3 on us-west-2
MOPITT 436x29 subset

TerraFusionTHEDDS LocalGeoTIFF on local disk1s32.8G

TerraFusion file was copied into local drive.
EC2 miniconda 3.8 pydap + pyresample
MOPITT 436x29 subset
THREDDS run on the same machine.

TerraFusionTHEDDS LocalnetCDF4 on local disk30.7s32.8G

TerraFusion file was copied into local drive.
EC2 miniconda 3.7.6 pydap + pyresample on us-east-1
MODIS 2030x1354 subset

TerraFusionTHEDDS LocalnetCDF4 on local disk10s32.8G

TerraFusion file was copied into local drive.
EC2 miniconda 3.8 pydap + pyresample
MODIS 2030x1354 subset
THREDDS run on the same machine.

Publishing Performance

ProductFromTo

Time

SizeNote
MERRA2ArcGIS ServerAWS RDS1.6sEmpty input tableCreate MD input table using psql command line tool.
MERRA2ArcGIS ServerArcGIS Portal31.5s2 Rasters in MDUpload service using upload.py arcpy script.
MERRA2ArcGIS ServerArcGIS Portal15.8s2 Rasters in MDDelete service using portal.py arcpy script.
MERRA2ArcGIS ServerAWS RDS23.5s2 Rasters in MDDelete mosaic dataset using delete.py arcpy script.
MERRA2ArcGIS ServerAWS RDS1.6s2 rows in input tableDelete input table using psql command line tool.

Network Transfer Performance

ProductFromTo

Time

SizeNote
TerraFusionS3ArcGIS Server189Mbs/sec32G

See Also