Page History

Instruction for running HEG on Command line mode (batch job) for a bunch of similar files

If you have lots of similar hdf files that need to be processed by HEG, you can use a script similar to the one given below (for example the one provided in HEG's bin directory) or just simply use HEG GUI. If you want to run a batch using HEG GUI please read section 8 of the GUI "Help".

Please follow the instructions below for Unix/Linux or Windows depending on your system.

Unix/Linux:

...

1. Create a directory and put all of your similar hdf files (for the same product) in that directory

...

1. Run HEG GUI so that you create a template parameter file

...

1. Select File -> Open and choose one of the file in the directory created in step a; click open

...

1. Select Field, and then click on "down" arrow.

...

1. Select output file type, projection, etc and click "Accept"

...

1. Click "Save". Choose your parameter file name in the popped up window and then click "Save" on it. Please note that HEG will append "_sawth " to the end of the name that you chose for your parameter file to indicate that the products that you intended to convert are SWATH type. It will append "_grid" for grid products.

...

1. Close HEG GUI. Now edit your created parameter file and make the following changes:

   ...

   1. 1. Change input file name to "input"

   ...

   1. 1. change the bounding box (corner lats/lons) to:
         

   ...

   1. 1.      SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )
         

   ...

   1. 1.      SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)

   ...

   1. 1. change your output file name to "input_out" with appropriate extension ".tif" or ".hdf"

   ...

   1. 1. save and close this file and rename it to heg_template_unix.prm
         

           Following is an example of such a template file created for a MODIS product:
   
     
           NUM_RUNS = 1
           BEGIN
           INPUT_FILENAME = /home/abe/GIS_Proj/HEG_MODIS/2007/elt_season/input
           OBJECT_NAME = MOD_Grid_Snow_500m
           FIELD_NAME = Fractional_Snow_Cover
           BAND_NUMBER = 1
           SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )
           SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)
           RESAMPLING_TYPE = NN
           OUTPUT_PROJECTION_TYPE = UTM
           ELLIPSOID_CODE = WGS84
           UTM_ZONE = 0
           OUTPUT_PROJECTION_PARAMETERS = ( 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 )
           OUTPUT_PIXEL_SIZE = 500.0
           OUTPUT_FILENAME =      /home/abe/GIS_Proj/HEG_MODIS/2007/out/melt_season/input_out.tif
   

   ...

           OUTPUT_TYPE = GEO
           END

   h8. Create a Shell Script file using following lines and make it executable type (e.g use "chmod 0755 myScript.sh"):      

           #! /bin/sh
   
     
           # in this script it is assumed that HEG executables are in
           # "/home/user/heg2.9/HEG/bin" directory, and that we are
           # processing a file that contains GRIDS. For processing files
   

   ...

           # that contain SWATH use "swtif" instead of "resample"

   ...

   
   
     
           for i 

   ...

   in  `find .` ; do

   ...

   
             hdffile=`file $i | awk '

   ...

   {print $2

   ...

   }'`

   ...

   
             if 

   ...

   [ "$hdffile" = "Hierarchical" 

   ...

   ] ; then

   ...

   
               sed 's/input/'$i'/' heg_template_unix.prm > $i.prm

   ...

   ...

   ...

   /home/username/heg2.9/HEG/bin/resample -p $i.prm

   ...

   ...

   fi

   ...

   
           done

   ...

   
   9. Before running the script set 3 environment variables (assuming that HEG is installed in /home/username/heg2.9/HEG/ 
   ...
    ):
   ...
       setenv MRTDATADIR /home/username/heg2.9/HEG/data
   ...
   
       setenv PGSHOME /home/username/heg2.9/HEG/TOOLKIT_MTD
   ...
   
       setenv HEGUSER ME
   ...
   
   10. Now run the shell script for a HEG batch job!
   
   Windows: 
     Create a directory and put all of your similar hdf files in that directory
   Run HEG GUI to create a template parameter file
   Select File -> Open and choose one of the file in the directory created in step a; click open
   Select Field, and then click on "down" arrow.
   Select output file type, projection, etc and click "Accept"
   Click "Save". Choose you parameter file name in the popped up window and then click "Save" on it. Please note that HEG will append "_sawth " to the end of the name that you chose for your parameter file to indicate that the products that you intended to convert are SWATH type. It will append "_grid" for grid products.
   Close HEG GUI. Now edit your created parameter file and make the following changes:
   Change input file name to "input"
   change the bounding box (corner lats/lons) to:
        SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )
        SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)
   change your output file name to "input_out" with appropriate extension ".tif" or ".hdf"
   save and close this file and rename it to heg_template_unix.prm
           Following is an example of such a template file created for a MODIS product:
   ...
   
   
     
           NUM_RUNS = 1
   ...
   
           BEGIN
   ...
   
           INPUT_FILENAME
   ...
    = /home/abe/GIS_Proj/HEG_MODIS
   ...
   /2007
   ...
   /elt_season
   ...
   /input
   ...
   
           OBJECT_NAME = MOD_Grid_Snow_500m
   ...
   
           FIELD_NAME = Fractional_Snow_Cover
   ...
   
           BAND_NUMBER = 1
   ...
   
           SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )
   ...
   
           SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)
   ...
   
           RESAMPLING_TYPE = NN
   ...
   
           OUTPUT_PROJECTION_TYPE = UTM
   ...
   
           ELLIPSOID_CODE = WGS84
   ...
   
           UTM_ZONE = 0
   ...
   
           OUTPUT_PROJECTION_PARAMETERS = ( 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 )
   ...
   
           OUTPUT_PIXEL_SIZE = 500.0
   ...
   
           OUTPUT_FILENAME =
   ...
         /home/abe/GIS_Proj/HEG_MODIS
   ...
   /2007
   ...
   /out/melt_season
   ...
   /input_out.tif
   ...
   
           
   ...
   OUTPUT_TYPE = GEO
   ...
   
           END
   ...
   Warning:
   ...
   Please 
   ...
    be 
   ...
    aware 
   ...
    that 
   ...
    in 
   ...
    windows 
   ...
    if 
   ...
    you 
   ...
    edit 
   ...
    your 
   ...
    file 
   ...
    with 
   ...
    some 
   ...
    editors 
   ...
    such 
   ...
    as 
   ...
    wordpad 
   ...
    the 
   ...
    editor 
   ...
    puts 
   ...
    carriage 
   ...
    return 
   ...
    "^M" 
   ...
    at 
   ...
    the 
   ...
    end 
   ...
    of 
   ...
    each 
   ...
    line. 
   ...
    This 
   ...
    will 
   ...
    prevent 
   ...
    HEG 
   ...
    from 
   ...
    running 
   ...
    and 
   ...
    will 
   ...
    cause 
   ...
    error. 
   ...
    Make 
   ...
    sure 
   ...
    that 
   ...
    your 
   ...
    editor 
   ...
    will 
   ...
    work 
   ...
    like 
   ...
    UNIX 
   ...
    editors. 
   ...
    If 
   ...
    you 
   ...
    have 
   ...
    linux 
   ...
    or 
   ...
    cygwin, 
   ...
    do 
   ...
    editing 
   ...
    there. 
   ...
    Also 
   ...
    the 
   ...
    linux 
   ...
    application 
   ...
    "dos2unix" 
   ...
    cleans 
   ...
     up 
   ...
    "^M" 
   ...
    from 
   ...
    a 
   ...
    file.
   ...
   
   The script MyHEG_batchScript.bat 
   ...
    in 
   ...
   BatchRunning.zip
   ...
    uses 
   ...
    MS 
   ...
    DOS 
   ...
    script 
   ...
    commands 
   ...
    to 
   ...
    run 
   ...
    a 
   ...
    batch 
   ...
    job 
   ...
    for 
   ...
    HEG 
   ...
    in 
   ...
    windows. 
   ...
    First 
   ...
    it 
   ...
    edits 
   ...
    a 
   ...
    template 
   ...
    parameter 
   ...
    file 
   ...
    (created 
   ...
    using 
   ...
    HEG 
   ...
    GUI 
   ...
    or 
   ...
    by 
   ...
    other 
   ...
    means 
   ...
    as 
   ...
    instructed 
   ...
    above) 
   ...
    to 
   ...
    the 
   ...
    desired 
   ...
    format. 
   ...
    The 
   ...
    edited 
   ...
    file 
   ...
    that 
   ...
    the 
   ...
    script 
   ...
    creates, 
   ...
    using 
   ...
    the 
   ...
    .bat 
   ...
    file 
   ...
    for 
   ...
    filename 
   ...
    substitution, 
   ...
    will 
   ...
    contain 
   ...
    the 
   ...
    LineFeed 
   ...
    characters 
   ...
    at 
   ...
    the 
   ...
    end 
   ...
    of 
   ...
    each 
   ...
    line. 
   ...
    Since 
   ...
    HEG 
   ...
    does 
   ...
    not 
   ...
    recognize 
   ...
    this 
   ...
    character 
   ...
    (causing 
   ...
    error) 
   ...
    they 
   ...
    should 
   ...
    be 
   ...
    removed. 
   ...
    In 
   ...
    future 
   ...
    this 
   ...
    will 
   ...
    be 
   ...
    worked 
   ...
    out 
   ...
    to 
   ...
    be 
   ...
    done 
   ...
    with 
   ...
    MS 
   ...
    DOS 
   ...
    commands 
   ...
    if 
   ...
    possible.
   ...
   Currently 
   ...
    script 
   ...
    uses 
   ...
    CYGWIN's 
   ...
    dos2unix.exe 
   ...
    tool 
   ...
    that 
   ...
    does 
   ...
    cleaning 
   ...
    for 
   ...
    us.
   ...
   To 
   ...
    use 
   ...
    this 
   ...
    executable 
   ...
    we 
   ...
    require 
   ...
    to 
   ...
    have 
   ...
    a 
   ...
    few 
   ...
    other 
   ...
    cygwin's 
   ...
    dll 
   ...
    files 
   ...
    in 
   ...
    the 
   ...
    same 
   ...
    directory 
   ...
    that 
   ...
    dos2unix.exe 
   ...
    exist.
   ...
      Please 
   ...
    also 
   ...
    note 
   ...
    that 
   ...
    if 
   ...
    you 
   ...
    use 
   ...
    MS 
   ...
    wordpad 
   ...
    for 
   ...
    creating 
   ...
    parameter 
   ...
    file, 
   ...
    you 
   ...
    can 
   ...
    use 
   ...
    this 
   ...
    executable 
   ...
    on 
   ...
    command 
   ...
    line 
   ...
    to 
   ...
    get 
   ...
    rid 
   ...
    of 
   ...
    LineFeed 
   ...
    characters 
   ...
    in 
   ...
    your 
   ...
    .prm 
   ...
    file.
   ...
   
   Here 
   ...
    are 
   ...
    the 
   ...
    steps 
   ...
    for 
   ...
    using 
   ...
    our 
   ...
    Windows 
   ...
    batch 
   ...
    script:
   Download BatchRunning.  zip 
   The sample template parameter file is created for an ASTER product. You need to create your own template parameter file possibly by using HEG GUI, and editing it to put in the desired form as the sample (i.e. having "infile" “infile” for the input filename, and in output filename, and modifying the subset lat/lon as instruvted in step b above.)
   Extract the files from BatchRunning.zip. In the main directory you will have all executables (except for the resample.exe, swtif.exe, etc. needed for HEG executable), scripts and dll files
   In this directory we have indir and outdir. You should put all your input files in indir. The outdir will contain your template parameter file. After executing the MyHEG_batchScript.bat file your output files will go to outdir directory.
   Outputs will have swath or grid area in the original file although in the template parameter file you may use a global subset area.
   The Modified parameter files will exist in the main directory for later reference
   ...