Before attempting to begin using the SDP Toolkit version 5.2 some preliminary work ought to be done to ensure as smooth a transition from your previous Toolkit version as possible. This document discusses the changes that you should make before upgrading to TK5.2. The following is an outline of this document:



NOTE: In this document:
$PGSHOME is the Toolkit top level directory location (i.e. /tools/cots/TOOLKIT).
$BRAND is the Toolkit name for the O/S you are running (one of: dec, hp, ibm, sgi, sgi32, sgi64, sun5) .


UPGRADING THE PROCESS CONTROL FILE (PCF)

The first thing that you should do is make a local copy of $PGSHOME/runtime/$BRAND/PCF.relB0 and edit this file to suit your individual needs (do not use $PGSHOME/runtime/PCF.relB0.template).

NOTE: system administrators should make any necessary global changes to this file or a copy of this file and leave it in $PGSHOME/runtime/$BRAND so that all users can get a copy of the PCF for their personal use.

Do not attempt to use TK5.2 with a PCF that was being used with an older version of the Toolkit.

NOTE: for a list of major changes made to the PCF for TK5.2 see: Appendix A--Changes to the PCF.

When you have finished upgrading to the new PCF you should run this new PCF through the Toolkit PCF checking utility, pccheck.sh. This utility can be found in the Toolkit bin directory ($PGSHOME/bin/$BRAND). It should be run like this:

pccheck.sh -i PCF.relB0

Check the comments (in the PCF) of any entries that show up with errors and take appropriate action.

Tips On Updating Your PCF

  • ASCII metadata files

    The method for generating ASCII metadata files for non-HDF output products has changed. Although this change does not require any changes to your source code, it does require a change to how you specify your ASCII metadata file in your PCF (please see non-HDF product metadata file generation, below).

  • Default directories

    If you have changed any of the default directories (indicated in each relevant section of the PCF with '!' character as the first character on the line), make sure you replace the entry(ies) in your new PCF, otherwise you will get the Toolkit defaults which may not even work for you.

  • Toolkit Defaults

    If you have changed the default values of any Toolkit runtime parameters (or added values where there weren't any), make sure you update your new PCF to reflect these changes.

    If you have changed the location of any files that came with the Toolkit (e.g. earthfigure.dat) to a non-standard location, make sure that you update your new PCF to reflect these changes.

    NOTE: You will be unable to use old leapsec.dat and utcpole.dat files. If you need to change these files, you must start with the files delivered with TK5.2 and make changes to those files (see: special notes on time files, below).

  • Version Numbers On SUPPORT files

    In previous versions of the Toolkit file versioning was not allowed for files listed in the SUPPORT INPUT/OUTPUT FILES sections of the PCF. Entries in these sections did not, therefore, require a version number. This has changed for TK5.2. File versioning is now allowed in these sections of the PCF. All entries in these sections must have a file version number. Make sure that as you add entries to your new PCF in these sections that the entries include a version number at the end of the line (even if they did not have one in your old PCF).

  • AA data directory

    If you have installed the AA data on your system and you are using this data (you are using the PGS_AA_... tools), make sure you replace all occurrences of AA_DATA_INSTALL_DIR in your new PCF with the actual directory location of the AA data.

    NOTE: it is not necessary to re-install this data, there has been no change in this data since the last Toolkit delivery.


METADATA RELATED CHANGES

Substantial changes have been made in the metadata generation process. It is strongly recommended that anyone using the Toolkit MET tools reviews the MET section of the Release B.0 SCF Toolkit Users Guide (6.2.1.4) as well as (or perhaps especially) Appendix J.

Things that will affect immediate use of TK5.2:

    • Metadata Configuration Files (MCFs):

      Metadata Configuration Files (MCFs) should now be obtained from ECS. A template MCF is no longer delivered with the Toolkit. Old MCF files will NOT work with TK5.2. The new procedure for obtaining MCFs is discussed in the Release B.0 SCF Toolkit Users Guide in Appendix J.

      NOTE: for a description of the changes made to the MCF see Appendix B--Changes to the MCF.

    • non-HDF product metadata file generation:

The procedure for producing metafiles for non-HDF products has been changed. The changes should not require any changes to source code. Changes must be made to the PCF.

NOTE: this is also known as the ASCII dump procedure.

In Toolkit 5.1.1 the metadata file name was explicitly given a name in the PCF and associated with a logical ID in an OUTPUT FILES section of the PCF.

In Toolkit 5.2 the same logical ID as used in TK5.1.1 should be moved to the RUNTIME PARAMETERS section of the PCF. The value should be changed to indicate the logical ID and version number of the specific product for which the metadata is being generated. The file name is no longer necessary! The toolkit will look up the file name of the associated output product and use that name with a ".met" extension as the name of the ASCII metadata file.

Examples:

The OLD PCF entries looked like this:

? PRODUCT OUTPUT FILES
! /default/outputs
100|myOutputProduct|||||1
101|myOutputProductMetadata|||||1

The NEW PCF entries should look like this:

? PRODUCT OUTPUT FILES
! /default/outputs
100|myOutputProduct|||||1

? USER DEFINED RUNTIME PARAMETERS
101|<some appropriate comment>|100:1

NOTE: click here for a description of USER DEFINED RUNTIME PARAMETERS.

NOTE: the ".met" file generated will NOT be in the same directory as the output product file, it will instead be in the directory where the PGE is executing (this behavior now mirrors the behavior of HDF formatted output product metadata files).

NOTE: some users are likely using the default asciidump scenario, in this case the users should move the logical ID 10255 as shown in the example above for logical ID 101.

NOTE: if users want to make generic ASCII dumps of the metadata they can define a dummy file to be associated with the metadata, it is NOT necessary that the file actually exists in order for the metadata file to be generated (an entry in the PCF MUST, however, exist).

Example:

? PRODUCT OUTPUT FILES
! /default/outputs
100|asciidump|||||1

? USER DEFINED RUNTIME PARAMETERS
101|<some appropriate comment>|100:1

(in this example, no file named asciidump actually exists on disk, the metadata tools can be used now to create a file called asciidump.met)

NOTE: this will actually be backward compatible AT THE SCF (but NOT at the DAAC), i.e. if the PCF is not changed then TK5.2 will default to using the old method.


SPECIAL NOTES ON TIME FILES

If you are running simulations using future times you must make adjustments to your time files (utcpole.dat, leapsec.dat). If you have already altered these files (for earlier versions of the Toolkit) to accommodate your needs, you MUST start again with the files delivered with TK5.2 (there has been a format change). Time files derived from previous versions of the Toolkit will NOT work with TK5.2.

Using the files as delivered with TK5.2 the latest dates that can be successfully processed by tools that require these files are:

leapsec.dat: 1997-12-27
utcpole.dat: 1998-03-26 (updates already available)

NOTE: some tools require both files, therefore the limiting factor is likely to be leapsec.dat.

NOTE: the tool groups affected by these files are CBP, CSC, EPH, IO_L0, and TD. If you attempt to use these tools with times in future of the times supported by the time files, the tools will return an error.

Changing the time files

Given below are instructions on how you can change your new time files (i.e. those delivered with TK5.2) to allow for processing of times beyond those that are currently supported.

WARNING !

These changes are only to allow simulations to run without getting errors from the Toolkit time tools. Once you make these changes to the time file(s) your results may no longer be temporally accurate or scientifically valid.

Note: Any time file altered to support simulations using future times should not then be updated using the Toolkit time file update utilities (update_utcpole.sh and update_leapsec.sh).

    • Changing leapsec.dat

Do not change the leapsec.dat file that was delivered with the Toolkit. Make a local copy and alter that. Even if you would like to create a new file for a large group of people, make a copy somewhere that everyone can read it.

Edit your local copy and change the date of the first line in the header from:

Checked(unchanged): 1997-04-29T23:06:10Z using USNO ...

to:

Checked(unchanged): 2010-04-29T23:06:10Z using USNO ...

NOTE: The exact date is not significant, but it should be greater than any times for which you plan to do simulations.

Then replace the line in your PCF:

10301|leapsec.dat|~/database/common/TD||||1

with:

10301|leapsec.simulation|/usr/local/data||||1

(where, in this case, a copy of the file leapsec.dat was made and named leapsec.simulation, then edited--as above--and placed in the directory /usr/local/data)

    • Changing utcpole.dat

Do not change the utcpole.dat file that was delivered with the Toolkit. Make a local copy and alter that. Even if you would like to create a new file for a large group of people, make a copy somewhere that everyone can read it.

Edit your local copy and change the date of the first line of data (3rd line in the file) from:

41317 +0.061000 0.002000 +0.051000...

to:

43817 +0.061000 0.002000 +0.051000...

then change the last line of data (last line in the file) from:

50923 -0.143697 0.021777 +0.221351...

to:

53423 -0.143697 0.021777 +0.221351...

WARNING: be very careful when editing this file, change only the characters you need to change, do not "cut-and-paste" data into this file (the file is read as binary by the Toolkit and the byte count of the header and each individual record must not be altered).

NOTE: the exact dates of change are not significant, but the date of the last record should be greater than any times for which you plan to do simulations. Note that the dates being changed here are Modified Julian Dates.

NOTE: it is critical that the numerical CHANGE to the date of the first line of data be equivalent to the numerical CHANGE to the date of the last line of data.

NOTE: this example change (given above) changes the range of the utcpole.dat file (and hence the range of times that can successfully be processed by the Toolkit using this file) from ~1/72-3/98 to ~11/78-2/05.

Then replace the line in your PCF:

10401|utcpole.dat|~/database/common/CSC||||1

with:

10401|utcpole.simulation|/usr/local/data||||1

(where, in this case, a copy of the file utcpole.dat was made and named utcpole.simulation, then edited--as above--and placed in the directory /usr/local/data)


APPENDIX A--MAJOR CHANGES TO THE PCF FOR TK5.2

  • File versioning is now allowed in SUPPORT INPUT and SUPPORT OUTPUT sections. All file entries in this section have now been given a file version number (in PCF.relA there were no file version numbers on these entries).

  • Added new logical IDs for critical new files:
    10801: spacecraft tags definition file (this is required by many of the geolocation tools).
    11001: units conversion file (this is required by the CUC units conversion tool).

    NOTE: the Toolkit at the SCF is backwards compatible with TK5.1.1, i.e. if it does not find these entries in the PCF it will default to a behavior mimicking TK5.1.1 behavior (this won't be the case at the DAAC, but this shouldn't make a difference to SCF developers).

  • Added new logical ID for critical new runtime parameter:
    10220: the value of this parameter is the toolkit version string. the value of this string should be: SCF B.0 TK5.2

  • Default location of several data files moved to ~/database/common/... from ~/database/$BRAND/...

    NOTE: for the SCF the Toolkit will put links from the old locations of these files to the new locations (i.e. an old PCF type entry will work).


APPENDIX B--CHANGES TO THE MCF FOR TK5.2

Changes to the ECS Science Data Model mean that MCFs built to the old data model will not generate a valid (i.e. complete) set of metadata using any version of the Toolkit. Users should now obtain a custom MCF from ECS. Note that this is an ECS metadata issue and not directly related to the Toolkit (i.e. the Toolkit cannot enforce compliance with this new data model if the Toolkit is not given a valid MCF). The new procedure for obtaining MCFs is discussed in detail in the Release B.0 SCF Toolkit Users Guide in Appendix J.

There has, however, been a change to the MCF that does directly affect the Toolkit. An old MCF will NOT work with TK5.2 (i.e. an error will be returned by the Toolkit metadata initialization tool). A new object must appear in the MCF in order for the Toolkit to properly initialize its MET routines. The ODL for this object is as follows:

OBJECT = ProductionDateTime
        Data_Location = "TK"
        NUM_VAL = 1
        TYPE = "DATETIME" 
        Mandatory = "TRUE" 
END_OBJECT = ProductionDateTime

This object should appear just BELOW the lines:

GROUP = INVENTORYMETADATA

GROUPTYPE = MASTERGROUP

This object may be inserted in an OLD MCF file in order to allow the Toolkit MET routines to be initialized using such a file.

Again, this will allow the Toolkit to function without error, but in order to guarantee that an ECS-compliant set of metadata is generated, a new MCF should be obtained from ECS directly.

  • No labels