In phase II of the granule deletion process, the user runs the EcDsDeletionCleanup utility against granules that have been marked for deletion in Phase I. The EcDsDeletionCleanup utility permanently removes science files and optionally metadata for granules which are marked for deletion from both the Online Archive and the backup/tape Archive.
For granules marked for deletion with either the –physical or –dfa option of the EcDsBulkDelete utility, the EcDsDeletionCleanup utility will remove all science files for the granule from the archives. If the granules were marked for deletion with the –physical option, and the –noassoc option was not specified, all files for associated Browse, PH, QA and MP granules will also be removed from the archives.
If granules were marked for deletion using the –physical option of the EcDsBulkDelete utility, the EcDsDeletionCleanup utility will also clean up all metadata for the marked granules. This includes removing all inventory database entries for the marked granules, and removing corresponding metadata XML files from the archives, as well.
When the EcDsDeletionCleanup utility is executed, it will check if there is any unfinished work from the previous run(s). If so, the utility will prompt the user with a selection menu. The operator may choose to rerun the unfinished run(s) only, which will be resumed from the interrupted point and continue physical cleanup for granules identified for deletion in the previous interrupted run(s). Alternately, the operator may choose to complete unfinished run(s) and start a new run, which will complete cleanup from the previous interrupted run(s), but also clean up granules identified for deletion by the new run.
Unlike the Bulk Search, Bulk Delete and Undelete utilities, EcDsDeletionCleanup requires user interactions during execution.
The generic format for the command line of the EcDsDeletionCleanup utility is the following:
- EcDsDeletionCleanup.pl
–mode <MODE>
[-batch <number>]
[-grbatch <number>]
[-phbatch <number>]
[-log <log_file_name>]
[-xmlbatch <number>]
[-databatch <number>]
[-logbatch <number>]
Table 4.7.10-4 provides a description of the parameters used in executing the EcDsDeletionCleanup.pl script.
Table 4.7.10-4. Command Line Parameters for EcDsDeletionCleanup.pl (1 of 3)
| Parameter Name | Mandatory | Description |
|---|---|---|
| Mode | Yes | The ECS mode in which the utility is to operate. This parameter may be omitted if the environment variable MODE is set. |
| Log | No | The name of the file to which utility messages will be logged. If this is not provided, the utility will prompt the user at runtime to either supply a log file name, or to accept the default log file name, EcDsDeletionCleanup.log. The log file will be stored in the /usr/ecs/<MODE>/CUSTOM/logs directory. |
Table 4.7.10-4. Command Line Parameters for EcDsDeletionCleanup.pl (2 of 3)
| Parameter Name | Mandatory | Description |
|---|---|---|
| batch | No | This is a tuning parameter. It represents the batch size for populating the DsStPendingDelete table. This parameter may be omitted if the environment variable BATCH_SIZE_GRANULE is set. If the environment variable BATCH_SIZE_GRANULE is set, and -batch <number> is also specified, the value from command line parameter –batch will be used. If neither the environment variable BATCH_SIZE_GRANULE is set nor –batch is specified, the user will be prompted at runtime to enter a value (a value of 10000 is suggested by the prompt text). |
| grbatch | No | This is a tuning parameter. It represents the batch size used for physical granule file cleanup. This parameter may be omitted if the environment variable BATCH_SIZE_GRANULE is set. (Note: the environment variable BATCH_SIZE_GRANULE applies to both –batch and –grbatch). If the environment variable BATCH_SIZE_GRANULE is set, and -grbatch <number> is also specified, the value from command line parameter –grbatch will be used. If neither the environment variable BATCH_SIZE_GRANULE is set nor –grbatch is specified, the user will be prompted at runtime to enter a value (a value of 10000 is suggested by the prompt text). |
| phbatch | No | This is a tuning parameter. It represents the phbatch size for PH granule deletion. Because PH granule deletion could be time consuming, setting a high batch size for PH granule deletion could lock the database for a long period of time; therefore, a low value of phbatch is recommended. This parameter may be omitted if the environment variable BATCH_SIZE_PH is set. If the environment variable BATCH_SIZE_PH is set and -phbatch <number> is also specified, the value from command line parameter –phbatch will be used. If neither the environment variable BATCH_SIZE_PH is set nor –phbatch is specified, the user will be prompted at runtime to enter a value (a value of 5 is suggested by the prompt text). |
| xmlbatch | No | This is a tuning parameter. It represents the batch size for processing the deletion of XML files from the XML Archive. The utility will iterate through the deletion of XML files retrieving and deleting "xmlbatch" files per iteration. This controls memory growth of the utility when processing a large number of granules. If this parameter is not specified, the utility will prompt for a value (suggesting a size of 1000). |
Table 4.7.10-4. Command Line Parameters for EcDsDeletionCleanup.pl (3 of 3)
| Parameter Name | Mandatory | Description |
|---|---|---|
| databatch | No | This is a tuning parameter. It represents the batch size for processing the deletion of data files from the "offline/backup" Archive (note: this archive is typically on tape). The utility will iterate through the deletion of data files retrieving and deleting "databatch" files per iteration. This controls memory growth of the utility when processing a large number of granules. If this parameter is not specified, the utility will prompt for a value (suggesting a size of 10000). |
| logbatch | No | The "logbatch" parameter controls the frequency of progress messages in the log. If this parameter is not specified, the utility will prompt for a value (suggesting a size of 100). |
The following sections describe two typical physical cleanup scenarios. In order to simplify the command line, we assume that the user has set the following environment variables before running the EcDsDeletionCleanup utility:
setenv MODE <MODE>
The utility will prompt the user to enter the database password during runtime.
Run a New Physical Cleanup
Command for starting a new run:
- EcDsDeletionCleanup.pl [-batch <number>]
[-grbatch <number>]
[-phbatch <number>]
[-log <log_file_name>]
If no log file name is specified on the command line, the utility will prompt the user to enter one. The user also may select to use the default log file name, which will be EcDsDeletionCleanup.log. The log file will be created under the directory /usr/ecs/<MODE>/CUSTOM/logs/.
The physical cleanup utility EcDsDeletionCleanup will prompt for the database password, and then prompt for any required parameter(s) which have not been set via corresponding environment variables and were not specified on the command line.
The utility will first check if there are any granule(s) that have been marked for deletion (with either the –physical or –dfa option); if there are none, the utility will terminate.
If there are granules marked for deletion, the following menu will be displayed for user selection:
==== Menu for Lag Time ====
- Select granules for a specific day (lag<n> or date <mm/dd/yyyy> format)
- Select all granules older than a specific day (lag<n> or date <mm/dd/yyyy> format)
- Quit
Select 1, 2 or 3: _
The user needs to enter 1, 2 or 3 for:
- Only cleanup granules whose deletion date falls into a single day specified by the lag time;
- Cleanup all granules whose effective deletion date is older than the date specified by the lag time;
- Nothing to cleanup, exit.
If the user chooses menu selection 1 or 2, the user will next be prompted to enter either a lag time in units of days OR a date (in <mm/dd/yyyy> format, such as 04/18/2003). An entry of zero is equivalent to today's date. (See below for a more detailed description of how lag time is used.) Once a lag time or a date is entered, the user will be requested to confirm the entry. If the user answers "N", the utility will prompt the user to re-enter the lag time or date.
A lag time is used to exclude or include a set of granules marked for deletion from the current cleanup run.
For menu selection 1, if an integer <n> is entered for lag time, only granules which were marked for deletion at any time on the date <n> days ago (with respect to the current system date) will be eligible for clean up in the current run. For example, if the utility is run on 03/04/2008 and a lag time of 3 is specified, only granules which were marked for deletion at any time on 03/01/2008 will be eligible for clean up.
If a date such as 05/11/2008 is entered, only granules which were marked for deletion at any time on 05/11/2008 will be eligible for clean up in the current run.
For menu selection 2, if an integer <n> is entered for lag time, only granules which were marked for deletion more than <n> days ago (with respect to the current system date) will be eligible for clean up in the current run. For example, if the utility is run on 03/04/2008 and a lag time of 3 is specified, only granules which were marked for deletion on or before midnight on 03/01/2008 will be eligible for clean up in the current run. If a date such as 05/11/2008 is entered, only granules which were marked for deletion on or before midnight on 05/11/2008 will be eligible for clean up in the current run.
After a lag time is confirmed, the utility will display another menu for user selection:
==== Menu for Data Type ====
1. Specify datatype(s) and version for deletion by an input file
The file format: one ESDT.Version <AST_L1BT.001 or AST_L1B*.001> per line
2. Select all datatypes for deletion
3. Quit
Select 1, 2 or 3: _
The user needs to enter 1, 2 or 3 for:
- Cleanup granules marked for deletion which have an ESDT shortname and versionid in the input file. The input file lists one ESDT per line, in <shortname.versionid> format. A wildcard * may be used as part of the ESDT shortname;
- Cleanup all granules marked for deletion, regardless of their ESDT.version;
- Nothing to cleanup, exit.
Selecting 1 or 2 will start physical cleanup. The utility will present a list of ESDTs and granule counts to be deleted. The operator is prompted to continue or not. If "y" is selected the operator is then prompted to determine whether the utility should run interactively or not. If the operator selects to run interactively, the utility will delete a set of granules from the archive based upon the value of "databatch." When the utility completes the "databatch" number of granule deletions, it presents an updated list of the remaining granules to be deleted and prompts the operator to continue again. This process is repeated until all files are deleted. If the operator specifies "n" to the prompt for running interactively, the utility will process all selected granules without any prompting. This is useful when the operator wants to run the utility unattended and is confident about what is going to be deleted. In all cases the utility progress and error information will be written in the log file.
Rerun unfinished Physical Cleanup
The EcDsDeletionCleanup utility always checks if there were any unprocessed granule(s) left over from a previous unfinished run(s). If so, when the EcDsDeletionCleanup utility is invoked normally (see 4.7.10.4.1), leftover information will be displayed and logged, and a menu will be displayed for the user to select how to run the cleanup:
Previous run was not completed, you can choose to:
- Rerun unfinished run only
- Start a new run which includes unfinished run(s)
- Quit
Select 1, 2 or 3:
Select 1 to complete the unfinished run(s) only. Cleanup will resume from the interrupted point in the previous run(s). (For example,, start to cleanup leftover XML files which had not been cleaned up in previous run(s).)
Select 2 to complete the unfinished run(s) and start a new run. The new run will clean up granules which have been marked for deletion since the unfinished previous run(s) and which meet the lag time and data type criteria for the new run.
Overview
Content Tools