Differences

This shows you the differences between two versions of the page.

--- cookbook:tsm [2018/10/18 22:03] – [Syncronization] Scott Cunliffe
+++ cookbook:tsm [2025/01/21 22:07] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Tivoli Storage Manager Sync Cookbook ======
-Tivoli Storage Manager (or TSM) is IBM's enterprise backup solution, and provides both a command line interface program that can be used to query the information required for synchronization, and ODBC driver that allows you to directly query data from the TSM database.
+Tivoli Storage Manager (or TSM) is IBM's enterprise backup solution, and provides both a [[technote:commandline|Command Line]] interface program that can be used to query the information required for synchronization and ODBC driver that allows you to directly query data from the TSM database.
 TSM is designed around several database tables that must be queried to get a complete list of tapes.
@@ Line 7: / Line 7: @@
 These tables are:
    * **The DRM Table**. This table lists all tapes are considered critical for recovery purposes.
-   * **The Volume Table**. This table lists all tapes that contain active data for a given TSM system. Any Volume that become scratch and is no longer managed by the DRM subsystem will be dropped from this table.
+   * **The Volume Table**. This table lists all tapes that contain active data for a given TSM system. Any [[object:volume|Volume]] that become scratch and is no longer managed by the DRM subsystem will be dropped from this table.
-   * **The Library Table**. This table is only available on servers that are considered library masters and lists all Volumes that are in an ATL environment. This table will also include scratch tapes that are currently in an ATL.
+   * **The Library Table**. This table is only available on servers that are considered library masters and lists all [[object:volume|Volumes]] that are in an ATL environment. This table will also include scratch tapes that are currently in an ATL.
-<note tip>The location field in the Volume and state field DRM tables do not represent the same value. The location field in the Volume table represents the perceived current location of a Volume, while the state field in the DRM table may represent the current location, but may also represent the destination location of a tape.
+<note tip>The location field in the [[object:volume|Volume]] and state field DRM tables do not represent the same value. The location field in the [[object:volume|Volume]] table represents the perceived current location of a [[object:volume|Volume]] , while the state field in the DRM table may represent the current location, but may also represent the destination location of a tape.
 </note>
-<note tip>TSM will drop Volumes from both the Volume and DRM tables when they become scratch. Although these Volumes may now be in an ATL and visible by querying the Library Master Server, they may also be elsewhere, including a scratch rack and not appear in any TSM table. To identify any scratch tapes that may have evaded being recalled from off-site, it is good practice to set the SCRATCH field during each synchronization and to periodically run the [[cli:tmss10movescratch|TMSS10MoveScratch]] command</note>
+<note tip>TSM will drop [[object:volume|Volumes]] from both the [[object:volume|Volume]] and DRM tables when they become scratch. Although these [[object:volume|Volumes]] may now be in an ATL and visible by querying the Library Master Server, they may also be elsewhere, including a scratch rack and not appear in any TSM table. To identify any scratch tapes that may have evaded being recalled from off-site, it is good practice to set the SCRATCH field during each synchronization and to periodically run the [[cli:tmss10movescratch|TMSS10MoveScratch]] command</note>
+===== Data Extraction From TSM =====
+The first step in synchronizing your TSM Library with TapeTrack is to extract the [[object:volume|Volume]] data from your TSM instance.
+Using the Tivoli Storage Manager administrative tool dsmadmc, the [[object:volume|Volume]] data needs to be extracted from both the DRM and [[object:volume|Volume]] tables and exported into csv file format.
+{{dsmadmc_extract.png}}
 ===== The dsmadmc command =====
@@ Line 20: / Line 27: @@
 === From the DRM table: ===
-<sxh>
-dsmadmc -dataonly=yes -id=userid -pa=password -tcpserveraddress=tsm01.gazillabyte.local -displaymode=table -outfile=TSM-DRM.csv -commadelimited "select volume_name,state,stgpool_name from drmedia"
+From the DRM table, select the fields:
-</sxh>
+  * volume_name: Used to determine the [[object:volume|Volume-ID]] in TapeTrack.
+  * state: Used to determine the location the [[object:volume|Volume]] should be.
+  * stgpool_name: Used to set the [[object:volume|Volume]] [[object:volume|Description]].
+<code>
+dsmadmc -dataonly=yes -id=userid -pa=password -tcpserveraddress=tsm01.gazillabyte.local \
+        -displaymode=table -outfile=TSM-DRM.csv -commadelimited "select volume_name,state,stgpool_name from drmedia"
+</code>
 === From the Volume table: ===
-<sxh>
-dsmadmc -dataonly=yes -id=userid -pa=password -tcpserveraddress=tsm01.gazillabyte.local -displaymode=table -outfile=TSM-Volumes.csv -commadelimited "select volume_name,location,stgpool_name from volumes"
-</sxh>
-===== Syncronization =====
+From the [[object:volume|Volume]] table, select the fields:
+  * volume_name: Used to determine the [[object:volume|Volume-ID]] in TapeTrack
+  * location: Used to determine the location the [[object:volume|Volume]] is.
+  * stgpool_name: Used to set the [[object:volume|Volume]] [[object:volume|Description]].
+<code>
+dsmadmc -dataonly=yes -id=userid -pa=password -tcpserveraddress=tsm01.gazillabyte.local \
+        -displaymode=table -outfile=TSM-VOL.csv -commadelimited "select volume_name,location,stgpool_name from volumes"
+</code>
+Output CSV files should be written to a directory where the TapeTrack Sync command can access them as input files for the Sync process.
+===== Synchronization =====
 <note important>You will need to install the [[cli:sync_suite|TapeTrack Sync software]] to complete these instructions.</note>
-Synchronization with TapeTrack is performed by calling the [[cli:TMSS10Sync|TMSS10Sync]] command line program, along with:
+Synchronization with TapeTrack is performed by calling the [[cli:TMSS10Sync|TMSS10Sync]] command-line program, along with:
-  - The CSV file produced from the DRM table
+  * The CSV file produced from the DRM table
-  - The CSV file produced from the Volume table
+  * The CSV file produced from the Volume table
-  - A synchronization definition file that instructs the program how to interpret the TSM table outputs.
+  * A synchronization definition file/s that instructs the program how to interpret the TSM table outputs.
+{{tsm_sync.png}}
+The [[cli:TMSS10Sync|TMSS10Sync]] is called individually for the DRM and [[object:volume|Volumes]] output csv files.
 === Example Command Line Arguments ===
@@ Line 42: / Line 68: @@
 Sync command line using DRM output
-<sxh>
+<code>
 TMSS10Sync -a -d TSM-DRM.ttidef -S user:-password@serveraddress < TSM-DRM.csv
-</sxh>
+</code>
 Where:
-  * ''-a'' tells the program to add new tape volumes if they are encountered.
+  * ''-a'' Add new [[object:volume|Volumes]] if they are encountered.
-  * ''-d'' is the path to the Synchronization Definition File.
+  * ''-d'' Path to the Synchronization Definition File (.ttidef file).
-  * ''-S''tells the program what [[cli:server_argument|Server]] to connect to.
+  * ''-S'' The Logon string argument sets the [[cli:server_argument|Server Logon Information]].
   * ''TSM-DRM.csv'' is the dsmadmc output from the DRM table
 Sync command line using Volume output
-<sxh>
+<code>
-TMSS10Sync -a -d TSM-DRM.ttidef -S user:-password@serveraddress < TSM-VOL.csv
+TMSS10Sync -a -d TSM-VOL.ttidef -S user:-password@serveraddress < TSM-VOL.csv
-</sxh>
+</code>
 Where:
-  * ''-a'' tells the program to add new tape volumes if they are encountered.
+  * ''-a'' Add new tape volumes if they are encountered.
-  * ''-d'' is the path to the Synchronization Definition File.
+  * ''-d'' Path to the Synchronization Definition File.
-  * ''-S'' tells the program what [[cli:server_argument|Server]] to connect to.
+  * ''-S'' The Logon string argument sets the [[cli:server_argument|Server Logon Lnformation]].
   * ''TSM-VOL.csv'' is the dsmadmc output from the Volume table
 === Example batch file showing Sync process with dsmadmc extracted data and process output ===
-<sxh>
+<code>
 #!/bin/bash
@@ Line 91: / Line 117: @@
 # Synchronize off the DRM table.
 #
-TMSS10Sync -a -d $ETC/TSM-DRM.ttidef -S user:-$PW@tapetrack.gazillabyte.local < $OUTDIR/TSM-DRM.csv > $OUTDIR/TSM-DRM-Sync.stdout 2> $OUTDIR/TSM-DRM-Sync.stderr \
+TMSS10Sync -a -d $ETC/TSM-DRM.ttidef -S user:-$PW@tapetrack.gazillabyte.local < $OUTDIR/TSM-DRM.csv > $OUTDIR/TSM-DRM-Sync.stdout 2> $OUTDIR/TSM-DRM-Sync.stderr
 echo "Step 3: Extracting Volume information from TSM Server: " $TSMIP
@@ Line 113: / Line 139: @@
  echo TMSS10MoveScratch -S user:-$PW@tapetrack.gazillabyte.local -R "LIBR"
-</sxh>
+</code>
-=== Example Synchronization Definition ===
+=== Example Synchronization Definition File (.ttidef file) ===
 <note tip>TSM is highly configurable, but by default has the following locations that a tape may be in:
-  * **Mountable** The Volume is available for use.
+  * **Mountable**: The Volume is available for use.
-  * **Courier** The Volume is waiting to be picked-up to be take off-site.
+  * **Courier**: The Volume is waiting to be picked-up to be taken offsite.
-  * **Courier Retrieve** The Volume has been picked-up to be taken off-site.
+  * **Vault**: The Volume is currently believed to be offsite.
-  * **Vault** The Volume is currently believed to be off-site.
+  * **Vault Retrieve**: The Volume needs to be returned from offsite.
-  * **Vault Retrieve** The Volume needs to be returned from off-site.
+  * **Courier Retrieve**: The Volume has been picked-up from offsite to return onsite.
-The following definition file assumes default TSM locations and two Repositories (OFFS and LIBR) in TapeTrack
+The following definition files assume default TSM locations and two Repositories (OFFS and LIBR) in TapeTrack
 </note>
-<sxh>
+===TSM-VOL TTIDEF File===
-#
+<code>
-# Set input file
+# Volume TTIDEF
-#
-SetFile("TSM-Volume.csv");
+# Set CSV delimiter
-#
+SetCSVDelimiter(",");
-# Set the Customer and Media as literal values as they never change
 #
+# Set Customer and Media values as they remain constant
 SetLiteral(CUSTOMER, "ACME");
 SetLiteral(MEDIA, "LTO");
-#
-# Set the delimiter to a CSV
+# Extract Volume-ID
-#
-SetCSVDelimiter(",")
-#
-# Set the Volume ID
-#
 Extract(VOLUME, 1, 8, 0);
-#
-# Set the volume description from pool name
 #
+# Extract Scratch value for comparison
+Extract(SCRATCH, 2, 20, 0);
+# Extract Description
 Extract(DESCRIPTION, 3, 100, 0);
+RemoveSpaces(DESCRIPTION);
+# If Scratch = Vault skip, anything else set SCRATCH to true
+AddTranslation(SCRATCH, "VAULT", "!Bypass!");
+AddTranslation(SCRATCH, "*", "true");
+# Exclude any lines starting with . or /
+AddString(EXCLUSION, 0, ".*");
+AddString(EXCLUSION, 0, "/*");
+</code>
+===TSM-DRM TTIDEF File===
+<code>
+# DRM TTIDEF
+# Set CSV delimiter
+SetCSVDelimiter(",");
 #
-# Get the repository value
+# Set Customer and Media values as they remain constant
-#
+SetLiteral(CUSTOMER, "ACME");
+SetLiteral(MEDIA, "LTO");
+# Extract Volume-ID
+Extract(VOLUME, 1, 8, 0);
+# Extract Repository value
 Extract(REPOSITORY, 2, 20, 0);
-#
+RemoveSpaces(REPOSITORY);
-# Translate repository value based off initial value
-# If Repository = Vault, Courier or Courier Retrieve set Repository to OFFS
+# Extract Scratch value
-# If Repository = Anything else (Mountable, Vault Retrieve) set Repository to LIBR
+Extract(SCRATCH, 2, 20, 0);
-#
+RemoveSpaces(SCRATCH);
-AddTranslation(REPOSITORY, "Vault", "OFFS");
-AddTranslation(REPOSITORY, "Courier*", "OFFS");
+# Translate Scratch value
+AddTranslation(SCRATCH, "VAULTRETRIEVE", "true");
+AddTranslation(SCRATCH, "*", "false");
+# Translate Repository value
+AddTranslation(REPOSITORY, "VAULT", "OFFS");
 AddTranslation(REPOSITORY, "*", "LIBR");
-#
+AddSkipOnRepositoryChange("LIBR", "OFFS", "*");
-# Set all volumes to scratch
-#
+# Exclude any lines starting with . or /
-SetLiteral(SCRATCH, "false");
+AddString(EXCLUSION, 0, ".*");
-</sxh>
+AddString(EXCLUSION, 0, "/*");
+</code>
+<- :cookbook:networker|NetWorker Sync Cookbook ^ :cookbook:introduction|Cookbook ^ :cookbook:veeam|VEEAM Sync Cookbook ->
-<- :cookbook:networker|NetWorker Sync Cookbook ^ :cookbook:introduction|Cookbook ^ ->
+{{tag> cookbook}}