You are here: Home User Information Facility Services HPSS Data Storage Service HPSS Archiving Tools (RHIC/LSST Users only)

HPSS Archiving Tools (RHIC/LSST Users only)

by Shigeki Misawa last modified Jan 28, 2008 04:31 PM
Documentation about the native HPSS archive tools that are available to RHIC and LSST users. (US Atlas users should use Dcache instead)

RHIC and LSST users can put data into HPSS using one of the following native HPSS archive client tools. (US Atlas users should use Dcache to put data into HPSS.) The three methods outlined below require an valid Kerberos TGT for authentication purposes. Instructions on how to get a Kerberos TGT can be found on the Overview of Kerberos web page.

Users should note that the HPSS system is specifically designed to handle large files (> 1GB) files, small files will result in extremely poor HPSS performance. For reference, small file performance of HPSS is measured at a FEW FILES PER MINUTE per tape drive. In contrast, for large files, HPSS performance is measured in excess of 50MB per second per tape drive. Since HPSS is a shared facility that handles high bandwidth data streams from the RHIC and LHC accelerators, in addition to archived user files, users are strongly encouraged to aggregate small files into larger files using tools like tar before archiving files.

Kerberized PFTP

The Kerberized pftp command, hpss_pftp (/usr/local/bin/hpss_pftp), is a ftp-like program that can be used to put data into HPSS, take data out of HPSS, and manipulate the HPSS file system name space. To start up Kerberized pftp, run hpss_pftp -A kerberos. Note that hpss_pftp is a wrapper  around  the native HPSS pftp command that  invokes the
pftp command with Kerberos support. Documentation for the native HPSS pftp command can be found here.


HSI

HSI is an interactive shell interface to HPSS. Since it is a relatively heavy weight application, this interface should be used sparingly. DO NOT use HSI in any scripts and avoid any operations that may involve many file/file system meta data lookups. To run HSI, just type hsi. Documentation for HSI can be found here.

HTAR

Htar is an tar like program that aggregates multiple files in the local file system into a single file that resides in HPSS. Documentation for htar can be found here.

Document Actions