You are here: Home Experiment Information US ATLAS Grid Operations How to Delete a Data Set

How to Delete a Data Set

by Hironori Ito last modified Jan 06, 2010 02:03 PM
It will show how to delete a given dataset in your local site. Be very careful with this instruction as it can delete all record of a dataset from your site.
  1. Get the program listed below:
  2. Make sure to have a DQ2 client setup since it will use DQ2 API.  ( See  https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2ClientsHowTo) (Check if you can use dq2 client commands.)
  3. Make sure to have srmrm command in your path.  (If not found, try installing OSG client package.  See at https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/ClientInstallationGuide )
  4. Delete a dataset at given site by
  5.  python deleteDatasetSite.py --dataset=DATASETNAME --site=SITENAME --logFile=LOGFILENAME
    
    • If logFile is not specified, it will default to /tmp/log_deletedataset.log
  6. Example.
    • Delete data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293 at DUKE_LOCALGROUPDISK
      • To see if a file exists in LFC.
      • dq2-ls -L DUKE_LOCALGROUPDISK -fp data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293
        
        data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293
        srm://atlfs02.phy.duke.edu/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
        total files: 1
        total size: 6838347
        date: 2009-12-02 11:12:11
        
        
      • To see if a file exists in SE
      • srmls srm://atlfs02.phy.duke.edu:8443/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
          6838347 /srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
        
      • Delete this dataset by
      • python deleteDatasetLocal.py --dataset=data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293 --site=DUKE_LOCALGROUPDISK
        Using default log file /tmp/log_deletedataset.log
        Delete data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293 from DUKE_LOCALGROUPDISK
        Deleting replica from the central replica catalog
        Deleting srm://atlfs02.phy.duke.edu/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1 from LFC t3lfcv01.usatlas.bnl.gov
        Remvoing files from SRM by srmrm -2 srm://atlfs02.phy.duke.edu:8443/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
        srmrm -2 srm://atlfs02.phy.duke.edu:8443/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
        
      • Just to see if it is really deleted by repeating the same commands from the above
      • dq2-ls -L DUKE_LOCALGROUPDISK -fp data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293 
        data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293
        [ ]     data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1      622C2E20-31DF-DE11-AF31-0030489473F2    ad:c6c318c1     6838347
        total files: 1
        total size: 6838347
        date: 2009-12-02 11:12:11
        
        srmls srm://atlfs02.phy.duke.edu:8443/srm/v2/server?SFN=/srv/data/srm/atlaslocalgroupdisk/data09_900GeV/AOD/f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293/data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293._0001.1
        srm client error:
        Return status:
         - Status code:  SRM_FAILURE
         - Explanation:  null
        
      • Also, DUKE_LOCALGROUPDISK is no longer registered as a replica site in DDM central catalog.
      • dq2-list-dataset-replicas data09_900GeV.00141403.physics_MinBias.merge.AOD.f181_m293
         INCOMPLETE:
         COMPLETE: AGLT2_DATADISK,ANL_LOCALGROUPDISK,BNL-OSG2_DATADISK,CERN-PROD_DATADISK,CERN-PROD_TZERO,DESY-HH_DATADISK,FZK-LCG2_DATADISK,GOEGRID_DATADISK,GRIF-SACLAY_DATADISK,IFIC-LCG2_DATADISK,IN2P3-CC_DATADISK,INFN-MILANO-ATLASC_DATADISK,INFN-T1_DATADISK,MWT2_DATADISK,NDGF-T1_DATADISK,NET2_DATADISK,NIKHEF-ELPROD_DATADISK,PIC_DATADISK,RAL-LCG2_DATADISK,SLACXRD_DATADISK,SWT2_CPB_DATADISK,TAIWAN-LCG2_DATADISK,TOKYO-LCG2_DATADISK,TRIUMF-LCG2_DATADISK,UKI-NORTHGRID-LIV-HEP_DATADISK,UKI-SCOTGRID-GLASGOW_DATADISK,WISC_DATADISK
        
  7. FAQ
    1. Make sure that your VOMs proxy is mapped to the local account with "write" permission for files you are intended to delete.
    2. Make sure that your VOMS role can delete LFC entries.
      1. To check your voms role, look at voms-proxy-info -all
      2. To check ACL of file in LFC, use lfc-getacl  
Document Actions
Filed under: