Week of
Mon, Dec 10

  • HPSS is back online    12/11/2018
    Tue Dec 11 17:04:25 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    HPSS is up and running.

  • Authentication Migration Status - rftpexp available    12/11/2018
    Tue Dec 11 16:15:03 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov, sdcc_users-l@lists.bnl.gov, bnl-shared-tier3-l@lists.bnl.gov

    Following the IPA migration today, there was an authentication issue on the rftpexp servers. This has been resolved. Please report any further issues via RT.

    Earlier today, some of the IPA servers crashed. This affected access to RHIC SSH gateways and the RHIC processor farm between approximately 12:00 noon and 1:30 PM. The cause of the crash was identified, and addressed at 1:30 PM. Please report any issues accessing the SSH gateways or processor farm interactive systems via RT.

    RACF/SDCC Staff

Week of
Mon, Dec 3

  • HPSS service down    12/7/2018
    Fri Dec 7 16:43:34 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: HPSS service will be shutdown for maintanance update.

    Duration: From Tuesday December 11 7 AM, expect to be up before 5 PM.

    Group Responsible: HPSS

    Affected Area: HPSS services, both read and write.

    Expected User Impact: All HPSS services

    Maintenance Type: Downtime Submitted By: David Yu, dyu@bnl.gov

    Description: Apply patch code for supporting LTO-8. Also, to give HPSS a refresh before RHIC run 19.

  • ***IMPORTANT*** SDCC/RACF Authentication Changes on 12/11    12/7/2018
    Fri Dec 7 16:03:35 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov, sdcc_users-l@lists.bnl.gov, bnl-shared-tier3-l@lists.bnl.gov

    On December 11, 2018 at 10:00AM EST, the SDCC/RACF will move our password protected systems and services to a new, unified authentication system: IPA (Identity, Policy, and Audit). In order to access facility services after the cutover time, users will need to register a password in the new IPA authentication system.

    The following two methods will be made available at the cutover time on 12/11 to allow users to register a password in the IPA system. Users may choose to use either method:

    1. Interactive Command Line a) Login to the RHIC (rssh.rhic.bnl.gov), ATLAS (atlasgw.usatlas.bnl.gov), or SDCC (ssh.sdcc.bnl.gov) SSH gateways using your SSH public key (as usual) b) Once on a gateway, run "pwchange" - this command will need to be used only once for this transition process c) Follow the prompts to register your password. 2. Web Interface a) Access the following URL: https://migration.sdcc.bnl.gov/passwd As this page is protected, you will need to use your current password and current Identity Provider, be it RHIC, USATLAS or SDCC. b) Fill out the form to register your password.

    Examples of facility services that will require the new IPA password beginning on December 11 include the following: a) Password-based access to interactive nodes b) RHIC & USATLAS AFS file systems c) Access to facility web pages protected by password 1) This includes the ssh key upload page, and services like Gitea d) HPSS HSI/HTAR e) BNL Box

    Examples of services that will *NOT* be affected by the transition: a) SSH gateway logins (via SSH public key) b) RCF mail/webmail (this interface has and will continue to use a separate password)

    For users of RHIC or USATLAS AFS services, the following changes will also need to be made on December 11: a) For technical reasons, AFS users on interactive farm nodes at the RACF/SDCC facility will no longer automatically receive AFS tokens upon login (when logging in via password). As such, users will need to run "aklog" after logging in to obtain an AFS token b) The rhic.bnl.gov and usatlas.bnl.gov AFS server restarts at the time of transition will invalidate all existing AFS tokens. You'll need to obtain a new token if you require one. There will also be a momentary (up to a few minutes) interruption in AFS service for non-replicated volumes while the AFS fileservers are restarted c) Authentication for the RHIC and USATLAS AFS cells will be moved to the SDCC.BNL.GOV Kerberos5 realm. External users will need to authenticate to the SDCC.BNL.GOV Kerberos realm before running aklog to obtain an AFS token. Documentation on setting up your krb5.conf (Linux), or krb5.ini files for this realm are available here: https://www.racf.bnl.gov/docs/authentication/new-sdcc-kerberos-realm-configuration-files d) External AFS users will need to be running OpenAFS 1.6.5 or newer client software, as the RHIC and USATLAS AFS cells will be switching to AES-based AFS tokens

    This announcement is also available here: https://www.racf.bnl.gov/docs/authentication/migration

    SDCC/RACF Staff

Week of
Mon, Nov 26

  • **CORRECTED DATES FOR STAR** Interactive Processor Farm Node Shutdowns     11/30/2018
    Fri Nov 30 15:46:15 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov, bnl-shared-tier3-l@lists.bnl.gov

    The STAR, PHENIX, LBNE, DayaBay, Astro, and ATLAS T3 interactive processor farm hosts will be shutdown for maintenance next week, according to the following schedule:

    1. 10:00 AM-11:30 AM - Mon 12/3 PHENIX - rcas2061-2069 LBNE - lbne0001-0002 DayaBay - daya0001-0002 Astro - astro0001-0011 ATLAS T3 - acas0001-0004, spar0101-0102

    2. 10:00 AM-11:30 AM - Tue 12/4 PHENIX - rcas2070-2079 LBNE - lbne0003 DayaBay - daya0004-0005, daya0009 Astro - astro0022-0044 ATLAS T3 - acas0005-0008, spar0103-spar0104

    3. 10:00 AM-11:30 AM - Wed 12/5 STAR - rcas6005-6008

    4. 10:00 AM-11:30 AM - Thu 12/6 STAR - rcas6009-rcas6010

    Please logout of the affected systems before the scheduled maintenance. Hosts will be brought back online within 1.5 hours of being shutdown

    Chris Hollowell (hollowec@bnl.gov)

  • Interactive Processor Farm Node Shutdowns    11/30/2018
    Fri Nov 30 15:25:23 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov, bnl-shared-tier3-l@lists.bnl.gov

    The STAR, PHENIX, LBNE, DayaBay, Astro, and ATLAS T3 interactive processor farm nodes will be shutdown for maintenance next week, according to the following schedule:

    1. 10:00 AM-11:30 AM - Mon 12/3 STAR - rcas6005-6008 PHENIX - rcas2061-2069 LBNE - lbne0001-0002 DayaBay - daya0001-0002 Astro - astro0001-0011 ATLAS T3 - acas0001-0004, spar0101-0102

    2. 10:00 AM-11:30 AM - Tue 12/4 STAR - rcas6009-6010 PHENIX - rcas2070-2079 LBNE - lbne0003 DayaBay - daya0004-0005, daya0009 Astro - astro0022-0044 ATLAS T3 - acas0005-0008, spar0103-0104

    Please logout of the affected systems before the scheduled maintenance.

    Chris Hollowell (hollowec@bnl.gov)

  • Phenix web Server maintenance    11/27/2018
    Tue Nov 27 09:25:05 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: adminweb04, adminweb07 and adminweb08 servers will be brought down for Maintenance on Wednesday morning 11/28/18.

    Duration: 11/28/18 8:00AM -8:30AM

    Group Responsible: General Services GCE

    Affected Area: Web services

    Expected User Impact: Connection to Phenix web pages will be terminated and unavailable.

    Maintenance Type: Downtime Submitted By: Joe Frith jfrith@bnl.gov

    Description: Servers being migrated to new cluster.

  • Admin Web Server Maintenance on Tuesday morning 11/27/18.    11/26/2018
    Mon Nov 26 15:23:06 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: Adminweb01&03,www2,ww4 and www_rcf servers will be brought down for Maintenance on Tuesday morning 11/2718.

    Duration: 11/27/18 8:00AM -8:30AM

    Group Responsible: General Services GCE

    Affected Area: Web services

    Expected User Impact: Connection to RACF/RHIC internal web pages will be terminated and unavailable.

    Maintenance Type: Downtime Submitted By:Joe Frith jfrith@bnl.gov

    Description: Servers being migrated to new cluster.

Week of
Mon, Nov 12

  • NX06 and NX07 Maintenance    11/16/2018
    Fri Nov 16 14:10:36 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: NX06 and NX07 servers will be brought down for Maintenance on Tuesday morning 11/20/18.

    Duration: 11/20/18 8:00AM -8:30AM

    Group Responsible: General Services GCE

    Affected Area: NX services

    Expected User Impact: Connection to servers will be terminated, please make sure to save your work.

    Maintenance Type: Downtime Submitted By: Joe Frith jfrith@bnl.gov

    Description: Servers being migrated to new cluster.

  • NX01 and NX02 Maintenance    11/16/2018
    Fri Nov 16 11:31:10 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: NX01 and NX02 servers will be brought down for Maintenance on Monday morning 11/19/18.

    Duration: 11/19/18 8:00AM -8:30AM

    Group Responsible: General Services GCE

    Affected Area: NX services

    Expected User Impact: Connection to servers will be terminated, please make sure to save your work.

    Maintenance Type: Downtime Submitted By: Joe Frith jfrith@bnl.gov

    Description: Servers being migrated to new cluster.

  • cssh, rftpexp maintenance    11/13/2018
    Tue Nov 13 09:01:15 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: Correction rftpexp02 not rftpexp01 will be taken offline:

    cssh03, cssh04,and rftpexp02 will be taken offline for maintenance. Duration: 11/14/18 8:00AM -8:30AM Group Responsible: General services Affected Area: SSH and FTP connections Expected User Impact: User connected to servers will be disconnected and will need to reconnect. Maintenance Type: Service Interruption Submitted By: Joe Frith jfrith@bnl.gov Description: Servers being migrated to new cluster.

  • cssh, rftpexp maintenance    11/13/2018
    Tue Nov 13 08:41:38 EST 2018

    This item has been posted to rhic-rcf-l@lists.bnl.gov

    Summary: cssh03, cssh04,and rftpexp01 will be taken offline for maintenance.

    Duration: 11/14/18 8:00AM -8:30AM Group Responsible: General services Affected Area: SSH and FTP connections Expected User Impact: User connected to servers will be disconnected and will need to reconnect. Maintenance Type: Service Interruption Submitted By: Joe Frith jfrith@bnl.gov Description: Servers being migrated to new cluster.

Last Modified December 11, 2018
RACF Staff