You are here: Home Experiment Information US ATLAS Grid Development AutoPyFactory

AutoPyFactory

by Jose Caballero last modified Jul 05, 2012 10:38 AM

This documentation is obsolete and not maintained. Documentation is now in SVN, distributed within the rest of AutoPyFactory package. It can be found in SVN, or after checking out, underneath directory docs/

Index

Introduction
Design
Deployment
Deployment using RPM
Deployment using RPM but running as user
Deployment on user's home directory
Configuration
factory.conf
queues.conf
Monitoring
AutoPyFactory and the Cloud
Architecture
Software deployed in the image
Issues during tests
New wrapper
Motivation
Wrapper.sh
Plug-ins architecture
Contact
Talks and publications

Introduction

 

ATLAS, one of the experiments at LHC at CERN, is one of the largest users of grid computing infrastructure. As this infrastructure is now a central part of the experiment's computing operations, considerable efforts have been made to use this technology in the most efficient and effective way, including extensive use of pilot job based frameworks.

In this model the experiment submits 'pilot' jobs to sites without payload. When these jobs begin to run they contact a central service to pick-up a real payload to execute.

The first generation of pilot factories were usually specific to a single VO, and were very bound to the particular architecture of that VO. A second generation is creating factories which are more flexible, not tied to any particular VO, and provide for more features other than just pilot submission (such as monitoring, logging, profiling, etc.)

AutoPyFactory has a modular design and is highly configurable. It is able to send different types of pilots to sites, able to exploit different submission mechanisms and different charateristics of queues at sites. It has excellent integration with the PanDA job submission framework, tying pilot flows closely to the amount of work the site has to run. It is able to gather information from many sources, in order to correctly conigure itself for a site and its decision logic can easily be updated.

Integrated into AutoPyFactory is a very flexible system for delivering both generic and specific wrappers which can perform many useful actions before starting to run end-user scientific applications, e.g., validation of the middleware, node profiling and diagnostics, monitoring and deciding what is the best end-user application that fits the resource.

AutoPyFactory now also has a robust monitoring system and we show how this has helped setup a reliable pilot factory service for ATLAS.

Design

AutoPyFactory can serve to different queues in different ways thanks to its modular design based on plug-ins. There are currently 5 types of plug-ins:

WMS Status Plug-in: it queries a given WMS system. E.g. PandaWMSStatusPlugin queries PanDA. The only requirements is the WMS must provide for an API. This API has to return the number of jobs in different status (ready, running, finished...) per queue, in such a way that information can be converted internally into APF nomenclature.

Batch Status Plug-in: it queries the specific batch system being used to submit jobs (or pilots). E.g. CondorBatchStatusPlugin queries condor (condor_q)

Scheduler Plug-in: it is the component in charge of making a decision based on the information provided by the two Status Plug-ins. It implements a given algorithm to decide how many jobs (or pilots) should be submitted next cyle. E.g. ActivatedSchedPlugin decides the number of new jobs based on the number of jobs in a ready status in the WMS service, with some restrictions to prevent burning a CE. FixedSchedPlugin always submits a fixed number of jobs. Etc.

Execution Plug-in: It is the component in charge of submitting new jobs (or pilots), based on the decision made by the Sched Plug-in. Examples are CondorGT2BatchSubmitPlugin, CondorGT5BatchSubmitPlugin, CondorLocalBatchSubmitPlugin, CondorCREAMBatchSubmitPlugin, and CondorEC2BatchSubmitPlugin.

Configuration Plug-in: a plug-in to retrieve extra configuration files. E.g PandaConfigPlugin queries SchedConfig.

Deployment

Deployment using RPM

 

 

 

Installation as root via RPMs has now been quite simplified. These instructions assume Red Hat Enterprise Linux 5.X (and derivates) and the system Python 2.4.3. Other distros and higher Python versions should work with some extra work.

1) Install and enable a supported batch system. Condor is the current supported default. Software available from http://www.cs.wisc.edu/condor/. Condor/Condor-G setup and configuration is beyond the scope of this documentation. Ensure that it is working properly before proceeding.

2) Install a grid client and set up the grid certificate+key under the user APF will run as. Please read the CONFIGURATION documentation regarding the proxy.conf file, so you see what will be needed. Make sure voms-proxy-* commands work properly.

3) Add the racf-grid YUM repo to your system:

$ rpm -ivh http://dev.racf.bnl.gov/yum/grid/production/rhel/5Client/x86_64/racf-grid-release-0.9-1.noarch.rpm

The warning about NOKEY is expected. This release RPM sets up YUM to point at our repository, and installs the GPG key with which all our RPMs are signed. By default the racf-grid-release RPM sets our production repository to enabled (see /etc/yum.repos.d/racf-grid-production.repo ). If you are testing APF and want to run a pre-release version, enable the racf-grid-development or racf-grid-testing repository.

4) If you will be performing local batch system submission (as opposed to remote submission via grid interfaces) you must confirm that whatever account you'll be submitting as exists on the batch cluster.

5) Install the APF RPM:

$ yum install autopyfactory

This performs several setup steps that otherwise would need to be done manually:

Creates 'apf' user that APF will run under.
Enables the factory init script via chkconfig.
Pulls in the panda userinterface Python library RPM from our repository.
Pulls in the python-simplejson RPM from the standard repository

6) Configure APF queues/job submission as desired. Read the CONFIGURATION documentation in order to do this. Be sure to configure at least one queue in order to test function.

7) Start APF:

$ /etc/init.d/factory status

Look at the output of ps to see that APF is running under the expected user, e.g.:

apf    22106 1.3 0.1 318064 12580 pts/2  Sl 17:13 0:00 /usr/bin/python /usr/bin/factory.py  conf /etc/apf/factory.conf  debug  sleep=60  runas=apf  log=/var/log/apf/apf.log

Tail the log output and look for problems.

$ condor_q | grep apf

Deployment using RPM but running as user

Deployment on user's home directory

 

 

User installation assumes that APF will be installed in the users home directory using the standard Python distutils setup commands. It assumes that pre-requisites have already been installed and properly configured, either within the user's home directory or on the general system.

Prerequisites:
Python
Condor (Condor-G)
Panda Client library
simplejson

Configuration

factory.conf

 

 

baseLogDir where outputs from pilots are stored
NOTE: No trailing '/'!!!
baseLogDirUrl where outputs from pilots are available via http
NOTE: No trailing '/'!!!
baseLogHttpPort What port to run the HTTP server to export logs.
Make sure this matchesthe value in the baseLogDirUrl.
batchstatus.condor.sleep time the BatchStatus Plugin waits between cycles.
Value is in seconds.
batchstatus.maxtime maximum time while the info is considered reasonable.
If info stored is older than that, is considered not valid, and some NULL output will be returned.
cycles maximum number of times the queues will loop.
None means forever.
cleanlogs.keepdays maximum number of days the condor logs will be kept, in case they are placed in a subdirectory for an APFQueue that is not being currently managed by AutoPyFactory.
For example, an apfqueue that has been created and used for a short amount of time, and it does not exist anymore. Still the created logs have to be cleaned at some point...
factoryId Name that the factory instance will have in the APF web monitor.
Make factoryId something descriptive and unique for your factory,
for example <site>-<host>-<admin> (e.g. BNL-gridui11-jhover)
factoryAdminEmail Email of the local admin to contact in case of a problem with an specific APF instance.
factory.sleep sleep time between cycles in mainLoop in Factory object.
Value is in seconds.
factoryUser account under which APF will run
monitorURL URL for the web monitor
logserver.enabled determines if batch logs are exported via HTTP.
Valid values are True|False
logserver.index determines if automatic directory indexing is allowed when log directories are browsed.
Valid values are True|Faose
proxyConf local path to the configuration file for automatic proxy management.
NOTE: must be a local path, not a URI.
proxymanager.enabled to determine if automatic proxy management is used or not.
Accepted values are True|False
queueConf URI plus path to the configuration file for APF queues.
NOTE: Must be expressed as a URI (file:// or http://)
versionTag APF version number as it will be displayed in the web monitor
wmsstatus.maximum maximum time while the info is considered reasonable. If info stored is older than that, is considered not valid, and some NULL output will be returned.
wmsstatus.panda.sleep time the WMSStatus Plugin waits between cycles.
Value is in seconds.

queues.conf

 

Generic variables override determines if values from this config file have precedence over the same values comming from different sources of information. If True then schedconfig does not clobber configuration file values. Valid values are True|False.
cloud is the cloud this queue is in. You should set this to suppress pilot submission when the cloud goes offline N.B. Panda clouds are UPPER CASE, e.g., UK
vo Virtual Organization
grid Grid middleware flavor at the site. (e.g. OSG, EGI, NorduGrid)
batchqueue the Batch system related queue name. E.g. the PanDA queue name (formerly called nickname)
wmsqueue the WMS system queue name. E.g. the PanDA siteid name
enabled determines if each queue section must be used by AutoPyFactory or not. Allows to disable a queue without commenting out all the values. Valid values are True|False.
status can be "test", "offline" or "online"
queue.sleep sleep time between cycles in APFQueue object. Value is in seconds.
autofill says if the info from this filled should be completed with info from a ConfigPlugin object
cleanlogs.keepdays maximum number of days the condor logs will be kept

 

WMS Status Plugin variables wmsstatusplugin WMS Status Plugin.

 

Config Plugin variables configplugin Config Plugin. Optional. E.g. Panda.

 

Batch Submit Plugin variables batchstatusplugin Batch Status Plugin.

 

Sched Plugin variables schedplugin specific Scheduler Plugin implementing the algorithm deciding how many new pilots to submit next cycle.
Configuration for Activated plugin sched.activated.default default number of pilots to be submitted when the context information does not exist is not reliable To be used in Activated Scheduler Plugin.
sched.activated.max_jobs_torun maximum number of jobs running simoultaneously. To be used in Activated Scheduler Plugin.
sched.activated.max_pilots_per_cycle maximum number of pilots to be submitted per cycle. To be used in Activated Scheduler Plugin.
sched.activated.min_pilots_per_cycle minimum number of pilots to be submitted per cycle. To be used in Activated Scheduler Plugin.
sched.activated.min_pilots_pending minimum number of pilots to be idle on queue waiting to start execution. To be used in Activated Scheduler Plugin.
sched.activated.max_pilots_pending maximum number of pilots to be idle on queue waiting to start execution. To be used in Activated Scheduler Plugin.
sched.activated.testmode.enabled Boolean variable to trigger special mode of operation when the wmssite is in in status = test
sched.activated.testmode.pilots number of pilots to submit when the wmssite is in status = test and sched.activated.testmode.enabled is True
Configuration for Fixed plugin sched.fixed.pilotspercycle fixed number of pilots to be submitted each cycle, when using the Fixed Scheduler Plugin.
Configuration for Simple plugin sched.simple.default default number of pilots to be submitted when the context information does not exist or is not reliable. To be used in Simple Scheduler Plugin.
sched.simple.maxpendingpilots maximum number of pilots to be idle on queue waiting to start execution. To be used in Simple Scheduler Plugin.
sched.simple.maxpilotspercycle maximum number of pilots to be submitted per cycle. To be used in Simple Scheduler Plugin.
Configuration for SimpleNQueue plugin sched.simplenqueue.default default number of pilots to be submitted when the context information does not exist or is not reliable. To be used in SimpleNQueue Scheduler Plugin.
sched.simplenqueue.maxpilotspercycle maximum number of pilots to be submitted each cycle. To be used in SimpleNQueue Scheduler Plugin.
sched.simplenqueue.nqueue desired number of pilots in idle status waiting in queue to start execution. To be used in SimpleNQueue Scheduler Plugin. sched.simplenqueue.depthboost = multiplying factor which allows more pilots to be submitted than nqueue if there are sufficient activated jobs - helps when the jobs are short
sched.simplenqueue.pilotlimit sets a hard limit on the total number of pilots at a site, active + queued
sched.simplenqueue.transferringlimit sets a limit on the number of jobs in transferring status allowed at a site - when this limit is reached no pilots will be submitted to allow backlogs to clear
Configuration for Trivial plugin sched.trivial.default default number of pilots to be submitted when the context information does not exist or is not reliable. To be used in Trivial Scheduler Plugin.

 

Batch Submit Plugin batchsubmitplugin Batch Submit Plugin. Currently available options are: CondorGT2, CondorGT5, CondorLocal, CondorEC2, CondorCREAM.
Configuration for condorgt2 batchsubmit.condorgt2.gridresource name of the CE (e.g. gridtest01.racf.bnl.gov/jobmanager-condor)
batchsubmit.condorgt2.submitargs list of command line input options to be included in the submission command *verbatim* e.g. batchsubmit.condorgt2.submitargs = -remote my_schedd will drive into a command like condor_submit -remote my_schedd submit.jdl
batchsubmit.condorgt2.condor_attributes list of condor attributes, splited by comma, to be included in the condor submit file *verbatim* e.g. +Experiment = "ATLAS",+VO = "usatlas",+Job_Type = "cas" Can be used to include any line in the Condor-G file that is not otherwise added programmatically by AutoPyFactory. Note the following directives are added by default:
batchsubmit.condorgt2.environ list of environment variables, splitted by white spaces, to be included in the condor attribute environment *verbatim* Therefore, the format should be env1=var1 env2=var2 envN=varN split by whitespaces.
batchsubmit.condorgt2.proxy name of the proxy handler in proxymanager for automatic proxy renewal (See etc/proxy.conf) None if no automatic proxy renewal is desired.
Configuration for condorgt5 batchsubmit.condorgt5.gridresource name of the CE (e.g. gridtest01.racf.bnl.gov/jobmanager-condor)
batchsubmit.condorgt5.submitargs list of command line input options to be included in the submission command *verbatim* e.g. batchsubmit.condorgt2.submitargs = -remote my_schedd will drive into a command like condor_submit -remote my_schedd submit.jdl
batchsubmit.condorgt5.condor_attributes list of condor attributes, splited by comma, to be included in the condor submit file *verbatim* e.g. +Experiment = "ATLAS",+VO = "usatlas",+Job_Type = "cas" Can be used to include any line in the Condor-G file that is not otherwise added programmatically by AutoPyFactory. Note the following directives are added by default: transfer_executable = True should_transfer_files = YES when_to_transfer_output = ON_EXIT_OR_EVICT stream_output=False stream_error=False notification=Error copy_to_spool = false
batchsubmit.condorgt5.environ list of environment variables, splitted by white spaces, to be included in the condor attribute environment *verbatim* Therefore, the format should be env1=var1 env2=var2 envN=varN split by whitespaces.
batchsubmit.condorgt5.proxy name of the proxy handler in proxymanager for automatic proxy renewal (See etc/proxy.conf) None if no automatic proxy renewal is desired.
Configuration for condorcream batchsubmit.condorcream.webservice web service address (e.g. ce04.esc.qmul.ac.uk:8443/ce-cream/services/CREAM2)
batchsubmit.condorcream.submitargs list of command line input options to be included in the submission command *verbatim* e.g. batchsubmit.condorgt2.submitargs = -remote my_schedd will drive into a command like condor_submit -remote my_schedd submit.jdl
batchsubmit.condorcream.condor_attributes list of condor attributes, splited by comma, to be included in the condor submit file *verbatim* e.g. +Experiment = "ATLAS",+VO = "usatlas",+Job_Type = "cas" Can be used to include any line in the Condor-G file that is not otherwise added programmatically by AutoPyFactory. Note the following directives are added by default: transfer_executable = True should_transfer_files = YES when_to_transfer_output = ON_EXIT_OR_EVICT stream_output=False stream_error=False notification=Error copy_to_spool = false
batchsubmit.condorcream.environ list of environment variables, splitted by white spaces, to be included in the condor attribute environment *verbatim* Therefore, the format should be env1=var1 env2=var2 envN=varN split by whitespaces.
batchsubmit.condorcream.queue queue within the local batch system (e.g. short)
batchsubmit.condorcream.port port number.
batchsubmit.condorcream.batch local batch system (pbs, sge...)
batchsubmit.condorcream.gridresource grid resource, built from other vars using interpolation: batchsubmit.condorcream.gridresource = %(batchsubmit.condorcream.webservice)s:%(batchsubmit.condorcream.port)s/ce-cream/services/CREAM2 %(batchsubmit.condorcream.batch)s %(batchsubmit.condorcream.queue)s
batchsubmit.condorcream.proxy name of the proxy handler in proxymanager for automatic proxy renewal (See etc/proxy.conf) None if no automatic proxy renewal is desired.
Configuration for condorec2 batchsubmit.condorec2.gridresource ec2 service's URL (e.g. https://ec2.amazonaws.com/ )
batchsubmit.condorec2.submitargs list of command line input options to be included in the submission command *verbatim* e.g. batchsubmit.condorgt2.submitargs = -remote my_schedd will drive into a command like condor_submit -remote my_schedd submit.jdl
batchsubmit.condorec2.condor_attributes list of condor attributes, splited by comma, to be included in the condor submit file *verbatim*
batchsubmit.condorec2.environ list of environment variables, splitted by white spaces, to be included in the condor attribute environment *verbatim* Therefore, the format should be env1=var1 env2=var2 envN=varN split by whitespaces.
batchsubmit.condorec2.ami_id identifier for the VM image, previously registered in one of Amazon's storage service (S3 or EBS)
batchsubmit.condorec2.instance_type hardware configurations for instances to run on.
batchsubmit.condorec2.user_data up to 16Kbytes of contextualization data. This makes it easy for many instances to share the same VM image, but perform different work.
batchsubmit.condorec2.access_key_id path to file with the EC2 Access Key ID
batchsubmit.condorec2.secret_access_key path to file with the EC2 Secret Access Key
batchsubmit.condorec2.proxy name of the proxy handler in proxymanager for automatic proxy renewal (See etc/proxy.conf) None if no automatic proxy renewal is desired.
Configuration for condorlocal batchsubmit.condorlocal.submitargs list of command line input options to be included in the submission command *verbatim* e.g. batchsubmit.condorgt2.submitargs = -remote my_schedd will drive into a command like condor_submit -remote my_schedd submit.jdl
batchsubmit.condorlocal.condor_attributes list of condor attributes, splited by comma, to be included in the condor submit file *verbatim* e.g. +Experiment = "ATLAS",+VO = "usatlas",+Job_Type = "cas" Can be used to include any line in the Condor-G file that is not otherwise added programmatically by AutoPyFactory. Note the following directives are added by default: universe = vanilla transfer_executable = True should_transfer_files = YES when_to_transfer_output = ON_EXIT_OR_EVICT stream_output=False stream_error=False notification=Error periodic_remove = (JobStatus == 5 && (CurrentTime - EnteredCurrentStatus) > 3600) || (JobStatus == 1 && globusstatus =!= 1 && (CurrentTime - EnteredCurrentStatus) > 86400) To be used in CondorLocal Batch Submit Plugin.
batchsubmit.condorlocal.environ list of environment variables, splitted by white spaces, to be included in the condor attribute environment *verbatim* To be used by CondorLocal Batch Submit Plugin. Therefore, the format should be env1=var1 env2=var2 envN=varN split by whitespaces.
batchsubmit.condorlocal.proxy name of the proxy handler in proxymanager for automatic proxy renewal (See etc/proxy.conf) None if no automatic proxy renewal is desired.
GRAM2 variables globusrsl.gram2.globusrsl GRAM RSL directive. If this variable is not setup, then it will be built programmatically from all non empty globusrsl.gram2.XYZ variables. If this variable is setup, then its value will be taken *verbatim*, and all possible values for globusrsl.gram2.XYZ variables will be ignored.
globusrsl.gram2.globusrsladd custom fields to be added *verbatim* to the GRAM RSL directive, after it has been built either from globusrsl.gram2.globusrsl value or from all globusrsl.gram2.XYZ variables. e.g. (condorsubmit=('+AccountingGroup' '\"group_atlastest.usatlas1\"')('+Requirements' 'True'))
Arbitrary gram2 variables Any arbitrary GRAM2 variable can be specified using this nomenclature:
globusrsl.gram2.var = value
which will be added to the globusrsl line as (var=value). List of possible GRAM2 variables is
globusrsl.gram2.arguments
globusrsl.gram2.count
globusrsl.gram2.directory
globusrsl.gram2.dryRun
globusrsl.gram2.environment
globusrsl.gram2.executable
globusrsl.gram2.gramMyJob
globusrsl.gram2.hostCount
globusrsl.gram2.jobType
globusrsl.gram2.maxCpuTime
globusrsl.gram2.maxMemory
globusrsl.gram2.maxTime
globusrsl.gram2.maxWallTime
globusrsl.gram2.minMemory
globusrsl.gram2.project
globusrsl.gram2.queue
globusrsl.gram2.remote_io_url
globusrsl.gram2.restart
globusrsl.gram2.save_state
globusrsl.gram2.stderr
globusrsl.gram2.stderr_position
globusrsl.gram2.stdin
globusrsl.gram2.stdout
globusrsl.gram2.stdout_position
globusrsl.gram2.two_phase
Documentation can be found here: http://www.globus.org/toolkit/docs/2.4/gram/gram_rsl_parameters.html
GRAM5 variables globusrsl.gram5.globusrsl GRAM RSL directive. If this variable is not setup, then it will be built programmatically from all non empty globusrsl.gram5.XYZ variables. If this variable is setup, then its value will be taken *verbatim*, and all possible values for globusrsl.gram5.XYZ variables will be ignored.
globusrsl.gram5.globusrsladd custom fields to be added *verbatim* to the GRAM RSL directive, after it has been built either from globusrsl.gram5.globusrsl value or from all globusrsl.gram5.XYZ variables. e.g. (condorsubmit=('+AccountingGroup' '\"group_atlastest.usatlas1\"')('+Requirements' 'True'))
Arbitrary gram5 variables Any arbitrary GRAM5 variable can be specified using this nomenclature:
globusrsl.gram2.var = value
which will be added to the globusrsl line as (var=value). List of possible GRAM5 variables is
globusrsl.gram5.arguments
globusrsl.gram5.count
globusrsl.gram5.directory
globusrsl.gram5.dry_run
globusrsl.gram5.environment
globusrsl.gram5.executable
globusrsl.gram5.file_clean_up
globusrsl.gram5.file_stage_in
globusrsl.gram5.file_stage_in_shared
globusrsl.gram5.file_stage_out
globusrsl.gram5.gass_cache
globusrsl.gram5.gram_my_job
globusrsl.gram5.host_count
globusrsl.gram5.job_type
globusrsl.gram5.library_path
globusrsl.gram5.loglevel
globusrsl.gram5.logpattern
globusrsl.gram5.max_cpu_time
globusrsl.gram5.max_memory
globusrsl.gram5.max_time
globusrsl.gram5.max_wall_time
globusrsl.gram5.min_memory
globusrsl.gram5.project
globusrsl.gram5.proxy_timeout
globusrsl.gram5.queue
globusrsl.gram5.remote_io_url
globusrsl.gram5.restart
globusrsl.gram5.rsl_substitution
globusrsl.gram5.savejobdescription
globusrsl.gram5.save_state
globusrsl.gram5.scratch_dir
globusrsl.gram5.stderr
globusrsl.gram5.stderr_position
globusrsl.gram5.stdin
globusrsl.gram5.stdout
globusrsl.gram5.stdout_position
globusrsl.gram5.two_phase
globusrsl.gram5.username Documentation can be found here: http://www.globus.org/toolkit/docs/5.2/5.2.0/gram5/user/#gram5-user-rsl
Executable variables executable path to the script which will be run by condor. The executable can be anything, however, two possible executables are distributed with AutoPyFactory: - libexec/wrapper.sh - libexec/runpilot3-wrapper.sh
executable.arguments input options to be passed verbatim to the executable script. This variable can be built making use of an auxiliar variable called executable.defaultarguments This proposed ancilla works as a template, and its content is created on the fly from the value of other variables. This mechanism is called "interpolation", docs can be found here: http://docs.python.org/library/configparser.html These are two examples of this type of templates (included in the DEFAULTS block): executable.defaultarguments = --wrappergrid=%(grid)s \ --wrapperwmsqueue=%(wmsqueue)s \ --wrapperbatchqueue=%(batchqueue)s \ --wrappervo=%(vo)s \ --wrappertarballurl=http://dev.racf.bnl.gov/dist/wrapper/wrapper.tar.gz \ --wrapperserverurl=http://pandaserver.cern.ch:25080/cache/pilot \ --wrapperloglevel=debug executable.defaultarguments = -s %(wmsqueue)s \ -h %(batchqueue)s -p 25443 \ -w https://pandaserver.cern.ch -j false -k 0 -u user


Monitoring

 

APF logs are available through the panda monitor: yes they are! APF sets the GTAG environment variable, which is passed by the pilot through to panda and shows up in the monitor. Look at most any job running at a European site, e.g.,

http://panda.cern.ch:80/server/pandamon/query?job=1209939301

Under "pilotID" you have the link to the stdout logfile. Something which could be improved is to also automatically link to the stderr file (s/out/err) and to the condor log file (s/out/log).

This should be wrapper independent (it's passed in as part of the condor set environment). If you're not getting this something needs checked.

On the factory site there are two important variables which control this:

baseLogDir = /disk/panda/factory/auto/logs

baseLogDirUrl = http://svr017.gla.scotgrid.ac.uk/factory/auto/logs

baseLogDir is the local physical disk path; baseLogDirUrl is the http URL prefix.

What's missing from the panda monitor: an overview of the pilots going to a site, so know if the site is broken or the factories serving it have died, etc.

Where is this information now: in Peter's monitor! See the last talk as s/w week for some details (I gave the talk, but the content was all his).

So, should this be in panda monitor or not? It should be crosslinked from the monitor, but the key point was to have this on an independent database, not to add load to Oracle. It's monitoring, not accounting, some losses are ok and you throw the information away after a week.

Any factory can dispatch calls to the factory monitor, just by defining

monitorURL = http://py-dev.lancs.ac.uk/mon/

in their configuration. Peter has been gradually ramping up the number of factories to test scaling, so he can report on how well that's going.

In the end this should move to CERN, but we had the (usual) problems in obtaining and configuring a machine for it so this hasn't progressed much.

This really is a shifter tool as well. It's used to help diagnose problems with site infrastructure and to submit tickets (especially when pilots don't start or abort before the payload can be executed).

AutoPyFactory and the Cloud

Architecture:

 

 

The design consists on two AutoPyFactory (APF) queues working simultaneously.

The first AutoPyFactory queue submits jobs to a local pool when needed to serve a given WMS jobs queue (e.g. a PanDA siteid). This local condor pool may have very few actual nodes, or even no one. In this case, jobs will be waiting in queue.

The second AutoPyFactory queue inspects this local batch system, looking for jobs pending to be run. In this case, this list of pending jobs is acting as the WMS queue. If there are pending jobs (which is the equivalent to activated status in PanDA, for example) OSG Worker Node VMs are submitted to an IaaS provider via Condor-G (currently there is one submit AutoPyFactory plugin for submission to Amazon EC2 resources. In the future, to other "cloud" resources too).

This WN starts a condor startd daemon, which contacts a given condor Central Manager and become part of its pool.

AutoPyFactory plugins can be adjusted in such a way that VMs jobs submissions to the IaaS provider rate matches the associated WMS activated jobs load.

Software deployed in the image:

 

  • Red Hat Scientific Linux 5.7 base (RAW)
  • OSG WN, ATLAS WN, CA Certs. The environment variables $OSG_GRID, $OSG_DATA, $OSG_APP are defined.
  • CVMFS is installed and configured to point to BNL replicas/cache.
  • Condor startd, associated with BNL test Central Manager.

Issues during tests:

 

Problem with network: NAT, ..

When a startd contact a Central Manager, the latter must be able to call back the startd.

However, the WM is behind a NAT, preventing this call back from being possible. To solve this, the startd contacts the Central Manager via the Condor Connection Broker (CCB).

problem with $HOME directory

First trial failed with an error line like

02 Feb 17:00:36|runJob.py | Job command 1/1 failed: res = (2560,
'/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.2/AtlasSetup/scripts/setup_runtime.sh:
line 36: cd: /root: Permission
denied\n/var/lib/condor/execute/dir_24497/Panda_Pilot_24618_1328201237/PandaJob_1418631160_1328201243')

The reason was that jobs were running as user 'nobody' whose home directory in /etc/passwd is '/'. But it is possible that the shell environment variable $HOME is still pointed to /root.

problem with LFC (ATLAS specific):


during the devel steps, with the VM running on a box inside the BNL perimeter, it was not possible to contact the LFC catalog. For example, commands like lfc-mkdir failed. The reason is that because the host is inside BNL perimeter, the DNS gives the internal IP for the LFC host, but the LFC host has not a conduit for the sub-network where the VM is running.

NOTE: that should not be a problem when running on the cloud, outside BNL.

New wrapper

 

 

Motivation

 

The first piece of code that PanDA system submits to sites by different job submission mechanisms is called "pilot wrapper". This is the first code that executes on the worker node, performs some environment checks, and downloads from a valid URL the following piece of code to continue operations, called in the PanDA nomenclature as "pilot".

This "pilot wrapper" is not unique. There are a multiplicity of versions for this part of the system, depending on the final pilot type, and the grid flavor, for example.

This multiplicity forces to maintain several pieces of software even though they have a common purpose.

On the another hand, for practical reasons, these pilot wrappers are implemented in BASH language, with the consequent lack of flexibility and inherent difficulties to implemented complicated operations. One practical case is the need to generate weighted random numbers to pick up an specific development version of the ATLAS code only a given percentage of the times. This weighted random numbers generation is more complicated in BASH language.

Finally, the new AutoPyFactory pilot submission mechanism was introduced in the scenario. This new pilot submission tool was implemented in its first version to submit a specific ad-hoc pilot wrapper, with a different set of input options and with different formats. Moreover, this specific pilot wrapper is only valid for ATLAS in EGEE, being invalid for other purposes or in OSG sites. This discrepancy adds to the multiplicity of pilot wrapper versions, and introduces difficulties for its deployment as a submission tool to replace the already existing AutoPilot.

A final reason is that these wrappers require some improvements. One example is the absence of proper validation on the number and format of the input options. Given these improvements are important, it will be always easier to introduce and maintain them in a single piece of code than in several.

For these reasons it was agreed that a refactoring of the different pilot wrappers was needed. The proposal is to create a single pilot wrapper implemented in BASH language, performing the minimum amount of checking operations. This unique code should be valid for any kind of final application, grid flavor environment, submission tool, etc. In particular, it will allow the easy deployment of AutoPyFactory as pilot submission tool.

After checking the presence of required programs needed to continue with operations, and setting up the corresponding grid environment if needed, a second piece of code will be downloaded from a valid URL to continue operations. This second code will now be written in Python, which allows for more complex operations implemented in an easier manner. Therefore, its maintainability and scalability will be improved. This will require the reimplementation of all BASH code from the multiple pilot wrappers, except those operations already done by the new unified wrapper, in Python. Finally, in this second step, the final payload code to be run will be chosen, downloaded, and executed.

Wrapper.sh

 

A generic wrapper with minimal functionalities

input options:

3td width='28%' valign='top'> --wrappermode=mode

--wrappervo=vo the Virtual Organization.
--wrapperwmsqueue=wmsqueue is the wms queue (e.g. the panda siteid)
--wrapperbatchqueue=batchqueue is the batch queue (e.g. the panda queue)
--wrappergrid=grid_middleware is the grid flavor, i.e. OSG or EGI.
The reason to include it as an input option, instead of letting the wrapper to discover by itself the current platform is to be able to distinguish between these two scenarios:
(a) running on local cluster
(b) running on grid, but the setup file is missing

(b) is a failure and should be reported, whereas (a) is fine.

A reason to include wrappergrid as an option in this very first wrapper is that for sites running condor as local batch system, the $PATH environment variable is setup only after sourcing the OSG setup file. And only with $PATH properly setup is possible to perform actions as curl/wget to download the rest of files, or python to execute them.
--wrapperpurpose=purpose will be the VO in almost all cases, but not necessarily when several groups share the same VO.
An example is VO OSG, shared by CHARMM, Daya, OSG ITB testing group...
--wrapperserverurl=url is the url with the PanDA server instance
--wrappertarballurl=url is the base url with the wrapper tarball to be downloaded
--wrapperspecialcmd=special_cmd is special command to be performed, for some specific reason, just after sourcing the Grid environment, but before doing anything else.
This has been triggered by the need to execute command $ module load <module_name> at NERSC after sourcing the OSG grid environment.
--wrapperplugin=plugin is the plug-in module with the code corresponding to the final wrapper flavor.
--wrapperpilottype=pilot_type is the actual pilot code to be executed at the end.
--wrapperloglevel=log_level is a flag to activate high verbosity mode. Accepted values are debug or info.
allows performing all steps but querying and running a real job.



Note: before the input options are parsed, they must be re-tokenized so whitespaces as part of the value (i.e. --wrapperspecialcmd='module load osg') create no confussion and are not taken as they are splitting different input options.

The format in the condor submission file (or JDL) to address the multi-words values is:

arguments = "--in1=val1 ... --inN=valN --cmd=""module load osg"""

This first wrapper perform basic actions:
(1) check the environment, and the availability of basic programs

  • curl
  • python
  • tar
  • zip

(2) downloads a first tarball with python code as passes all input options to this code. With passed options, the python code will download a second tarball with the final pilot code.

 

Plug-ins architecture

 

This is the suggested architecture:

         AutoPyFactory ---> wrapper.sh ---> wrapper.py

wrapper.sh downloads a tarball (wrapper.tar.gz), untars it, and invoked wrapper.py. The content of the tarball is something like this

       - wrapper.py
- wrapperutils.py
- lookuptable.conf
- plugins/base.py
- plugins/<pilottype1>.py
- plugins/<pilottypeN>.py

The different plug-ins corresponds with the different wrapper flavors, so far written in BASH. For example, trivialWrapper.sh, atlasProdPilotWrapper.sh, atlasProdPilotWrapperCERN.sh, atlasProdPilotWrapperUS.sh, etc.) All of these wrappers share a lot of common functionalities, with only small differences between them. To take advantage from that, the different wrapper flavors will be implmented as plug-ins.

Look-up table

The current mechanism to choose the right plugin is implemented by inspecting a lookup table like this one:

# --------------------------------------------------------------------------------------------------------------------------------------------------------------  
# VO PURPOSE GRID WMSQUEUE BATCHQUEUE PLUGIN PILOTTYPE
# --------------------------------------------------------------------------------------------------------------------------------------------------------------

# --- default values ---
ATLAS * * + + atlasprodpilot pilotcode,pilotcode-rc
ATLAS devel * + + atlasprodpilotdev pilotcode-dev
OSG * OSG + + trivial trivialPilot

# --- testing sites ---
OSG * * TEST2 TEST2 trivial trivialPilot
OSG * * TEST3 TEST3 trivial trivialPilot
ATLAS * * ANALY_TEST-APF ANALY_TEST-APF-condor atlasprodpilot pilotcode,pilotcode-rc
ATLAS * * ANALY_TEST-APF2 ANALY_TEST-APF2-condor atlasprodpilot pilotcode,pilotcode-rc
ATLAS * * BNL_TEST_APF BNL_TEST_APF-condor atlasprodpilot pilotcode,pilotcode-rc

+ means any value is accepted, but one must be provided
* means any value is accepted, or no value was provided
- means no value was provided

if no value was provided for a given field:
1) first '-' will be used
2) if the field is not '-', then '*' will be checked

if a value was provided for a given field:
1) the value is searched
2) if the value is not in the field, the '+' will be checked
3) finally, '*" will be checked

Each column has the same value for every row.

The first N columns are the patterns, and the rest are outputs. That means that a number N of input values will be provided each time, the row maching with those inputs is selected, and the output values will be returned.

Columns will be parsed for matching from left to right. This means the first column is the most important field for matching, the second column is the next most important field, and so on.

When one of the provided inputs matches exactly the content of the field in the table, that row is selected. In the given input is not provided, or it is not in the table, then fields with symbols are inspected.

If no row matches completely with the input values, then None should be returned.

Contact

There is a mailing list. To join follow instructions here

Talks and publications

 

 

-->

-->

-->

Document Actions