PBS and Other Queues

Allocator can allocate licenses across multiple schedulers, including Accelerator queues, PBS clusters and others. In this section we discuss how to integrate Allocator and PBS.

Preparing a PBS Environment

We need an environment called PBS to be available, at least, on the host where Allocator is running. An example of this environment could be this: 
#
# This should be $VOVDIR/local/environments/PBS.start.csh
#
set    path = `vovenv PREPEND /opt/pbs/bin $path `
setenv MANPATH         `vovenv PREPEND -: /opt/pbs/share/man  $MANPATH`
setenv LD_LIBRARY_PATH `vovenv PREPEND -: /opt/pbs/lib $LD_LIBRARY_PATH`

This environment is needed by the command vovlapbs which probes the status of PBS and sends it to Allocator.

Enabling qmgr Access from the Allocator Server Machine

Allocator actively manages one or more PBS static resources that correspond to software licenses or license options. These resources are termed "LA-managed resources". Allocator manages and updates the resource availability in PBS by calling qmgr from the Allocator server machine. It is therefore important that the user running Allocator has manager privileges on the PBS instance.

In the following example, PBS is running on the machine pbshost1. Allocator is running as user cadmgr on the machine called lahost1:
pbshost1 # qmgr
Max open servers: 49
Qmgr: set server flatuid=true
Qmgr: set server managers += cadmgr@lahost1.company.com  ... Or... 
Qmgr: set server managers += cadmgr@*
Qmgr: list server managers
   managers = cadmgr@*

Configuring the PBS Scheduler

All resources managed by Allocator must be declared like any other static resource in the "resources:" line in the sched_config file on the host that is running the PBS scheduler. In specific, you have to change the "resources:" line in sched_config. This file is typically in /var/spool/pbs/sched_priv/sched_config. In the following example, we add the three Allocator-managed resources License_spice License_lic_drc and License_lic_erc. Notice that we cannot use the colon ":" in the license names, as is common in Accelerator. 
# Example: fragment of sched_config
resources: "ncpus, mem, arch, host, vnode, aoe, eoe, License_spice, License_lic_drc, License_lic_erc"
Then you have to tell the PBS scheduler to re-read the config file. You can do that by sending the signal HUP to the scheduler process, as in this example: 
# cd /var/spool/pbs/sched_priv
# kill -HUP  `cat sched.lock`
Finally, you need to use qmgr to create the resources and to declare them as schedulable by the queue: 
# qmgr 
qmgr:  create resource License_spice type=long,flag=q
qmgr:  create resource License_lic_drc type=long,flag=q
qmgr:  create resource License_lic_erc type=long,flag=q

% qmgr -c "list resource"

Configuring Allocator

On the Allocator side, you have to add a site to the config file in la.swd/vovlad/config.tcl. 
## Fragment of vovlad/config.tcl file 
## We initially set the default weight to -1 (DO_NOT_SHARE) 
## Then we selectively set the weight of various resources to the desired value.
LA::AddSite lava1_NOTUSED PBS1  {} -scheduler PBS -remotehost lava1 -timezone PST8PDT -defaultweight -1

## Assign a weight to the resources that are actually shared with PBS
foreach lic { lic_drc lic_erc spice } {
   LA::DefineRemoteResourceName PBS1 License:$lic License_$lic

    LA::SetResourceWeight PBS1 License:$lic 75 ; # Use the "official" resource name, not the PBS name.
}

Using PBS with Allocator

There is no change in the way jobs are submitted to PBS. If the submitted job requires a license static resource that is managed by Allocator, then the availability of that license will change over time every few seconds depending on the workload at the various sites.
% qsub -l License_lic_drc=1 -- /bin/sleep 1000 
% qsub -N DRC -J 1-10  -l License_lic_drc=1 -- /bin/sleep 1000 
% qsub         -l select=4:ncpus=1 -lplace=scatter -l License_vera=1 -- /bin/sleep 100
% qsub -J 1-10 -l select=4:ncpus=1 -lplace=scatter -l License_vera=1 -- /bin/sleep 100

Also in this Section