Sort Vnodes

Sort nodes on a key that you specify.

Vnode Sorting

Overview of vnode sorting including parameters and examples.

PBS can sort vnodes according to a key that you specify. This can be used when deciding which vnodes to use for jobs. Sorting vnodes comes into play after a placement set has been selected, or when a job will run on vnodes associated with a queue, or when placement sets are not used, because in those cases there may be more vnodes available than are needed. You can sort vnodes on one or more different keys, and for each key, you can sort from high to low, or the reverse.

You can sort on the vnode priority attribute and vnode resources.

The default way to sort vnodes is according to the value of the vnode priority attribute, from higher to lower.

When you sort vnodes according to the assigned or unused amount of a resource, the vnode list is re-sorted after every job is run. This is because each job may change the usage for that resource.

You configure sorting vnodes on a key by setting values for the Node Sort Keys parameter. It is a primetime option, meaning that you can configure it separately for primetime and non-primetime, or you can specify it for all of the time.

When vnodes are not sorted on a key, their order is undefined.

You can specify up to 20 sort keys, where you want a primary sort key, a secondary sort key, etc.

If you specify more than one entry for Node Sort Keys, the first entry is the primary sort key, the second entry is the secondary sort key, which is used to sort equal-valued entries from the first sort, and so on.

Parameters

Node Sort Keys
Defines sorting on resource or priority values on vnodes. Resource must be numerical, for example, long or float. Up to 20 Node Sort Keys entries can be used, in which case the first entry will be the primary sort key, the second will be used to sort equivalent items from the first sort, etc.
Sort Priority
Determine whether the vnodes are sorted from high to low (HIGH) or low to high (LOW).
Sort Type
Determine whether sorting for the resource is done by:
  • total - the resources available value
  • assigned - the resources assigned value
  • unused - the value given by resources available - resources assigned
Defaults to total.
Modifiers
Determines whether the sorting is implemented for prime time, non-prime time or both.

Example 1

Sort vnodes by the highest number of unused CPUs:

Node Sort Keys: ncpus HIGH unused all

Example 2

Sort vnodes by the highest amount of memory assigned to vnodes, but only during primetime:

Node Sort Keys: “mem HIGH assigned” prime

Example 3

Sort vnodes according to speed. You want to run jobs on the fastest host available. You have 3 machines, where HostA is fast, HostB is medium speed, and HostC is slow.

Set node priorities so that faster machines have higher priority:

Qmgr: set node HostA priority = 200

Qmgr: set node HostB priority = 150

Qmgr: set node HostC priority = 100

Specify that vnodes are sorted according to priority, with highest priority first:

Node Sort Keys: "sort_priority HIGH" ALL

Example 4

The old “nodepack” behavior can be achieved by this:

Node Sort Keys: “ncpus low unused”

Example 5

In this example of the interactions between placement sets and Node Sort Keys, we have 8 vnodes numbered 1-8. The vnode priorities are the same as their numbers. However, in this example, when unsorted, the vnodes are selected in the order 4, 1, 3, 2, 8, 7, 5, 6. This is to illustrate the change in behavior due to Node Sort Keys.

Node Sort Keys: “sort_priority LOW”

Using Node Sort Keys, the vnodes are sorted in order, 1 to 8. We have three placement sets:

A: 1, 2, 3, 4 when sorted by Node Sort Keys; 4, 1, 3, 2 when no node sort key is used

B: 5, 6, 7, 8 when sorted by Node Sort Keys; 8, 7, 5, 6 when no node sort key is used

C: 1-8 when sorted, 4, 1, 3, 2, 8, 7, 5, 6 when not sorted.

A 6-vnode job will not fit in either A or B, but will fit in C. Without the use of Node Sort Keys, it would get vnodes 4, 1, 3, 2, 8, 7. With Node Sort Keys, it would get vnodes 1 - 6, still in placement set C.

Sort Vnodes on a Key

Sort vnodes based on a specified key.

Important: The Node Sort Keys parameter can only be updated if the cluster was added by a user with passwordless sudo permissions and therefore may not appear as a Scheduling parameter.
  1. Click the Configure tab.
  2. Choose the HPC to configure.

    Choose an HPC
    Figure 1. Choose an HPC
  3. Click Scheduling from the PBS Professional menu located on the left-hand side of the web page.
  4. Scroll down to the Node Sort Keys parameter.
  5. Choose on of the following options:
    • Click when there are existing node sort keys.
    • Click when there are no existing node sort keys.
  6. For Node Sort Keys, choose one of the following options:.
    • Enter sort_priority to sort vnodes according to the value of the vnode priority attribute.
    • Enter a resource name to sort vnodes according to the assigned or unused amount of a resource.
      Note: Resource must be numerical, for example, long or float.
  7. For Sort Priority, choose one of the following options:
    • HIGH - sort from high to low.
    • LOW - sort from low to high.
  8. For Sort Type, choose one of the following options:
    • Total - Use the resources_available value.
      Note: This is the default setting when sorting on a resource.
    • Assigned - Use the resources_assigned value.
    • Unused - Use the value given by resources available - resources assigned
  9. Determine whether node sorting is implemented for prime time, non-prime time or both:
    • To configure the sort key for both prime and non-prime time, enable both Prime and Non-Prime.
    • To configure the sort key for only prime time, enable Prime and disable Non-Prime.
    • To configure the sort key for only non-prime time, enable Non-Prime and disable Prime.
  10. To add another sort key:
    1. click .
    2. Repeat steps 6 through 9.
    Tip: To delete a node sort key click .
  11. Click to save the change.