Implement Backfilling

Implement backfilling to keep resources from becoming idle when the top job cannot run.

Backfilling

Overview of backfilling.

Backfilling means fitting smaller jobs around the higher-priority jobs that the scheduler is going to run next, in such a way that the higher-priority jobs are not delayed. When the scheduler is using backfilling, the scheduler considers highest-priority jobs top jobs. Backfilling changes the algorithm that the scheduler uses to run jobs:
  • When backfilling is off, the scheduler looks at each job in priority order, tries to run the job now, and if it cannot, it moves on to the next-highest-priority job.
  • When backfilling is on, the scheduler tries to run the top job now, and if it cannot, it makes sure that no other job that it runs in this cycle will delay the top job. It also fits smaller jobs in around the top job.
Backfilling allows you to keep resources from becoming idle when the top job cannot run.

Backfilling is a primetime option, meaning that you can configure it separately for primetime and non-primetime, or you can specify it for all of the time.

Parameters

Enable Backfill

Toggle that controls whether PBS Professional uses backfilling. If enabled, this scheduler attempts to schedule smaller jobs around higher-priority jobs when using strict ordering, as long as running the smaller jobs won’t change the start time of the jobs they were scheduled around. This scheduler chooses jobs in the standard order, so other high-priority jobs will be considered first in the set to fit around the highest-priority job.

When this parameter is enabled and Help Starving Jobs is enabled, this scheduler backfills around starving jobs.

Backfill Depth
The number of top jobs that PBS backfills around. For example, if set to 3, PBS backfills around the top 3 jobs.

How Backfilling Works

A scheduler makes a list of jobs to run in order of priority, for any queue that has an individual backfill depth, for the server if there are queues without a backfill depth set. These lists are composed according to execution priority. These are top jobs.

If you use backfilling, a scheduler looks for smaller jobs that can fit into the usage gaps around the highest-priority jobs in each list. A scheduler looks in each prioritized list of jobs and chooses the highest-priority smaller jobs that fit. Filler jobs are run only if they will not delay the start time of top jobs.

A scheduler creates a fresh list of top jobs at every scheduling cycle, so if a new higher-priority job has been submitted, it will be considered.

Backfilling is useful in the following circumstances:
  • When Strict Ordering is enabled, filler jobs are fitted around higher-priority jobs. Without backfilling, no job runs if the top job cannot run.
  • When Help Starving Jobs is enabled, filler jobs are fitted around starving jobs.

Backfilling Around N Jobs

You can configure the number of top jobs that PBS backfills around by setting the value of the Backfill Depth parameter.

If Backfill Depth is set to 3, the scheduler backfills around the top 3 jobs.

Configure Backfilling

Enable backfilling and modify backfilling behavior by setting the number of jobs that are to be backfilled around.

Important: The Enable Backfill parameter can only be updated if the cluster was added by a user with passwordless sudo permissions and therefore may not appear as a Scheduling parameter.
  1. Click the Configure tab.
  2. Choose the HPC to configure.

    Choose an HPC
    Figure 1. Choose an HPC
  3. Click Scheduling from the PBS Professional menu located on the left-hand side of the web page.
  4. Scroll down to the second list of Scheduling parameters.
  5. Click located to the right of Enable Backfill.
  6. Choose one of the following options to enable or disable backfilling:
    • To enable backfilling for both prime and non-prime time, enable Enable Backfill and enable both Prime and Non-Prime.
    • To enable backfilling for only prime time, enable Enable Backfill, enable Prime and disable Non-Prime.
    • To enable backfilling for only non-prime time, enable Enable Backfill, enable Non-Prime and disable Prime.
    • To disable backfilling disable Enable Backfill.
  7. Click to save the changes.
  8. Scroll up to the first list of Scheduling parameters.
  9. Click located to the right of Backfill Depth.
  10. For Backfill Depth, enter the number of jobs that are to be backfilled around.
    Tip: Recommendation: set this value to less than 100.
  11. Click to save the change.