Configure Container Integration

Container Support

Linux containers can be leveraged to constrain the amount of system resources used by jobs. Accelerator's container support is designed to be agnostic of the container solution. The examples provided are for Docker specifically.
Note: It is currently recommended to exclude contained jobs from preemption. This can be done at job submit time using the "-preemptable 0" submission option or by writing preemption rules to exclude jobs that request a "Container:X" resource.

Named Container Configurations

Containers are enabled by the administrator through named configurations that can be requested as a job resource. Each named configuration will contain a recipe of hooks to call to setup the container, and limits to impose upon it. Hooks are stand-alone shell scripts, each meant to perform a certain task, with specific UNIX permissions (root | user).

Named configuration files must be stored in the SWD/containers directory and end with a .cfg extension. Examples named configuration files are provided in the $VOVDIR/etc/config/containers directory, such as:

File: Container C1 definition:
### Example named-container configuration c1.
## Hook run directory
#
# The default behavior is to switch to the job directory before running the
# hooks. If this directory is not accessible until the hooks have been executed
# though, the job will fail. This issue can be resolved by setting
# containerHooksRunDir to a location that is guaranteed to be accessible prior
# to the hooks being executed. This causes the subtasker to switch to that
# directory instead, and the original job directory will be made available to
# the hooks via the VOV_CONTAINER_JOB_RUNDIR environment variable. The value of
# this variable can then be used by the hooks to specify the working directory
# for the container instance.
# containerHooksRunDir "/path/to/run/directory"

## Hook definitions
#
# containerHook <type> <mode> <path> <signature>
#   <type>      = setup | enter | cleanup | teardown
#   <privilege> = user  | root
#   <file>      = absolute path to hook file
#   <signature> = output of vovsignfile command
#
# Specifies which hook(s) should be called to interact with the container
# platform throughout the job's life cycle:
#
# 1. The setup hook is required if the container must be created prior to
#    spawning the job. This hook will always be used if root privileges are
#    required to create containers.
# 2. The enter hook is always required and is responsible for placing the
#    job into the container. If root privileges are not required to create
#    containers and the container platform supports it, the enter hook can
#    also create the container, as would be done via the "docker run" command,
#    for example. The enter hook must block for the duration of the job.
# 3. The cleanup hook can be used to remove temporary artifacts that are
#    generated by the job.
# 4. The teardown hook is required to stop and/or remove the container. This
#    hook would normally be used if a setup hook is used to create the
#    container.
#
# Any root-mode hook must be owned by root on the filesystem.
#
# The tasker sets at least the HOST, VOV_PROJECT_NAME, VOV_JOBSLOT, and
# VOV_CONTAINER_NAME environment variables in the job execution environment.
# It is intended for the hooks to use these variables to uniquify container
# instances. Hooks can be tested by running them manually in the shell as long
# as the required environment variables are set to some value. The only hook
# that is called with arguments is the enter hook, which expects the job command
# line to be passed in. An example test run of the enter hook may look like:
#
# % env HOST=foo VOV_PROJECT_NAME=vnc VOV_JOBSLOT=0 VOV_CONTAINER_NAME=c1 \
# /path/to/c1-enter.sh whoami
# containerHook setup    root "/path/to/c1-setup.sh"    1211238920
containerHook enter    user "/path/to/c1-enter.sh"    97261574
# containerHook cleanup  user "/path/to/c1-cleanup.sh"  5376821904
# containerHook teardown root "/path/to/c1-teardown.sh" 8237156649

Specify the Taskers that Support Containers

Each named container configuration will require a Container:X resource to be offered by every tasker that supports that specific container configuration. This can be done via the tasker.tcl file or the TaskerClass Table File file.

When a user includes Container:c1 in the resource request for their job, the request will be passed to the tasker that is selected to execute the job and the tasker will process the recipe defined in the configuration. If the recipe references a hook that does not exist, is not executable, or has a different signature than the one specified in the configuration, the job will fail.

Hooks

It is recommended to manually test each hook before using them in production environment. Use the tasker hosts for testing and production. An example test run of the "enter" hook may look like:
#% env HOST=foo VOV_PROJECT_NAME=vnc VOV_JOBSLOT=0 VOV_CONTAINER_NAME=c1 \
#/path/to/c1-enter.sh whoami

Multiple hooks can be utilized to integrate with the container solution. At a minimum, an "enter" hook will be required, as long as the container solution provides a single command to setup the container, run the job, and remove the container afterward. For container solutions that do not provide this feature, separate setup and teardown hooks can be configured. For either method, an optional cleanup hook can be configured to remove artifacts generated by the job from the file system, for example.

Hooks must also be stored in the SWD/containers directory and can be in script or binary form. Example hook scripts are provided in the $VOVDIR/etc/config/containers directory, such as:

File: c1-enter.sh
!/bin/bash -fxv
# Note: The enter hook must block for the duration of the job.
#
# Container c1 example enter hook: an all-in-one script that creates a
# container, launches a job inside of it, then exits and removes the container.
#
 
 
# The following environment variables are available and should be used to avoid
# container and/or host name conflicts:
#
#   HOST                (string)
#   VOV_PROJECT_NAME    (string)
#   VOV_JOBSLOT         (number)
#   VOV_CONTAINER_NAME  (string)
#
# The following environment variables are available if defined in the named
# container configuration file:
#
#   VOV_CONTAINER_CORES (number)
#   VOV_CONTAINER_RAM   (megabytes)
#   VOV_CONTAINER_TMP   (megabytes)

# For containers handling interactive jobs (-I), uncomment the following line.
# set -m
containerName=${VOV_PROJECT_NAME}_${HOST}_${VOV_CONTAINER_NAME}_${VOV_JOBSLOT}
uid=$(id -u ${USER})
# Handle the run directory specified in the named container configuration file.
if [[ -d $VOV_CONTAINER_JOB_RUNDIR ]]; then
  workDirOptions="--workdir $VOV_CONTAINER_JOB_RUNDIR --mount type=bind,source=${VOV_CONTAINER_JOB_RUNDIR},target=${VOV_CONTAINER_JOB_RUNDIR}"
else
  workDirOptions="--workdir $PWD"
fi
# Process limits into Docker options.
limitOptions=""
if [[ -n $VOV_CONTAINER_CORES && $VOV_CONTAINER_CORES > 0 ]]; then
  limitOptions+=" --cpus $VOV_CONTAINER_CORES"
fi
if [[ -n $VOV_CONTAINER_RAM && $VOV_CONTAINER_RAM > 0 ]]; then
  ramSpec="${VOV_CONTAINER_RAM}m"
  limitOptions+=" --memory $ramSpec"
fi
if [[ -n $VOV_CONTAINER_TMP && $VOV_CONTAINER_TMP > 0 ]]; then
  # Use in-memory tmpfs for /tmp in container.
  tmpBytes=$(($VOV_CONTAINER_TMP*1048576))
  limitOptions+=" --mount type=tmpfs,destination=/tmp,tmpfs-size=$tmpBytes"
fi
# Capture job environment in a file for Docker to import.
envFile=/tmp/${containerName}.env
env > $envFile

# Use the Docker "run" command to create a container based on the "myImage"
# container image, setup networking, specify the user, capture the environment,
# and bind-mount the required directories for the job. Finally, the job itself
# is passed in for Docker to execute. The image must have the ability to resolve
# the job owner's UID, have access to the VOV software installation, and be able
# to execute the vw job wrapper along with the job command.
# The "$@" variable will contain "vw <jobCmd> <jobArgs>".
Note: Pay close attention to the comments in each hook example. Failure to follow the guidelines provided will likely result in failure to integrate with the container solution, as well as job failure. Any root-mode hook must be owned by root on the filesystem.

Hook Signatures

The configuration for each hook that is defined must contain a signature. The signature is used by the tasker to verify the hook has not been tampered with since it was defined by the administrator. The signature can be obtained by using the vovsignfile utility. In the following example, the signature is highlighted in red:
# containerHook setup    root "/path/to/c1-setup.sh"    1211238920
containerHook enter    user "/path/to/c1-enter.sh"    97261574
# containerHook cleanup  user "/path/to/c1-cleanup.sh"  5376821904
# containerHook teardown root "/path/to/c1-teardown.sh" 8237156649

vovsignfile

Utility to obtain a security signature for files.


vovsignfile: Usage Message

    Utility to obtain a security signature for files.

    USAGE:

      vovsignfile [OPTIONS] <FILE>

    OPTIONS:

      -h    -- Show usage syntax.

    EXAMPLES:

      % vovsignfile -h
      % vovsignfile /path/to/file