Parametric distribution

Some Flux projects contain parameters that can be solved independently. Flux comes with a feature that allows the distribution of these parameters to speed-up this kind of computations. To use that feature, the following environment variables must be defined:

FLUX_PARAMETRIC: Indicator for using parametric distribution. Must be set to true
FLUX_PARAM_AUTO: Indicator for using automatic or manual computation of number of jobs. Must be set to Automatic or Manual
- In Automatic, Flux will submit a maximum of 127 jobs (+1 for the master) to solve the project. The number of solved parameters per job is automatically computed, and if there are less than 127 parameters, each job will solve 1 parameter.
- In Manual, the environment variable FLUX_PARAM_MAXCORES must be set to the maximum number of cores you want to allocate for the job. One of these cores will be used for the master, and the other will be jobs submitted by Flux to solve the project. The number of solved parameters per job is automatically computed, and if there are less parameters than remaining cores, each job will solve 1 parameter and the exceeding cores won’t be used and will be free for other jobs.

# Use parametric distribution
export FLUX_PARAMETRIC=true
export FLUX_PARAM_AUTO=Manual
export FLUX_PARAM_MAXCORES=32

To submit jobs and manage files, some basic PBS and Linux commands are used by Flux. These commands are already defined by default, but can be modified using environment variables:

COPYREC: recursive copy. Default: cp -r
SUBMITNAME: submit job with name option. Default: qsub -N
MOVE: move files. Default: mv
STATGREPCOUNT: Display all jobs, filter and count option. Default: qstat –a | grep –c

For example, using a classical Linux installation, but a batch scheduler using OAR, the COPYREC and MOVE should be identical, but the two other variables need modifications:

#Define commands for parametric distribution /!\ Check with batch
export SUBMITNAME=oarsub -n
export STATGREPCOUNT=oarstat | grep -c