Remote Server User Guide

When a job is submitted for execution to a remote server, ACC must perform the following functions.
  1. Identify all input and data files needed to run this job, move them from the client host to a location accessible from the remote server, and adjust these files to reflect new location.
  2. Trigger job to run on the remote server.
  3. Collect the files needed by the user and bring them back to the client host.
  4. Clean up the disks on remote server after the run.
Note: Each step, except #2 may not be required.

Different File Locations

Running the job may require additional moving of the files before and after the run, and even some additional file transfers between the stages of the solver.

In general, two locations need to be specified:
  • Run directory:

    Location where all input files are finally used to run the solver. In case of MPI run, this may represent multiple locations (one local disk attached to each host), although for AcuSolve, the run directory should be shared between all MPI processes.

  • Stage directory:

    Location which is accessible from outside of cluster and from hosts inside the cluster. This is just a temporary ‘parking’ area for the files before they end up at the location suitable for the run. The stage directory may not be suitable for running the solver – this is typical when stage is set on slower disk, or when it is used by many hosts. In both cases it is recommended to have a separate run directory location to speed up the execution.

For the remote run submission run directory may be left undefined:
  • Specifying only the run directory is appropriate for local runs (for MPI execution), but for remote execution, it should only be used together with stage directory.
Options associated with input files:
  • Rundir
  • Stagedir
  • Copyoptions
  • Pushfile
  • Filelist

Input Files

In order to properly move all required files, ACC needs to understand which files are needed for the run, and for some solvers this information has to be explicitly provided before job submission; however, in some cases, ACC is able to either identify these files, or there is an external utility to create such list.
  • For Altair Manufacturing Solver, the main input file contains the list of all relevant input files, and so ACC reads it from there.
  • OptiStruct provides an option to run in a special test mode (internal option -inventory) to provide such list.
  • In some cases when ACC can figure out the list of required files, but it happens to be incomplete, you must expand this list manually using option -pushfile.
When moving these files, ACC copies all of them to a single location. This will not work when the files are referenced by a full path, or even a complex path relative to the current location. These cases require that ACC will either:
  • Modify input files and adjusts these references – this creates potentially confusion for you, as reports from the run contain references to the files at unexpected locations. This is troublesome for error messages.
  • Direct the solver to accept that the files have been relocated. Only some solvers allow that: OptiStruct has various internal command line options to specify search path for includes (-incpath) or explicitly force to use all files from the single location (-asp, -localonly). For other solvers, it is expected that all additional files needed for the remote run are located in the same directory as the main input file.

In rare cases (very complex structure of includes for OptiStruct run), the set of input files may contain files with identical names but different locations. Currently, ACC cannot handle such runs for remote submission.

Output Files

After completion of the run at the remote server, ACC may need to copy results of computations to the local host, and optionally to clean all trash/scratch files. During the submission of remote job, ACC displays the following form, which allows you to decide what to do for all output files on the stage directory:
  • Copy everything
  • Copy only known output files (H3D files)
  • Do not copy anything
  • (Optional) Completely remove all traces of the run from remote server


Figure 1.

When the run directory is specified, ACC will automatically move all files between stage and run locations before and after the run, and ensure that the run location will be clean after the run.

Control Remote Job

Before a job is transferred to the remote server, ACC will need to know more details how to run this job. Specifics depend on the type of the remote server. For PBS submission, such details may include:
  • Memory requirements
  • Number of cores, hosts or pbschunks (for MPI runs)
  • Specific hosts or the queue name to submit.

These items may also be requested on the command line through following options.

Once the remote job is submitted, ACC allows to interactively check the status of such job. On the ACC GUI, there is menu option (View > Remote Jobs), and the command line option (-jobstatus). When the check confirms that the job completed on a remote server, ACC automatically starts the download of selected output files and cleans stage location as requested during submission. All subsequent inquires for this jobid will be void, and any further copying or cleaning (if desired) needs to be done manually by direct connecting to the remote server.

ACC options to control remote jobs:
  • pbschunks
  • pbscpu
  • pbsmem
  • pbsqueue
  • jobstatus

Submit Output Files

After a job is completed, ACC will collect all useful artifacts of the run and copy them to a separate location in a stage directory. Run directory (directories) if present and different from stage directory, will be cleaned.

When you are prompted to verify if the remote job is completed, and you confirm it is completed, the ACC will perform the action specified during submission:
  • Copy full results:

    All files created by the run will be copied to a client location (on the laptop), and erased from the stage location.

  • Copy only the post-processing files:

    Only the files of known type will be copied to the client location.

  • Delete results on remote server:

    When this option is selected, all files on the stage location will be deleted after the run. Otherwise, they will remain, and it is your responsibility to remove them when no longer needed.

Once the operations selected above are performed, ACC will move the job information to the History form, where it can be used for reference only. All files which are left on the client host or on the server are no longer controlled by ACC. It is your responsibility to remove them when no longer needed.