2019.01 Release

Enhancements

The following new features and enhancements were introduced this software release:
Product Internal Number Case Number Description
All VOV-8981 22923 Slave objects have a new hardware resource HT that specifies whether hyper-threading is enabled on the host where the slave is running (query example: vovselect ht from slaves).
All VOV-8823 22590 The -O option on nc hosts has been enhanced to support display of the @RAMUSED@ hardware resource.
All VOV-8808   Added the field "SLOTSAVAIL" to the SAllocatorVES table for use with vovselect and the vtk_select APIs.
All VOV-8809   The following queryable fields have been added to the RESOURCEMAPS object: PREEMPTION, OTHERS, RESRES, NOLOG, NOOOQ, NOMATCH, MATCHRECENT, LEFTOEXPIRE, UTILIZATION, OOQDUMP, RESDUMP, and JOBDUMP.
All VOV-8320   CLI lmmgr, ncmgr/wxmgr now allows you to specify the database port option when start a new project (in consistence with the web UI database configuration tool and the CLI vovdb_util configure tool).
All VOV-8810   Several new queryable fields have been added to the JOBS and FILES objects; see the output of vovselect fieldname from JOBS and vovselect fieldname from FIELDS. In addition there is a new queryable object named "IOS". This object allows querying from the inputs and/or outputs of a specific node. For example to get the ID field from all of the inputs of node 12345, the query would be vovselect id from ios.12345 where isinput.
All VOV-7917 20080
Important: The license grace period functionality has been removed due to a technical limitation.
The software issues alerts 30/14/7/1d prior to the earliest expiration detected (in case there are multiple lines of the same license feature in the same license file).

Licensing robustness and network fault-tolerance have been significantly improved. The software can operate in "disconnect mode", which means that it connects, gets a checkout, then disconnects from the license server. Checkout is refreshed every hour. The software will not fail for until 5 full days have passed since the last successful checkout. Alerts are issued for this as well.

As before, license servers may be specified as a colon-separated list via the RMonitor_LICENSE environment variable
All VOV-8766   When copying the <swd>/vovnginxd/conf/nginx.conf.template file to an nginx.conf file, a message is output to the server log indicating that a copy is being made, and also specifying the source and destination files.
FlowTracer VOV-8678 22270 Add support for the substitution of the @JOBID@, @ARRAYID@, and @INDEX@ keywords in job resource requests. Due to the risk of overloading the system with job queue buckets, these are opt-in, using the following settings in policy.tcl: 
substituteArrayIdInResources 
substituteArrayIndexInResources 
substituteJobIdInResources
For each, 0 is disabled (default) and 1 is enabled.
FlowTracer VOV-9005 22742 In FlowTracer, vovlsfd will "bsub" buckets based on priority; earlier it used to be rank followed by priority.
FlowTracer VOV-3186   Some pre-defined sets that are created on-demand were not getting created (if nonexistent) by the web UI. This issue is now fixed. The sets impacted by this change are Isolated nodes, Missing files, Blocking files, Failed jobs, Jobs with no inputs, Jobs with no outputs, Autoflow, Tainted. Removed the "High Level Flows" set from the web UI.
FlowTracer VOV-7968 21613 vovwxd slave log deletion is done in a separate process.
FlowTracer VOV-7920 21534 vovwxd will not delete the log file of a running slave during the logs cleanup routine.
FlowTracer VOV-8844 22622 vovfileready has been re-implemented to eliminate the side-effects present in the earlier implementation. Consequently, it is no longer necessary to call vovfileready -clean. Also, vovfileready no longer creates phantom barriers and non-executable jobs during its operation.
FlowTracer VOV-7069 21501 "STOP" action on a running job will check the ACL permissions for the user attempting to stop the job. A LEADER will no longer be allowed to stop other people's jobs if LEADER does not have STOP permission in ACL.
FlowTracer VOV-5668 20732 The following new features are supporte for stopping and suspending jobs:

The -signals, -include and -exclude options to nc stop and vovstop -f are now functional.

The trace parameters defaultStopSignalCascade and defaultSuspendSignalCascade have been added to specify default behavior when requesting to stop or suspend a job, respectively.

The job property STOP_SIGNALS can now be used to set stop-signal cascades on a per-job basis.
FlowTracer VOV-5148 20735 Enhanced 'Forget set & elements' menu option to forget the set and subsets hierarchically.
FlowTracer VOV-4898 20139, 20142 Added node status change logging capability with GUI hooks in the HTML and Tk interfaces to change the type of logging. Added reasons for all node status changes that are queryable using the @WHYSTATUS@ field of nodes.

Internally, all commands that change node status now pass along a reason for the change, which is logged in the journal if node status logging is enabled. The most recent reason is in the @WHYSTATUS@ field.

In vtk API commands that change status, a new "-why REASON" option has been added. Any reason can be passed, and it will get added to the @WHYSTATUS@ field, which sometimes gets stored as the @WHYSTATUS@ property on nodes (if the text reason is not in the internal list of known reasons). Client information is also included (user, executable name, security level, etc).

Node status logging can be enabled server-wide using a global setting (see vtk_node_status_logging command), or individually on jobs or files. It can also be enabled on sets, which apply only to the nodes in the set at the time the command is called (it applies it to the nodes in the set, not to the set itself).

Node status logging can be changed (enabled/disabled) using vtk_node_status_logging, through the node editor GUI (nc gui or vovconsole), or through the HTML interface on the Sets page, the Job Edit page, or the journal viewer (for jobs). There are 5 settings for node status logging:
  • UNSET - use the default server setting
  • DISABLED - logging turned off
  • ALL - all changes of all types are logged
  • INVALIDATIONS - changes logged only if node invalidated
  • DOWAcceleratorONE INVALIDATIONS - changed logged only for downcone invalidations (when the entire downcone of a node is invalidated).
See documentation for more details.

Resolved Issues

The following issues were resolved in this software release:
Product Internal Number Case Number Description
All VOV-9363   Fixed server crash when the Detach slave from server" link was clicked on the Slave Details page in the web UI.
All VOV-9333   Fixed an issue with vovserver failover that could lead to a corrupt VOV_HOST_HTTP_NAME setting if that value did not already contain a fully-qualified domain name.
All VOV-9333 23318 Fixed issue where an invalid slave resource expression would result in a permanent error message on the "slaves" web page.
All VOV-8578 21858 The following queryable fields have been added to the SAllocatorVES table for consistency with the vovslavemgr configure command: RAMSENTRYFAllocatorG, MINRAMFREE, EFFlowTracerOTALRAM, RETRYCHDIR, RETRYCHDIRSLEEP, RETRYCHDIRSLEEPBACKOFF, MAXWAITNOSTART, ALLOWCOREDUMP, DEBUGJOBCONTROL, AUTOKILMonitorETHOD, MINDISKSPACE.
All VOV-8685 22253 Fixed issues in D and U environments (which are used to Define and Undefine specific env vars) when they are both used, or they are nested (such as having multiple D calls in the env). Previously, it was possible to define something in the environment but have that item undefined in the "END" environment script. Now, it preserves the state using temporary environment variables so that each "START" and "END" environment script has consistent environment variables defined.
All VOV-9501 23821 Rebuilt binaries dependent on OpenSSL to remove a bad default OpenSSL configuration path.
All VOV-9485 23695 Fix a rare crash during slave startup caused by an intermittent failure in the Linux proc filesystem.
FlowTracer VOV-9253   Added documentation for vovlogchecker
FlowTracer VOV-9263   The vovconsole Alerts dialog now wraps lines, permitting very long alerts to be read in their entirety.
FlowTracer VOV-8512   Readonly user is not allowed to recompute a set
FlowTracer VOV-8100 21852 LEADERs can also add alerts and echo logs, not just ADMINs
FlowTracer VOV-7966 21609 The API will detect an invalid ACTION such as START and give an error message, instead of silently returning incorrect result.
FlowTracer VOV-7536 20835, 21521 Fixed issue with too many arrows between sets when building or re-building the flow while viewing the parent set. Now it recomputes the arrows between sets when events that could modify those arrows occur.
FlowTracer VOV-7355 20549 Increased the maximum number of dependency levels in a FlowTracer flow graph from 1024 to 4096.
FlowTracer VOV-8418 22102 The effect of an md5 barrier, if present, is now taken into account in all cases that a regular barrier would have had an effect.
FlowTracer VOV-8143 21920 vovdaemonmgr show will show the status of all instances of vovlsfd connected/configured against the current project, across all users of the project.
FlowTracer VOV-8033 21722 When starting an xterm, we want to have the project enabled but also pick up aliases and such from the users .csrhc or .profile (bash) file in their home dir. We also want .Xauthority from their home dir to make sure we can bring up the xterm. To do this, we fake the HOME dir temporarily. For this to work with multiple users, I create a subdirectory in the $SWD for each user ($SWD/.xterm_home/$USER ). There, I symlink the default .cshrc or .profile from the $SWD and also the Xauthority from the home dir. The default .csrhc (or .profile) sources the user's home dir version, then sources the VOVDIR setup file and enables the project. This leaves the user with a full login shell, complete with aliases, but ensures the correct VOVDIR and VOV_PROJECT_NAME are set up and enabled. Previously, the fake HOME dir was set to the SWD dir but since the Xauthority is linked, that did not work for multi-user. Hence, the added layer of subdirectories per user.
FlowTracer VOV-8733 22510 When a job finishes, any stdout or stderr given to the wrapper on the command line with the -O or -E option is removed if it is zero length. If the stdout/stderr file has contents but was not written during job execution, it is disconnected from the job unless it is the input to another job. These changes prevent a job from failing when multiple stdout files are found (from the -O option or parsing the command), but not all of those files are written during job execution.
FlowTracer VOV-8484   Suspend vw process upon receiving SIGTSTP when VOV_VW_PAUSE_ON_TSTP is set.
FlowTracer VOV-7610 20976 Fixed labels on jobs and files in FlowTracer and nc gui so the text does not overflow the borders of jobs or files.
FlowTracer VOV-8453   vovwxd now supports LSF queues. Please refer Accelerator Plus documentation for configuring Accelerator PlusLauncher slave. CONFIG(driver_script) should be set to vovlsf.tcl.
FlowTracer VOV-7825 8453 Fixed issue where some subslave warning and error messages were getting lost (not printed to any logs). Now, these issues get propagated to the WHY property of the job.
FlowTracer VOV-6396 20729 vtk_select_get and vtk_select_loop is improved to retrieve query results in dynamic sizes of chunks to prevent buffer overflow.
FlowTracer VOV-4100 20840 Added FDL flag for jobs ("FAllocatorGS distributed") that sets the new "@ISDISTRIBUTED@" flag on the job. This flag automatically enables NFS protection in the wrapper for that job, equivalent to using the "vw -N" option or setting VOV_VW_NFS_PROTECTION=2 when running the job. This can also be set or unset with vtk_transition_set. Use this flag for jobs that launch processes or sub-jobs on hosts other than the submission host.
FlowTracer VOV-9158 23297 Fixed vovlsfd issue where all occurrences of -R was getting removed from BSUB_OPTIONS.
FlowTracer 8002 21684 When VOV_FT_MULTIUSER environment variable is set before launching a slave, the slave will only accept jobs owned by the same user who started the slave. Jobs owned by other users will be refused.
Accelerator VOV-9498 23813 Fixed behavior of vovstop and related commands with regard to EXT:KILL - include and exclude specifications should work as expected now.

Errata

The following issues and defects are known to exist in this software release.

Product Internal Issue Description
All VOV-9759 The config subcommand of vovservermgr misprints confirmation messages
All VOV-9827 The vtk_slave_define Tcl command no longer supports automatic resources based on the vovslave name.
All VOV-9419 Setting config(useepoll) 1 in the policy.tcl config file has no effect.
All VOV-9771 The vtk_select_get command is not honoring the case of field names.
All VOV-9742 There is a new vovservermgr command that has been added for system administrators. This command has several subcommands that provide an easier way to set vovserver configuration and environment variables, and interface to memory chunking and scheduler tuning controls.
All VOV-9623 eventserver and epoll() are not starting with vovservermgr
All VOV-9748 Elaborate the vovservermgr -h help screen with more complete usage information.
FlowTracer VOV-9805 Priority based scheduling with vovwxd: vovlsfd should take a job's priority setting into account when submitting jobs.
FlowTracer VOV-9761 vovwxd ignores the setting of the VOV_FT_MULTIUSER environment variable.
FlowTracer VOV-9581 Take Ownership in the Node Editor is broken.