CUDA GPU Guide
Introduction
The EDEM GPU module makes use of the latest GPGPU technology (General Purpose computing on Graphical Processing Units). By utilizing cutting edge Nvidia GPU cards EDEM is able to deploy contact model calculations across literally thousands of processing cores and deliver simulation results many times faster than by using traditional desktop CPUs.
The EDEM GPU and Multi-GPU solver engine is based on CUDA. The CUDA code requires compute capability of 3.5 or later and is compatible with Windows or Linux.
The range of GPU cards available and the technology behind them is constantly changing and moving fast so if you are not sure of what hardware you need please get in touch and our engineers will be happy to talk about results from our most recent rounds of testing.
The CUDA GPU engine offers a range of precision modes including double, hybrid and single.
A CUDA API is available and documented in the API section and the API help section available in the EDEM folders.
As of EDEM 2018, EDEM GPU contains a fully double precision solver.
Hardware Setup
These are the steps necessary to be able to run EDEM CUDA GPU:
Graphics Card Drivers
Before running EDEM CUDA GPU, please make sure that the graphic card drivers installed in your machine are the latest version.
EDEM GPU Device Test
The device test is required before running EDEM CUDA GPU on a machine for the first time. The device test will check the GPU compatibility and make the GPU card(s) available to the EDEM Simulator. The test will also ensure that the current GPU devices support compute capability 3.5 or more, which needs to be supported in order to make use of Double Precision.
To run the test, go to Tools > Options >Simulator Engine>CUDA. Under CUDA Engine Settings select the relevant GPU devices and click on Start test. It should only take a few seconds. If there is more than 1 GPU card in the same machine the user can select which cards to use to run a simulation.
Under Test GPU Devices, click on Start test. If it is successful, the GPU devices are ready to be selected as Simulator Engines, if not, EDEM will provide any information it has about the problem.
The above example shows two cards having passed the CUDA GPU Device test.
Note that the test has to be run when EDEM CUDA GPU is run using the GUI, i.e. via a regular EDEM start of Remote Desktop connection. If running via command line the test is bypassed and if the GPU device is not compatible the simulation will not run.
Connectivity Test
In the event the Multi-GPU fails the test, the Multi-GPU Connectivity Test option performs a series of tests which can be used by the EDEM support team for diagnosing the issue.
Start Simulating
That is it! Once the test has passed, EDEM CUDA GPU is ready to go and you can start your simulation as normal.
Simulator Settings
The EDEM CUDA GPU solver engine will automatically allocate memory and resources as required as the simulation runs.
The EDEM CUDA GPU solver engine provides options for the user to review the memory settings in the Simulator > Tools > GPU Memory Calculator. This might be useful in situations where the simulation is getting close to the limit of memory available and the user can see the estimated memory usage. In addition when the CUDA GPU solver engine is selected in the EDEM Simulator the “Display GPU Memory” option can be selected which will display GPU memory usage on the EDEM visualizer.
Preserve Contact Between Triangles
When working with physics models which have a contact history, particles can be seen to ‘jump’ (contact force spikes) when the particle moves from one geometry element to the next. This is because EDEM regards this as a new contact so the contact history is not preserved. In the Advanced settings menu the option to "Preserve Contacts Between Triangles" is available. This will keep the contact history for a particle passing from one element to another.
A user-defined tolerance is required for this option, this is to allow for geometries with minor variation in angle to be considered continuous. The tolerance is defined by an angle, and is the maximum angle between planes at which they would be considered co-planar.
For example:
A difference in angle of 10 degrees where contact is considered continuous vs an angle of 70 where it is considered as a separate contact. Note the default value is 0.01 degrees, 10 degrees would be considered an extreme value.
Running from Command Line
To run EDEM GPU from the command line, the simulator engine has to be specified with the flag “-E” followed by the correspondent engine number:
-E 0 runs the CPU engine
-E 1 runs the GPU OpenCL engine
-E 2 runs the GPU CUDA engine
In addition, the user can set the Device Index for the GPU. Note that setting the device index bypasses the EDEM GPU Device Test, this is designed for use on hardware where the EDEM GUI is not available (such as running on a cluster). If more than 1 GPU card is available the required card can be specified, the name of each card is provided in the command line, for example:
Device index GPU device name
0 Nvidia Quadro GP100
1 Nvidia GeForce GTX 1080
To run using the Nvidia Quadro GP100 card the following command would be required:
edem.exe -c -i c:\processing\mysim.dem -E 2 -D 0
or to use the Nvidia GeForce GTX 1080 card the -D 1 flag would be required.
The flags from the CPU version of EDEM are still be valid in order to set number of CPU processors used in the factory domains (-p), real time to simulate (-r), data save interval (-w) and time step used (-t).
For an example, to run via the command line with the GPU engine using only one card you might enter:
edem.exe -console -i simulation_deck.dem -E 2 -D 0 -p 4 -r 2 -w 0.1 -t 1e-6
Known Issues and Limitations
The EDEM 2022 CUDA GPU Solver Engine contains the following known issues:
Data Saving
All data saved to the EDEM .h5 files is carried out using the CPU.
The OpenCL and CUDA solver engines have different ways of handling timesteps that require data saves. Any timesteps that require a data save when the OpenCL solver is the selected engine will be run fully on the CPU. There is no passing simulation data from the GPU to the CPU (unless specified within a GPU-API plugin). The is not the same for the CUDA solver. With the CUDA solver as the selected engine, all timesteps are performed on the GPU and the data is passed back to the CPU for saving the data.
Block Factories
Block factories will generate blocks of materials using the CUDA solver, however they don't have contact information, heavily compressed blocks are likely to become unstable. When using a block factory with CUDA a warning message will appear.
Contact Models
For a full list of compatible models see GPU compatability section
Track Collisions
Track collisions is not supported in EDEM CUDA GPU.
Dynamic Domain
Dynamic Domain is not supported in EDEM CUDA GPU.
Cylindrical Periodic Boundaries
Cylindrical Periodic Boundaries is not supported in EDEM CUDA GPU.
Time Integration
Position Verlet and Velocity Verlet time integration methods are not supported in EDEM CUDA GPU. Only Euler time integration is supported.
Troubleshooting
Here are some hints and tips that might help if you experience any issues while using the EDEM GPU Solver Engine:
CUDA Error Messages
If EDEM displays any error message related to CUDA, try updating to the latest driver first. GPU card manufacturers are regularly updating drivers to fix bugs so make sure you’re on the latest driver and check regularly.
Only once the test has been run successfully will the GPU option be accessible from the Selected Engine dropdown menu.