GPU OpenCL Guide

Introduction

The EDEM OpenCL GPU module makes use of the latest GPGPU technology (General Purpose computing on Graphical Processing Units). By utilizing cutting edge GPU cards EDEM is able to deploy contact model calculations across literally thousands of processing cores and deliver simulation results many times faster than by using traditional desktop CPUs.

The EDEM OpenCL GPU solver engine is based on OpenCL to ensure cross-platform support on both Windows and Linux and also ensures that a wide range of GPU cards are supported regardless of manufacturer, whether that be NVIDIA, AMD or Intel.

The EDEM OpenCL GPU solver engine is integrated directly into the EDEM environment. Setting and running simulations with the new EDEM OpenCL GPU engine is as simple as enabling the GPU engine and then running EDEM OpenCL the same way you would any other simulation.

The range of GPU cards available and the technology behind them is constantly changing and moving fast so if you are not sure of what hardware you need please get in touch and our engineers will be happy to talk about results from our most recent rounds of testing.

As of EDEM 2018, EDEM GPU contains a fully double precision solver.

Key improvements to the EDEM OpenCL in EDEM 2019 include support for GPU API models and multi-GPU support.

EDEM 2018.2 introduced support for multiple GPUs in a single simulation. This increases the available GPU memory, increasing the maximum size and number of particles per simulation as well as improving performance.

Hardware Setup

These are the steps necessary to be able to run EDEM OpenCL GPU:

Graphics Card Drivers

Before running EDEM OpenCL GPU, please make sure that the graphic card drivers installed in your machine are the latest version.

EDEM GPU Device Test

The device test is required before running EDEM OpenCL GPU on a machine for the first time. The device test will check each of the GPUs compatibility and make the GPU cards available to the EDEM OpenCL Simulator. The test will also ensure that the current GPU devices supports OpenCL version 1.2 or more, which needs to be supported in order to make use of Double Precision.

To run the test, go to Tools > Options >Simulator Engine>OpenCL. Under Setup GPU Devices select the relevant GPU devices and click on Start test. It should only take a few seconds. If there is more than 1 GPU card in the same machine the user can select which cards to use to run a simulation. The example below shows the simulator engine tree structure 2 GPUs selected.

Under Test GPU Devices, click on Start test. If it is successful, the GPU devices are ready to be selected as Simulator Engines, if not, EDEM OpenCL will provide any information it has about the problem.

The above example shows both the 2080Ti cards passed the GPU Device test.

Note that the test has to be run when EDEM OpenCL GPU is run using the GUI, i.e. via a regular EDEM start of Remote Desktop connection. If running via command line the test is bypassed and if the GPU device is not compatible the simulation will not run.

Start Simulating

That is it! Once the test has passed, EDEM OpenCL GPU is ready to go and you can start your simulation as normal.

Simulator Settings

The EDEM OpenCL GPU solver engine will automatically allocate memory and resources as required as the simulation runs.

The EDEM OpenCL GPU solver engine provides options for the user to review the memory settings in the Simulator > Tools > GPU Memory Calculator. This might be useful in situations where the simulation is getting close to the limit of memory available and the user can see the estimated memory usage. In addition when the GPU solver engine is selected in the EDEM Simulator the “Display GPU Memory” option can be selected which will display GPU memory usage on the EDEM visualizer.

When using Multi GPU the user is able to specify how the domain is divided by selecting the number of GPU devices and the direction of the Load Balance Split as shown:

The examples below show the simulation domain divided using 2 and 4 GPUs. Best practice is to select the most appropriate axis in the Load Balance Split to evenly divide the bulk material between the selected GPUs.

Running from Command Line

To run EDEM OpenCL GPU from the command line, the simulator engine has to be specified with the flag “-E” followed by the correspondent engine number:

-E 0 runs the CPU engine

-E 1 runs the GPU Open CL engine

In addition, the user can set the Device Index for the GPU. Note that setting the device index bypasses the EDEM OpenCL GPU Device Test, this is designed for use on hardware where the EDEM GUI is not available (such as running on a cluster). If more than 1 GPU card is available the required card can be specified, the name of each card is provided in the command line, for example:

Device index GPU device name

0 Nvidia Quadro GP100

1 AMD Radeon Pro WX 9100

2 Nvidia GeForce GTX 1080

To run using the Nvidia Quadro GP100 card the following command would be required:

edem.exe -c -i c:\processing\mysim.dem -E 1 -D 0

or to use the AMD Radeon Pro WX 9100 card the -D 1 flag would be required.

For multi GPU GPU you would add the indices of all GPU cards to the command line. For example, to run the simulation on GPU devices 0 and 1, as specified by the Device index, you would enter:

edem -c -i c:\processing\mysim.dem -E 1 -D 0+1

The flags from the CPU version of EDEM are still be valid in order to set number of CPU processors used in the factory domains (-p), real time to simulate (-r), data save interval (-w), Cell grid size for factory space and particle to particle contact search (-g) and time step used (-t).

For an example, to run via the command line with the GPU engine using only one card you might enter:

edem.exe -console -i simulation_deck.dem -E 1 -D 0 -p 4 -r 2 -w 0.1 –g 4 -t 1e-6

For multi GPU you would add the indices of all GPU cards to the command line, as shown previously. For example, to run the simulation on GPU devices 0 and 1 you might enter:

edem.exe -console -i simulation_deck.dem -E 1 -D 0+1 -p 4 -r 2 -w 0.1 –g 4 -t 1e-6

For multi GPU you can define the axis about which the domain is split between devices. An example of running a 2 GPU simulation via command line with a split in the Y axis is as follows:

edem.exe -console -i simulation_deck.dem -E 1 -D 0+1 --gpu-split Y -p 4 -r 2 -w 0.1 –g 4 -t 1e-6

Known Issues and Limitations

The EDEM 2020 GPU Solver Engine contains the following known issues:

Data Saving

All data saved to the EDEM .h5 files is carried out using the CPU.

The OpenCL and CUDA solver engines have different ways of handling timesteps that require data saves. Any timesteps that require a data save when the OpenCL solver is the selected engine will be run fully on the CPU. There is no passing simulation data from the GPU to the CPU (unless specified within a GPU-API plugin). The is not the same for the CUDA solver. With the CUDA solver as the selected engine, all timesteps are performed on the GPU and the data is passed back to the CPU for saving the data.

Contact Models

The following Contact Models are not supported on the GPU.

Bonding (Legacy)
Tribocharging
Electrostatics
Moving Plane (Deprecated, replaced with Geometry Motion > Conveyor Kinematics which are fully supported on GPU)

The following Base Contact Models are fully supported on the EDEM GPU and Multi GPU:

Hertz-Mindlin
Hertz-Mindlin with JKR
Edinburgh Elasto-Plastic Adhesion Model (EEPA)
Linear Spring
Hysteretic Spring

The following Rolling Friction models are fully supported in the EDEM GPU:

Standard Rolling Friction
RVD Rolling Friction

The following Additional Models are fully supported in the EDEM GPU:

Heat Conduction
Moving Plane
Archard Wear
Bonding V2
Relative Wear
Linear Cohesion V2

Track Collisions

Track collisions is not supported in EDEM OpenCL GPU.

Dynamic Domain

Dynamic Domain is not supported in the EDEM 2022 Multi GPU. Dynamic Domain is supported when using 1 GPU.

Cylindrical Periodic Boundaries

Cylindrical Periodic Boundaries are not supported in EDEM 2022 OpenCL GPU.

Time Integration

Position Verlet and Velocity Verlet time integration methods are not supported in EDEM 2022 OpenCL GPU. Only Euler time integration is supported.

Data Export

Exporting data having used EDEM OpenCL GPU may output the columns of the selected queries in a different order than the same simulation run in EDEM CPU.

Factories

The EDEM Open CL GPU uses a smaller, separate domain for the particle factory. This is because new particles are introduced by the CPU and particles within the factory domain need to be tracked to prevent overlapping with new particles. Once the particles leave the factory domain, they will be transferred to the GPU solver engine and are no longer tracked by the CPU factory. If any particles re-enter the factory domain, the CPU will not acknowledge them and this can lead to overlapping particles and ‘explosions’.

If several factories are so close to one another that their particles might pass through each other’s factory domain, the user can enable “Use Factory Group” under GPU factory in the GPU Parameters tab on the Simulator. This feature allows the user to group several factories together in order to make EDEM deal with them as one factory domain on the CPU solver engine.

The larger the number of particles inside the factory domain (one factory or one group of factories), the more CPU usage there will be and this will have an effect on performance. In extreme cases, where the factory covers the whole domain, all particles would be processed by the CPU solver engine.

To improve performance EDEM will detect when a factory stops generating particles. At this point the corresponding factory domain would become inactive and all the particle information within it will be handed over to the GPU Solver Engine to proceed with the calculations.

Solve Report

When running in GPU Total particles removed in the solve report show the incorrect number.

Troubleshooting

Here are some hints and tips that might help if you experience any issues while using the EDEM GPU Solver Engine:

OpenCL Error Messages

If EDEM displays any error message related to OpenCL, try updating to the latest driver first. GPU card manufacturers are regularly updating drivers to fix bugs so make sure you’re on the latest driver and check regularly.

The test is a standard hardware test for GPU. If the test fails, please make sure your Graphics Card drivers are up to date and try again or contact support with details of the GPU device used.

Only once the test has been run successfully will the GPU option be accessible from the Selected Engine dropdown menu.