GPU CUDA API Guide

CUDA Function Declarations

CUDA requires that all functions be declared according to where they are going to be called and executed. Given the custom API scripts are called and run on the GPU, they must be declared as “__device__” functions as so:

 

__device__

void calculateForce(NCudaApiTypesV1_0_0::CApiElement& element1,

                    NCudaApiTypesV1_0_0::CApiElement& element2,

     …

There are other Function declarations such as “__Global__” and “__Host__”, however these should not be required.

CUDA Programming Style

The CUDA API is closer to the C++ programming, in terms of style compared to the OpenCL API. This means that basic operations usually take a much simpler form. Some examples of this are as follows:

Type of Function	OpenCL Function	CUDA Function
Vector arithmetic	vvSubtract(a,b)	a - b
	vvAdd(a,b)	a + b
	vecMultiply(a,b)	a * b
	vecDivide(a,b)	a / b
Vector Operations	vvDot(a,b)	a.dot(b)
	vvCross(a,b)	a.cross(b)
	vecNormalize(a)	a.normalize()

 

For a full explanation of what functions are available see the API help documents, which can be found here:

“<EDEM install directory>\202X.X\EDEM\src\Api\Help”.

CUDA Class Declarations

Multiple different particle types can be used with the CUDA API whilst only multi-spheres operate on the OpenCL API. Given this difference, the CUDA API operates with a more generalized set of classes and functions compared to OpenCL.

There are some differences which stem from CUDA’s programming syntax compared to OpenCL’s. However there are other differences which originate directly from EDEM. For example, when retrieving the radius of an element in CUDA for a Multi-Sphere or a Sphero-Cylinder the following operation would be used:

element1.getPhysicalRadius() ;

The closest equivalent operation which would work for polyhedral particles is:

element1.getVolumeSphereRadius();

This is not a syntactical difference, but rather a real difference in the value. In this case it is due to the fact that Polyhedral particles do not have an inherent radius like Multi-Spheres and Sphero-Cylinders do.
Therefore when developing API scripts on CUDA, several sources are required to fully understand what functionality is available:

• The internal EDEM API documentation: This details the EDEM specific CUDA functions and classes. This can be found at “<EDEM install directory>\202X.X\EDEM\src\Api\Help” then select any html file.
• CUDA Documentation: This details standard CUDA functionality. For the full documentation: https://docs.nvidia.com/cuda/index.html
• Example code can be found on the Altair community page: https://community.altair.com/community. The following example codes provide an insight into how these different functions should be implemented.

CUDA Getter and Setter Functions

In order to maintain backward compatibility going forwards with the CUDA API, a range of getter and setter functions have been implemented. These allow users to retrieve the same information they could using the OpenCL solver. For example:

element1->position;

Is equivalent to:

element1.getPosition();

CUDA Debugging

This is similar to the OpenCL debugging process. If errors are present EDEM will detail the errors in a dialog box when your simulation is started. An additional Method for debugging which allows a user to gain real time information from the simulation is to run an attached simulation through the command line.

This involves the following steps:

1. Add printf() functions as desired throughout your API to output values of interest
2. Locate your EDEM.exe directory at “<EDEM install directory>\202X.X\EDEM\bin\”
3. Run the command EDEM.exe --no-free as such: C:\Program Files\Altair\2022.2\EDEM\bin\edem.exe --no-free
4. Load your API model into EDEM and run the simulation
5. The results of the output should be displayed in the command window.

CUDA Main Functions

Due to the differences in classes and data types between the CUDA file and the multi-sphere OpenCL file, modifications need to be made to the “ExternalForce” and “CalculateForce” functions.

The main Function naming in the CUDA API follows a different convention to the OpenCL API. In the CUDA file, the file name should not prefix the main Function names.

Examples of these modified main Function inputs are as follows:

__device__

void calculateForce(NCudaApiTypesV1_0_0::CApiElement& element1,

                    NCudaApiTypesV1_0_0::CApiElement& element2,

                    NCudaApiTypesV1_0_0::CApiInteraction& interaction,

                    NCudaApiTypesV1_0_0::CApiContact& contactData,

                    NCudaApiTypesV1_0_0::CApiSimulation& simulationData,

                    NCudaApiTypesV1_0_0::CApiContactResults& contactResult)

{

}

__device__

void externalForce(NCudaApiTypesV1_0_0::CApiParticle& particle,

                   NCudaApiTypesV1_0_0::CApiSimulation& simulation,

                   NCudaApiTypesV1_0_0::CApiTotalForce& result)

{

}

In addition to these functions, there are the configForTimestep functions. These allow custom functionality to be modified at each Time Step. Even if they are not used, it is mandatory to include these functions:

__device__

void configForTimeStepParticleProperty(NCudaApiTypesV1_0_0::CApiCustomPropertyData& particleProperties)

{

}

__device__

void configForTimeStepTriangleProperty(NCudaApiTypesV1_0_0::CApiCustomPropertyData& triangleProperties)

{

}

The configForTimeStepParticleProperty is used for both contact and external force models. When it is used in a plugin that combines calculateForce and externalForce models, those models must have the same custom properties and custom property order, otherwise the wrong values can get modified.

The configForTimeStepTriangleProperty does not get the actual element custom property values, rather the difference in values between timesteps (this is not the same as triangle contact deltas). This was done for easier mGPU implementation as the triangle custom property values are unused in any calculations, as they are intended for post-simulation analysis.

CUDA Data Precision

In the CUDA API you can specify the precision that you want their data to be handled with. This allows API scripts to take advantage of the speed benefits of the hybrid and single precision solvers. The three CUDA specific data types are: real, longReal and customPropReal. See the table below which shows each data type’s precision depending on the solver settings:

Data Type	Solver Precision			Usage
	Double	Hybrid	Single
real	double	single	single	Most situations
longReal	double	double	single	High precision data requirements
customPropReal	double	double	single	custom properties

 

To implement these data types simply specify them when declaring a variable:

real exampleVariable = ...

 

Here the real Data Type is used and is an appropriate datatype for most situations. For more information about these data types and the functions associated with them see the API help documents.

CUDA Asserts

An option to enable asserts for enhanced error diagnostics has been added to the CUDA solver. Enabling this causes a minor slowdown in simulation performance. This can be enabled from the GUI in the Simulator->Advanced Setting... window or from the command line using the option "--cuda-asserts 1". In order to use this effectively, EDEM should be running a command line simulation or EDEM should be started from the command line using the option --no-free, so that any additional console output from an assert can be sent to EDEM support. 

CUDA Header Files

Sphero-Cylinders

API users should be cautious when implementing contact models for the CUDA solver to not rely on the contact normal being in the same direction as the vector from the API element 'position' member to the contact point. While this does work for Multi-Sphere shapes, it will not work for Sphero-Cylinder or Polyhedral shapes. Users should instead try to use the contactNormal member which is passed in the overlapData API structure.

Apart from this, all contact properties for Sphero-Cylinders have a similar definition to those for Multi-Sphere particles.