Initial support for GPUs in Redecomp and the Tribol MFEM interface #56

ebchin · 2024-01-30T19:46:01Z

This PR adds initial support for GPU data in the Redecomp library using MFEM's built in GPU support. This PR adds initial plumbing and tests on GPU to ensure GPU data transfers in Redecomp are functioning correctly. Follow-on PRs will add additional GPU support in Tribol and Redecomp.

Transfers from parallel data (grid functions and quadrature functions) on device to redecomp data are done as follows:

Device parallel data is copied to host using mfem::Vector::HostRead().
MPI communication moves host data from parallel rank to its redecomp rank.
Host redecomp data is written to a host pointer obtained using mfem::Vector::HostWrite().

The final result is redecomp data on the host, which can be copied to the GPU as needed in MFEM. Transfers from redecomp data to parallel data are done in reverse order.

The support for GPU data is extended to the Tribol MFEM interface, which leverages the Redecomp library to prepare data for processing by Tribol. Since Tribol does not support GPUs, redecomp data is left on host where Tribol can compute contact contributions.

GPU support has been added to the following examples (and corresponding tests):

domain_redecomp.cpp: use the Redecomp library to transfer parallel data on device to redecomp data on device and back
mfem_common_plane.cpp: multirank common plane contact using MFEM (with GPU) and Tribol
mfem_mortar_lm_patch.cpp: multirank contact patch test using MFEM (with GPU) and Tribol mortar

srwopschall

@ebchin - This looks good so far. I mostly have clarity type comments in this first pass. This mostly covers the examples so far.

srwopschall · 2024-02-15T00:18:36Z

src/examples/domain_redecomp.cpp

@@ -96,6 +98,8 @@ int main( int argc, char** argv )
  // very coarse meshes, though the decomposition can still fail with the simple
  // checks implemented.
  double max_out_of_balance = 0.05;
+  // device configuration string (see mfem::Device::Configure())


Maybe a short note on why you would see mfem::Device::Configure()? E.g. see mfem::Device::Configure() for device configuration options? Or is it something else?

srwopschall · 2024-02-15T00:22:49Z

src/examples/domain_redecomp.cpp

@@ -110,13 +114,24 @@ int main( int argc, char** argv )
  app.add_option("-t,--tol", max_out_of_balance,
    "Max proportion of out-of-balance elements in RCB decomposition.")
    ->capture_default_str();
+  app.add_option("-d,--device", device_config, 
+    "Device configuration string, see mfem::Device::Configure().")


See note above for minor clarity comment.

srwopschall · 2024-02-15T00:23:43Z

src/examples/domain_redecomp.cpp

+  SLIC_INFO_ROOT(axom::fmt::format("device: {0}\n", device_config));
+
+  // enable devices such as GPUs
+  mfem::Device device(device_config);


Does this constructor do error checking on 'device_config' or should you do that here?

It's verified here: https://github.com/mfem/mfem/blob/master/general/device.cpp#L207

Mention what device_config option(s) you have tested, run. Maybe one is just fine for the purposes of this PR.

srwopschall · 2024-02-15T00:33:39Z

src/examples/domain_redecomp.cpp

@@ -186,6 +201,9 @@ void RedecompExample(mfem::ParMesh& pmesh, int order, double max_out_of_balance)
  {
    pmesh.GetNodes(par_x_ref_elem);
  }
+  // move data to device (if device is available) and set host flag to stale


Consider clarifying what 'data' is being moved, and what the intent is here. We spoke about how while a parmesh is always on host, various data underneath that parmesh can be sent to device. In terms of this example, what is the intent here? Consider explaining briefly.

srwopschall · 2024-02-15T00:35:03Z

src/examples/domain_redecomp.cpp

@@ -269,7 +287,11 @@ void RedecompExample(mfem::ParMesh& pmesh, int order, double max_out_of_balance)
  /////////////////////////////////////////////////////////////////////////////

  timer.start();
+  // redecomp does transfers on host, so par_x_ref_elem data will be copied to


Just checking: This routine TransferToSerial() will take care of copying the par_x_ref_elem data back to host if it is on device, and then to the transfer to serial?

Yes, indirectly. The GetSubVector() method called here will get the correct pointer and make sure the DOF data is up-to-date.

srwopschall · 2024-02-15T01:02:43Z

src/examples/mfem_mortar_lm_patch.cpp

-  // are present in this problem.
-  mfem::BlockVector B_blk { A_blk->RowOffsets() };
+  // Create a RHS vector storing forces and gaps. Note no external forces are present in this problem.
+  mfem::Vector B_blk(A_blk->Height());


block RHS vector for forces and gaps makes sense. Why change it? Don't necessarily need to 'block' designation/functionality? You stil call it B_blk though. If I just saw the new code I wouldn't question it, just wondering in the context of these diffs.

srwopschall · 2024-02-15T01:04:03Z

src/examples/mfem_mortar_lm_patch.cpp

-  mfem::BlockVector B_blk { A_blk->RowOffsets() };
+  // Create a RHS vector storing forces and gaps. Note no external forces are present in this problem.
+  mfem::Vector B_blk(A_blk->Height());
+  B_blk.UseDevice(true);


Consider adding a comment here about what UseDevice() will do when set to true. This sounds obvious, but this is specifically a new line of code that enables use of the gpu. Should we always set this to true, or should we set it based on device_config (if that is appropriate)? Seems like we would want it to be false if we want to run this example on cpu only?

srwopschall · 2024-02-15T01:08:02Z

src/examples/mfem_mortar_lm_patch.cpp

+  // This API call returns the mortar nodal gap vector to an (uninitialized) vector. The function sizes and initializes
+  // the vector.
+  mfem::Vector gap;
+  gap.UseDevice(true);


Same comment about whether we always want 'true' to be passed in here.

srwopschall · 2024-02-15T01:08:41Z

src/examples/mfem_mortar_lm_patch.cpp

+  P_submesh.MultTranspose(gap, gap_true);
+
+  // Create a solution vector storing displacement and pressures.
+  mfem::Vector X_blk(A_blk->Width());


Again, clarify the move from a block vector to a vector?

src/examples/mfem_mortar_lm_patch.cpp

ebchin · 2024-02-17T00:45:39Z

@srwopschall, I think your comments are addressed in the latest commit. Let's discuss what kind of comments are appropriate in the MFEM examples, as the things that will be computed on device vs. host there are a bit in flux.

srwopschall · 2024-02-21T23:08:41Z

src/examples/domain_redecomp.cpp

+  SLIC_INFO_ROOT(axom::fmt::format("device: {0}\n", device_config));
+
+  // enable devices such as GPUs
+  mfem::Device device(device_config);


Mention what device_config option(s) you have tested, run. Maybe one is just fine for the purposes of this PR.

srwopschall · 2024-02-21T23:17:31Z

src/examples/domain_redecomp.cpp

@@ -186,6 +202,12 @@ void RedecompExample(mfem::ParMesh& pmesh, int order, double max_out_of_balance)
  {
    pmesh.GetNodes(par_x_ref_elem);
  }
+  // Copy DOF data to device (if device is available), set host data as invalid,


Consider taking this out of example and ensuring it is in appropriate test.

srwopschall · 2024-02-21T23:19:15Z

src/examples/domain_redecomp.cpp

@@ -269,7 +291,15 @@ void RedecompExample(mfem::ParMesh& pmesh, int order, double max_out_of_balance)
  /////////////////////////////////////////////////////////////////////////////

  timer.start();
+  // Redecomp does transfers on host, so par_x_ref_elem DOF data will be copied
+  // to host and the device DOF data of redecomp_x_ref_elem will be marked as


"[...] to host (if the data is on device)..."

srwopschall · 2024-02-21T23:19:38Z

src/examples/domain_redecomp.cpp

+  // invalid, and set the device data as valid in the memory bit flags (see
+  // mfem::Memory::FlagMask for bit flag definitions). Now, DOF data must be
+  // copied from device before the host can access it.
+  redecomp_x_ref_elem.ReadWrite();


take this out. A user/host-code won't need to make this call.

srwopschall · 2024-02-21T23:20:02Z

src/examples/domain_redecomp.cpp

+  // invalid, and set the device data as valid in the memory bit flags (see
+  // mfem::Memory::FlagMask for bit flag definitions). Now, DOF data must be
+  // copied from device before the host can manipulate it.
+  redecomp_x_ref_node.ReadWrite();


take this ReadWrite() out.

srwopschall · 2024-02-21T23:34:55Z

src/examples/mfem_common_plane.cpp

-  // mass update operator. Lumping is performed using row summation.
+  // This block of code builds a small-deformation elasticity explicit, lumped mass update operator. Lumping is
+  // performed using row summation. Elasticity and mass matrix contributions are computed on host (as implemented in
+  // MFEM), but mass inversion is done on device. These are only computed once, then stored on device.


Minor (if you decide to keep this comment in spirit of what's done on cpu vs gpu): but mass inversion is done on device if device_config enables GPU kernals.

OR: take this comment out. If a user of this example enables gpus using device_config, we shouldn't explicitly state what is on CPU vs GPU because that may change.

Leaning toward (2).

srwopschall · 2024-02-21T23:40:36Z

src/examples/mfem_mortar_lm_patch.cpp

+  SLIC_INFO_ROOT(axom::fmt::format("mu:     {0}", mu));
+  SLIC_INFO_ROOT(axom::fmt::format("device: {0}\n", device_config));
+
+  // enable devices such as GPUs


Update this comment per other examples.

srwopschall · 2024-02-21T23:42:38Z

src/examples/mfem_mortar_lm_patch.cpp

+  tribol::initialize(mesh.SpaceDimension(), MPI_COMM_WORLD);
+  // Copy the coords grid function DOF data to device (if available), set the host pointer flag as invalid, and set the
+  // device pointer to valid.
+  coords.ReadWrite();


Let's remove this, but verify this is in the test? Remove comment here too.

srwopschall · 2024-02-21T23:43:21Z

src/examples/mfem_mortar_lm_patch.cpp

+
+  // Copy the coords grid function DOF data to device (if available), set the host pointer flag as invalid, and set the
+  // device pointer to valid.
+  coords.ReadWrite();


Remove this and the comment. Verify if this line is needed in the test.

srwopschall · 2024-02-21T23:52:34Z

src/examples/mfem_mortar_lm_patch.cpp

-  // are present in this problem.
-  mfem::BlockVector B_blk { A_blk->RowOffsets() };
+  // Create a RHS vector storing forces and gaps. Note no external forces are present in this problem.
+  mfem::Vector B_blk(A_blk->Height());


If you stick with mfem::Vector, consider renaming B_blk to just B, and similiarly for A, unless you are actually storing one block of data.

ebchin added 15 commits December 13, 2023 12:12

mfem mortar example on gpu

b110f33

move data from device to host

7910f58

Merge branch 'fix_warnings' into gpu_testing

086aed8

fix spacing on comments

7f19736

more example options

35e3921

merge mortar examples

b72d886

different example runs

81cb658

fix example list

cde86b9

get rid of extra include

e9f980a

put config header first

9d96675

fix naming

2b3097d

common plane initial changes

6a81a25

more gpu related changes

fb1dea6

Merge branch 'develop' into gpu_testing

d94a23c

example updates

06707fe

ebchin requested review from kennyweiss, jamiebramwell and srwopschall January 30, 2024 19:46

ebchin self-assigned this Jan 30, 2024

srwopschall requested changes Feb 15, 2024

View reviewed changes

ebchin added 4 commits February 15, 2024 11:27

Merge branch 'develop' into gpu_testing

20b5973

Merge branch 'develop' into gpu_testing

595382c

gpu tests; address comments

3dac751

Merge branch 'develop' into gpu_testing

bf47cb3

ebchin marked this pull request as ready for review February 17, 2024 00:44

ebchin requested a review from srwopschall February 17, 2024 00:44

srwopschall reviewed Feb 23, 2024

View reviewed changes

take unnecessary data transfers out of examples

e528475

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for GPUs in Redecomp and the Tribol MFEM interface #56

Initial support for GPUs in Redecomp and the Tribol MFEM interface #56

ebchin commented Jan 30, 2024 •

edited

Loading

srwopschall left a comment

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

ebchin Feb 15, 2024

srwopschall Feb 21, 2024

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

ebchin Feb 15, 2024

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

srwopschall Feb 15, 2024

ebchin commented Feb 17, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

srwopschall Feb 21, 2024

Initial support for GPUs in Redecomp and the Tribol MFEM interface #56

Are you sure you want to change the base?

Initial support for GPUs in Redecomp and the Tribol MFEM interface #56

Conversation

ebchin commented Jan 30, 2024 • edited Loading

srwopschall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebchin commented Feb 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebchin commented Jan 30, 2024 •

edited

Loading