Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gpu aware mpi #341

Merged
merged 47 commits into from
Feb 21, 2025
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
1651980
add initial implementation for gpu aware mpi communication of halos
Jul 22, 2024
f10b883
Merge branch 'develop' into gpu_aware_mpi
Jul 25, 2024
3d8c9c6
use tdp calls to launch kernels
Jul 29, 2024
4e3e8b6
use reduced halos for allocating send and receive buffers on the device
Jul 30, 2024
bf66890
update halo free function to gree device pointers
alexei-borissov Aug 1, 2024
4a026eb
debugging
alexei-borissov Aug 13, 2024
f854feb
remove print statements
alexei-borissov Aug 14, 2024
5d4af34
add model copies, make sure we're waiting on the target rather than t…
alexei-borissov Aug 26, 2024
3837d6a
remove debugging print statements
alexei-borissov Aug 26, 2024
4d3cbcf
don't double count halo sizes
alexei-borissov Sep 6, 2024
a6d35d5
add nvtx timers
alexei-borissov Oct 8, 2024
a702812
initial graph implementation
alexei-borissov Oct 10, 2024
63b4aea
debugging graph api implementation
alexei-borissov Oct 11, 2024
58432e0
debugging graph api implmentation
alexei-borissov Dec 2, 2024
f8e7529
debugging
alexei-borissov Dec 19, 2024
18c0639
Merge branch 'gpu_aware_mpi' into gpu_aware_mpi_graph_debug
alexei-borissov Dec 19, 2024
9523734
remove graph implementation
alexei-borissov Dec 19, 2024
a742f1b
Merge branch 'develop' into gpu_aware_mpi
alexei-borissov Dec 19, 2024
f3eb55d
reintroduce copy of cs to device
alexei-borissov Dec 19, 2024
97b8f25
check whether we're using the target halo implementation
alexei-borissov Jan 6, 2025
f3feb80
remove nvtx
alexei-borissov Jan 6, 2025
87d1507
remove excess calls to halo_post and wait
alexei-borissov Jan 6, 2025
53e19fc
tidy
alexei-borissov Jan 6, 2025
6650806
remove unused variable
alexei-borissov Jan 6, 2025
c4f0d5c
Merge branch 'develop' into gpu_aware_mpi
alexei-borissov Jan 9, 2025
6ddf756
comment copymodeltodevice
alexei-borissov Jan 9, 2025
f67b38c
add halo device model initialise and free functions
alexei-borissov Jan 13, 2025
dfe316c
remove commented code
alexei-borissov Jan 13, 2025
79ac9d7
Merge branch 'gpu_aware_mpi' into gpu_aware_mpi_graph_debug
alexei-borissov Jan 13, 2025
3863565
add function for initialising halo model on device
alexei-borissov Jan 17, 2025
a123254
reintroduce calls to graph launch that were removed when merging main…
alexei-borissov Jan 19, 2025
dfb45c6
remove debugging print statements
alexei-borissov Jan 20, 2025
b87282f
deactivate graph implementation
alexei-borissov Jan 20, 2025
f300c01
remove lb_halo_swap
alexei-borissov Jan 20, 2025
808cabc
remove commented code
alexei-borissov Jan 22, 2025
a10ae9b
set halo target to zero
alexei-borissov Jan 22, 2025
7bd91bb
don't allocate zero sized arrays
alexei-borissov Jan 22, 2025
5bda6d0
move initialise device functions inside check for ndevice non-zero
alexei-borissov Jan 22, 2025
cb537a3
add tdpassert
alexei-borissov Jan 22, 2025
2657175
don't construct graph nodes when request size is zero
alexei-borissov Jan 22, 2025
a28d69b
correct send recv counts
alexei-borissov Jan 22, 2025
9c911d4
use halo->count when deciding to skip building nodes which would have…
alexei-borissov Jan 23, 2025
e0b7de9
copy send and receive buffers from device to host if not using gpu aw…
alexei-borissov Jan 23, 2025
7d1cf9e
don't fiddle with haloscheme
alexei-borissov Jan 23, 2025
2cd14d2
remove debugging print statement
alexei-borissov Jan 23, 2025
ac457af
promote scount to size_t
alexei-borissov Jan 23, 2025
c544140
promote rcount to size_t
alexei-borissov Jan 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
move initialise device functions inside check for ndevice non-zero
alexei-borissov committed Jan 22, 2025
commit 5bda6d0d2672bcbe53fdf1cdf976817b45ae47c8
7 changes: 4 additions & 3 deletions src/lb_data.c
Original file line number Diff line number Diff line change
@@ -589,12 +589,13 @@ static int lb_init(lb_t * lb) {
cs_target(lb->cs, &cstarget);
tdpMemcpy(&lb->target->cs, &cstarget, sizeof(cs_t *),
tdpMemcpyHostToDevice);

lb_data_initialise_device_model(lb);
}

lb_mpi_init(lb);
lb_model_param_init(lb);

lb_data_initialise_device_model(lb);
lb_memcpy(lb, tdpMemcpyHostToDevice);

return 0;
@@ -1435,6 +1436,8 @@ int lb_halo_create(const lb_t * lb, lb_halo_t * h, lb_halo_enum_t scheme) {
tdpAssert( tdpMemcpy(h->target->recv, h->recv_d, 27*sizeof(double *),
tdpMemcpyHostToDevice) );

halo_initialise_device_model(h);

if (have_graph_api_) {
lb_graph_halo_send_create(lb, h, send_count);
lb_graph_halo_recv_create(lb, h, recv_count);
@@ -1444,8 +1447,6 @@ int lb_halo_create(const lb_t * lb, lb_halo_t * h, lb_halo_enum_t scheme) {
free(send_count);
free(recv_count);

halo_initialise_device_model(h);

return 0;
}