Add GPU profiling to CI

[sbryngelson]

Add GPU profiling on Phoenix to CI with nsight sys.

[sbryngelson]

Likely has to be done by someone at Gatech and so access to Phoenix for debugging purposes.

[sbryngelson]

More detail: Should run MFC profiling on a GPU node and write the command line results to CI. Should probably compare results to known results from master (like the benchmark workflow does).

[Malmahrouqi3]

Simple poc to generate non-gpu nsys report in the CI https://github.com/Malmahrouqi3/MFC-mo2/actions/runs/15799705418/job/44536371819
To process the results for single/multiple report(s), Nsight Systems UI would do it. I do not know about how to do diff, but surely there has to be a way via nsys cli.

No sure yet how it is gonna behave with Phoenix, but integrating this into self-hosted test.yml is the way.
Also, never tested it, but I guess it is possible to profile a run locally using the workflow file with github cli gh act -P ubuntu-latest=-self-hosted -W .github/workflows/profiling.yml or by simply installing Nsights then nsys profile -o report ./mfc.sh test -a --gpu

[sbryngelson]

@Malmahrouqi3 thanks, yes you can add it that way, it's a good POC. you actually just pass different commands to the nsys command, nsys --help or whatever, and it will give you the information on the command line and you would just dump that into the output of the CI. I'm not sure if a diff is the best way to go or not (presumably that would go into benchmarking since it would have to be PR vs. Master).

The easiest way to get a merge in, that could possibly be migrated to benchmarking later, would be to just run the nsys command with appropriate flags, and it will tell you what the most expensive subroutines are. We have some documentation on nsight profiling in the MFC docs ./mfc.sh run <...> --profile should work I think. But I think that will just create a report file you look at in the GUI.

[sbryngelson]

NVHPC_ACC_TIME or NVCOMPILER_ACC_TIME may be an easy workaround for this. Should just work at runtime.