Saving outputs when running sopflow with HiOp PriDec

[nkoukpaizan]

Issue type

• New feature
• Bug
• Discussion
• Other

Relates to

• OPFLOW
• SOPFLOW
• SCOPFLOW
• TCOPFLOW
• CMake build system
• Spack configuration
• Manual
• Web docs
• Other

Summary
When running sopflow at scale on Summit with HiOp PriDec on GPUs, the code would crash when attempting to write the output files for each contingency (when this was requested). It looked to me like ExaGo was trying to run multiple instances of opflow again to convert to PS format, and it was running out of GPU memory. Note that HiOp PriDec had already converged when this happened. In the end, I turned off the final output generation to collect the scaling data.

@abhyshr, would you mind elaborating on how the outputs are generated? Is there a way to get the converged solution directly?

cc @pelesh @maksud

[pelesh]

CC @nychiang @cnpetra

[cnpetra]

By "output" you mean the standard convergence history of the solvers or some other output like files?

[nkoukpaizan]

By "output" you mean the standard convergence history of the solvers or some other output like files?

The latter. I'm referring to the solution output file. This defines what that is for opflow. For sopflow, a file gets written per recourse problem.

Somehow, we need to get the solution vector and convert it to whatever format is requested, and I don't fully understand what ExaGO is doing there.

[nkoukpaizan]

To be more specific, I tracked the failure to:

ExaGO/src/sopflow/solver/hiop/sopflow_hiop.cpp

Line 495 in 12d2968

ierr = OPFLOWSolve(opflow);

. There is a OPFLOWSolve(opflow) within SOPFLOWSolverGetSolution_HIOP. I would expect the solution to be readily available at this stage.