General guideline on using AWS FSx Netapp Ontap on build machine

[narek-ham]

Hi All

@henrykie

Is there a recommended way to configure FSx Netapp Ontap for UE builds?

Inspired by

Building Games with Unreal Engine and Horde on AWS | Unreal Fest 2024

We are trying to use flexclone volume (also flex volume) via NFS and getting strange, slow behavior plus RAM issues in build log though we get max 50-70% RAM usage.

We have tried to go up in resources up to 15000 IOPS and 512mb/s but it did not help at all.

Basically we are unable to build the project.

Editor target build via Build.bat:

Using Visual Studio 2022 14.38.33145 toolchain (C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.38.33130) and Windows 10.0.22621.0 SDK (C:\Program Files (x86)\Windows Kits\10).
Determining max actions to execute in parallel (32 physical cores, 32 logical cores)
  Executing up to 32 processes, one per physical core
Using Unreal Build Accelerator local executor to run 422 action(s)
  Storage capacity 40Gb
---- Starting trace: 250505_123747 ----
UbaSessionServer - Disable remote execution (remote sessions will finish current processes)
------ Building 422 action(s) started ------
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again
UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again

Building dedicated client or server via runUAT.bat:

    09:08:22  UbaSessionServer - Disable remote execution (remote sessions will finish current processes)
    09:08:22  ------ Building 4042 action(s) started ------
    09:08:28  UbaSessionServer - Allocation failed in MapViewOfFile (The paging file is too small for this operation to complete.).. process will sleep and try again
    09:08:28  [1/4042] Compile [x64] Module.EditorSubsystem.cpp
    09:08:28  [2/4042] Link [x64] UnrealEditor-EditorSubsystem.lib
    09:08:28  UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again

    [108/4643] Copy [x86-64] ChaosClothingSimulationMesh.ispc.generated.h 
    UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86\vol1_clone1 failed with error code c0000033 
    UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86 failed with error code c0000033 
    UbaSessionServer - NtCreateFile for TraverseDir on \ failed with error code c0000033

Mup errors in Windows Event Viewer:

{Delayed Write Failed} Windows was unable to save all the data for the file ;E:000000000005df76\172.31.49.86\vol1_clone1\j_work_zspace.vs\zSpace\v17\Solution.VC.db-shm; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere.

The solution in this post does not seem to work

Could you share the configuration for creating/mounting /using NFS volume for build purposes?

Thanks in advance

Best Regards