[1.8] Java jspawnhelper / illegal instruction

[nmwael]

Hi I am getting this error:

(host_framework.c:477:edmm_modify_pages_type) error: SGX_IOC_ENCLAVE_MODIFY_TYPES failed: (0) Bad address (EFAULT)
(libos_exec.c:50:__libos_syscall_execve_rtld) [P3:T52:jspawnhelper] error: BUG() ../libos/src/sys/libos_exec.c:50
(pal_exception.c:260:handle_ud) warning: Unknown or illegal instruction executed at die_or_inf_loop at cpu.h, libsysdb.so+0x83304 (addr = 0xfffaee8304)

I am utilizing EDMM, could it be stack or brk being too small, values are these:

sys.stack.size = "4M"
sys.brk.max_size = "1M"

As far as I can see this is the source being executed, i'm having a bit of trouble getting the exact call that's making it fail.

https://github.com/openjdk/jdk/blob/jdk-21%2B0/src/java.base/unix/native/jspawnhelper/jspawnhelper.c#L132

[chiache]

Hello, I am happy to look into this, but can you provide more instructions for reproducing this issue?

[nmwael]

Hello, I am happy to look into this, but can you provide more instructions for reproducing this issue?

FYI, Currently on vaction, will get back to you with details (eg minimal quickstart) next week(17-21 feb).

[chiache]

Hi, @nmwael Just a reminder for the instructions so that we can look into this issue. Thanks!

[nmwael]

I am working on it:)

[nmwael]

I've been trying to reproduce this using gramine-direct and no edmm enabled and are unable todo so, I dont know how comparable gramine-direct are to gramine-sgx..? I'll move on to enable edmm with gramine-direct before switching to gramine-sgx

[chiache]

Hello, gramine-direct and gramine-sgx are identical at the library OS-level, but differ at the layer for host API translation (which we call PAL). There might be some distinctions between running gramine-direct and gramine-sgx, such as memory allocation/deallocation.

[nmwael]

I dont think it's actually related to gramine as such. It looks like something ~java goes crazy consuming memory when it think's there's 1tb memory it seems to throw all switches away at begin consuming. I've attached logs
JavaSpawnProcess.zip. So you can see the difference between edmm and no edmm runs. Having both edmm (sgx.edmm_enable = true) and enclave limit(sgx.enclave_size = "512M"
) fails. Are there a way to force gramine to report a specific memory size when utilizing edmm?

[chiache]

I have similar experience with JVM before. Some of JVM features require specific memory layout (for me, it was compressed OOPs), and I had to disable these features to prevent JVM from consuming too much memory.

Judging from the bug report, I don't think this is a problem specific to EDMM. My guess is that the library OS is trying to free a page, but failed to do so at the host level. When EDMM is available, SGX_IOC_ENCLAVE_MODIFY_TYPES is first used to "deactivate" a page before actually removing the page from the enclave. However, the in-kernel driver probably does not recognize this address and refuse to serve the operation. This is likely a bookkeeping bug, where the library OS believe certain page is allocated but actually was not at the host level.

In this case, reproducing this bug will be crucial to identifying this problem, since a bookkeeping bug is hard to diagnose from simply examining the code.

[nmwael]

Oh and here is my repo https://github.com/nmwael/javaspawnprocess.git , it might need some cleanup. So let me know if you will use it.

[chiache]

Thanks! I will try if I can reproduce it.

[nmwael]

looks to me as if the jvm consumes every bit of memory, and then when spawning the process nothings left. But i'm not completely sure, there is some notes on behavioural changes when theres more than 32gb of sysmem being reported. However I havent tried
-XX:MaxRAM=size
To see if that overrules it all. For the repo I tried reducing the application memory usage as much as possible.