8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp

[YaSuenag]

jhsdb jstack --mixed would not work when attaches to the process runs with -Xcomp.

It has been reported by @pchilano in #27728. You can reproduce the problem with Test.java (attached JBS). You can see following stack.

----------------- 646689 -----------------
"Thread-0" #24 prio=5 tid=0x00007f1cec18c890 nid=646689 waiting on condition [0x00007f1cd0158000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
0x00007f1cf3b7f462 __syscall_cancel_arch + 0x32
0x00007f1cf3b7375c __internal_syscall_cancel + 0x5c
0x00007f1cf3b766a8 ___pthread_cond_timedwait + 0x178
0x00007f1cf270e1e9 PlatformEvent::park_nanos(long) + 0x119
0x00007f1cf2005f4c JavaThread::sleep_nanos(long) + 0xfc
0x00007f1cf218789f JVM_SleepNanos + 0x28f
0x00007f1cdb95f299 java.lang.Thread.sleepNanos0(long) + 0x99 (Native method)

Thread.sleepNanos0 is the bottom stack, but actually it has more call frames. You can see them with -XX:+PreserveFramePointer.

----------------- 646841 -----------------
"Thread-0" #24 prio=5 tid=0x00007f4a0018c9e0 nid=646841 waiting on condition [0x00007f49e4fd7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
0x00007f4a0aa29462 __syscall_cancel_arch + 0x32
0x00007f4a0aa1d75c __internal_syscall_cancel + 0x5c
0x00007f4a0aa206a8 ___pthread_cond_timedwait + 0x178
0x00007f4a0970e1e9 PlatformEvent::park_nanos(long) + 0x119
0x00007f4a09005f4c JavaThread::sleep_nanos(long) + 0xfc
0x00007f4a0918789f JVM_SleepNanos + 0x28f
0x00007f49ef961099 java.lang.Thread.sleepNanos0(long) + 0x99 (Native method)
0x00007f49e7f477b4 * java.lang.Thread.sleepNanos(long) bci:33 line:509 (Compiled frame)
0x00007f49e7f41a64 * java.lang.Thread.sleep(long) bci:25 line:540 (Compiled frame)
0x00007f49e7f4037c * Test.run() bci:3 line:6 (Compiled frame)
0x00007f49ef943328 * java.lang.Thread.runWith(java.lang.Object, java.lang.Runnable) bci:5 line:1487 (Compiled frame)
                        * java.lang.Thread.run() bci:19 line:1474 (Compiled frame)
0x00007f49ef3385fd <StubRoutines (initial stubs)>
0x00007f4a08fc247e JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 0x4ce
0x00007f4a08fc2bb3 JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 0x2d3
0x00007f4a08fc31bb JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 0xab
0x00007f4a09185590 thread_entry(JavaThread*, JavaThread*) + 0xd0
0x00007f4a09004206 JavaThread::thread_main_inner() + 0x256
0x00007f4a09c66747 Thread::call_run() + 0xb7
0x00007f4a096fccc8 thread_native_entry(Thread*) + 0x128
0x00007f4a0aa20f54 start_thread + 0x2e4

Java frame might be use the register for frame pointer (RBP in AMD64) as general purpose register, so SA cannot rely it in stack unwinding.

hs_err log has mixed stack trace as "Native frames", it would be unwinded by NativeStackPrinter in HotSpot, and it works as mixed mode. NativeStackPrinter uses frame::next_frame() to find sender frame regardless whether Java frame or C frame, and it leverages sender FP/PC to create sender frame. On the other hand, SA separates CFrame and VFrame to unwind in mixed mode jstack, so sender FP/PC would not propagate to CFrame, thus the frame located at bottom of Java frame might not be shown.

It is difficult to unify unwinder in PStack in SA, so it would be reasonable to propagate sender FP/PC to the sender of CFrame.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8370176: Mixed mode jhsdb jstack cannot unwind call stack with -Xcomp (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27885/head:pull/27885
$ git checkout pull/27885

Update a local copy of the PR:
$ git checkout pull/27885
$ git pull https://git.openjdk.org/jdk.git pull/27885/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27885

View PR using the GUI difftool:
$ git pr show -t 27885

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27885.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back ysuenaga! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@YaSuenag The following label will be automatically applied to this pull request:

• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[YaSuenag]

/issue JDK-8370176

[openjdk]

@YaSuenag The primary solved issue for a PR is set through the PR title. Since the current title does not contain an issue reference, it will now be updated.

[mlbridge]

Webrevs

• 01: Full - Incremental (a75aa29d)
• 00: Full (9d023257)