feat(guest): replace musl with picolibc by andreiltd · Pull Request #831 · hyperlight-dev/hyperlight

andreiltd · 2025-08-28T11:48:24Z

This patch removes custom musl implementation used by the guest and replace it with picolibc v1.8.11.

Note: The picolibc submodule uses sparse checkout to exclude GPL/AGPL-licensed test and script files that are not needed for building. Only BSD/MIT/permissive-licensed source files are included.

jprendes

I love this PR!

src/hyperlight_guest_bin/src/host_bridge.rs

src/hyperlight_guest_bin/Cargo.toml

src/hyperlight_guest_bin/src/host_bridge.rs

jsturtevant · 2025-08-28T21:07:02Z

will help with #282

src/hyperlight_host/tests/sandbox_host_tests.rs

src/hyperlight_guest_bin/.cargo/config.toml

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

src/hyperlight_guest_bin/build.rs

jsturtevant · 2026-03-03T01:24:30Z

src/hyperlight_guest_bin/c/clock.c

did we write this file? is there attribution that is needed?

If this is really necessary, I think @jprendes just wrote a Rust implementation for this in hyperlight-js. maybe we could use it?

Yes, those are ours. We cannot use the implementation from hyperlight-js directly, it serves a bit different purpose. That being said I rewrote picolibc stubs to rust. The tradeoff is that some of the types and defines that you normally get from libc headers have to be redifined in Rust with compatible ABI. I think that is OK -- they have to be backwards compatible and I don't see them change anytime soon.

jsturtevant · 2026-03-03T01:24:44Z

src/hyperlight_guest_bin/c/clock.c

+    (void)__tz;
+
+    _current_time(current_time);
+    tv->tv_sec = current_time[0];


do we need to check for null here?

It's now checked in Rust implementation.

jsturtevant · 2026-03-03T01:24:53Z

src/hyperlight_guest_bin/c/clock.c

+        case CLOCK_REALTIME:
+        case CLOCK_MONOTONIC:
+            _current_time(current_time);
+            tp->tv_sec = current_time[0];


It's now checked in Rust implementation.

jsturtevant · 2026-03-03T01:29:47Z

src/hyperlight_guest_bin/src/host_bridge.rs

+        Some(vec![ParameterValue::ULong(count as u64)]),
+        ReturnType::VecBytes,
+    ) {
+        Ok(bytes) => {
+            let n = bytes.len();
+            unsafe {
+                core::ptr::copy_nonoverlapping(bytes.as_ptr(), buf as *mut u8, n);


should not rely on the host provided length. nonoverlapping could write past the buffer lenghth. We are in the guest here so it would just cause corruption but might be tough to track down I think

jsturtevant · 2026-03-03T01:35:47Z

src/hyperlight_guest_bin/src/host_bridge.rs

+        return -1;
+    }
+
+    let slice = unsafe { core::slice::from_raw_parts(buf as *const u8, count) };


do we need a null check on the buf here? https://doc.rust-lang.org/core/slice/fn.from_raw_parts.html#safety

data must be non-null and aligned even for zero-length slices

It's now checked in Rust implementation.

jsturtevant · 2026-03-03T01:44:43Z

NOTICE.txt

-obstacle to adoption, that text has been removed.
+Note: The picolibc submodule uses sparse checkout to exclude
+GPL/AGPL-licensed test and script files that are not needed for
+building. Only BSD/MIT/permissive-licensed source files are included.


Just wanted to surface this. We should include it in the PR description I think

Where is the sparsity pattern that makes this actually happen?

That would be here:
https://github.com/hyperlight-dev/hyperlight/pull/831/changes#diff-99cd76bae1f874496398d422f5fcc07ece8599f9f26cc81bb1cb154d0cf0003fR338

dblnz

Great work, Tomasz!! This is one hell of a diff and I like it 😆

dblnz · 2026-03-04T08:16:28Z

src/hyperlight_guest_bin/c/clock.c

If this is really necessary, I think @jprendes just wrote a Rust implementation for this in hyperlight-js. maybe we could use it?

src/hyperlight_guest_bin/src/lib.rs

src/hyperlight_guest_bin/build.rs

src/hyperlight_guest_bin/build_files.rs

Justfile

jprendes · 2026-02-25T11:19:37Z

src/hyperlight_guest_bin/c/clock.c

+
+extern void _current_time(uint64_t *ts);
+
+int gettimeofday(struct timeval *__restrict tv, void *__restrict __tz) {


Thought: If we were running bindgen on picolib headers, then we could easily implement this one in rust instead of C. IIUC, the main issue is the definition of timeval, clockid_t, and timespec in Rust, that we don't want to get wrong.

This file is no longer in the diff.

If we were running bindgen on picolib headers, then we could easily implement this one in rust

That would be correct yes, but I would rather much have a single c file defining the interface or redefine types in Rust with correct layout. It's hard to justify running the whole bindgen machinery for few u64.

jprendes · 2026-02-25T15:16:42Z

src/hyperlight_guest_bin/src/host_bridge.rs

+        ReturnType::VecBytes,
+    ) {
+        Ok(bytes) => {
+            let n = bytes.len();


maybe check that bytes.lenth() <= count?

Yeah, I think the timing is a bit unfortunate, could you check if your comments are resolved in the newest diff?

jprendes · 2026-02-25T15:56:36Z

src/hyperlight_guest_bin/src/host_bridge.rs

+    ) {
+        Ok(bytes) => {
+            let n = bytes.len();
+            unsafe {


Check that n <= count

This is fixed

jprendes · 2026-02-25T15:58:07Z

src/hyperlight_guest_bin/src/host_bridge.rs

+    }
+
+    let slice = unsafe { core::slice::from_raw_parts(buf as *const u8, count) };
+    let s = core::str::from_utf8(slice).unwrap_or("<invalid utf8>");


I think I would like to insist on the from_utf8_lossy, as mentioned we are allocationg anyway when doing s.to_string()

This is fixed

syntactically · 2026-03-05T13:41:28Z

Is the licensing approval finally done for this, making it ready for review/merge?

syntactically · 2026-03-05T13:52:22Z

src/tests/c_guests/c_simpleguest/main.c


 #define GUEST_SCRATCH_SIZE (0x40000) // default scratch size
 #define MAX_BUFFER_SIZE (1024)

+#define printf_f(fmt, ...)                          \


Does it make sense to see if we can either build picolibc in a configuration without stdio buffers, or to automatically do a flush on the guest-function-call-complete path?

I feel like the buffered-I/O-by-default regime will be very confusing to users---where, for example, if you are doing the "call a function and then restore the sandbox" pattern might often not see any output at all.

Yeah, I think we can disable posix console feature: https://github.com/picolibc/picolibc/blob/main/doc/build.md#stdio-options

Fuzzy on the details but IIRC that will change to char by char sending to host which I thought was insane.

Yeah, that makes sense... char-by-char is definitely unfortunate. Flush-on-guest-function-return would be nice if we could coalesce the flush exit and the function-return exit, but I'm not sure how easy that is?

syntactically · 2026-03-05T13:56:32Z

src/hyperlight_guest_bin/src/host_bridge.rs

+
+#[unsafe(no_mangle)]
+pub extern "C" fn _current_time(ts: *mut u64) -> c_int {
+    let bytes = call_host_function::<Vec<u8>>("CurrentTime", Some(vec![]), ReturnType::VecBytes)


Similarly, I think I might like to see this fail at least by default? As with my comments on #1173, I'm against exposing a clock, especially a wall clock, to the guest by default.

This is very common dependency, for example spidermonkey crashes immediately if we don't provide some time interface. Do we really want to error or at least return an unix epoch?

Interesting... I didn't realise so many things were using wall-clock time. Do you know what we were doing for spidermonkey before? I don't think we did actually have the real time?

I wouldn't be against having it always be the epoch personally (at least by default), but I'll let others chime in here as well.

at some point we had a fallback where we took some epoch (2000?) and added 1s or 1ms for each call

syntactically · 2026-03-05T13:56:46Z

src/hyperlight_guest_bin/src/host_bridge.rs

+static CURRENT_TIME: AtomicU64 = AtomicU64::new(0);
+
+#[unsafe(no_mangle)]
+pub extern "C" fn read(fd: c_int, buf: *mut c_void, count: usize) -> isize {


Where are the fds referenced in these functions allocated? I don't see an open, so are these always either 0/1/2 for stdin/out/err? I think it might make sense to make the behaviour a bit more limited (maybe reads from fd0 always return eof; writes to fd1 are HostPrint, and writes to fd2 are debug_print, or something like that?)

That's almost what you have here, right? except that:

fd2 also goes to HostPrint instead of debug_print (I didn't know debug_print was a thing!).

fd0 first tries HostRead, if that's not present returns -1 and sets errno = EIO (instead of EOF, which would be returning 0 and no errno)

any other fd is EBADF

So, do you suggest these changes?

fd2 to use debug_print instead of HostPrint

fd0 to return 0 (and untouched errno) in case of no HostRead

I don't care much about the fd2---it was just the first thing that sprang to mind.

Unless we have a compelling use case, I think I would prefer for read from fd0 to unconditionally be eof, without probing for a host function which seems a bit surprising and magical. If we ever have C code that needs to be able to read() from stdin, we could add some hyperlight_guest_bin API to let a little Rust wrapper code install its own read function.

I'm happy to go with that logic. @jprendes any objections?

you can call fflush(NULL) and that flushes all open FDs.
Can we leave buffering and add that on VM exit (or similar) in an upcoming PR?

also I think our current _putchar impl can go away, right?

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

dblnz

Great work, Tomasz!
I have nothing else to add.

src/hyperlight_guest_bin/build.rs

src/hyperlight_guest_bin/build_files.rs

dblnz · 2026-03-10T08:11:10Z

docs/picolibc.md

This looks great! Thanks for adding this description!

andreiltd requested review from danbugs, dblnz, devigned, jprendes, ludfjig, marosset, simongdavies and syntactically as code owners August 28, 2025 11:48

andreiltd marked this pull request as draft August 28, 2025 11:48

andreiltd force-pushed the libc-takeover branch from 617d834 to 3f8ffcf Compare August 28, 2025 12:01

andreiltd added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Aug 28, 2025

andreiltd force-pushed the libc-takeover branch 6 times, most recently from eaf1d54 to b18a374 Compare August 28, 2025 13:44

jprendes reviewed Aug 28, 2025

View reviewed changes

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/Cargo.toml Show resolved Hide resolved

jprendes reviewed Aug 28, 2025

View reviewed changes

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

src/hyperlight_guest_bin/src/host_bridge.rs Outdated Show resolved Hide resolved

jsturtevant mentioned this pull request Aug 28, 2025

[License Exception Request] [Hyperlight] [Older BSD like licenses] cncf/foundation#1117

Open

12 tasks

andreiltd force-pushed the libc-takeover branch 5 times, most recently from 499c59d to 2ae0a3a Compare August 29, 2025 18:42

jprendes reviewed Sep 4, 2025

View reviewed changes

src/hyperlight_host/tests/sandbox_host_tests.rs Outdated Show resolved Hide resolved

andreiltd force-pushed the libc-takeover branch from 2ae0a3a to d3826af Compare September 4, 2025 11:29

andreiltd mentioned this pull request Sep 5, 2025

Use mimalloc as a guest allocator #867

Draft

andreiltd force-pushed the libc-takeover branch from d2043ac to 2bd1ff4 Compare September 9, 2025 09:31

andreiltd force-pushed the libc-takeover branch from 65634a7 to 980cb20 Compare February 25, 2026 10:31

jprendes reviewed Feb 25, 2026

View reviewed changes

src/hyperlight_guest_bin/.cargo/config.toml Outdated Show resolved Hide resolved

andreiltd force-pushed the libc-takeover branch 2 times, most recently from c24ce45 to d9fb9e5 Compare February 25, 2026 11:33

feat(guest): replace musl with picolibc

16db912

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the libc-takeover branch 2 times, most recently from e4caabe to 7b99307 Compare February 25, 2026 17:59

bump picolibc to 1.8.11

fb1549f

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the libc-takeover branch from 7b99307 to fb1549f Compare February 25, 2026 18:12

andreiltd marked this pull request as ready for review February 26, 2026 20:11

jsturtevant reviewed Mar 3, 2026

View reviewed changes

src/hyperlight_guest_bin/build.rs Outdated Show resolved Hide resolved

jsturtevant reviewed Mar 3, 2026

View reviewed changes

dblnz reviewed Mar 4, 2026

View reviewed changes

jprendes reviewed Mar 5, 2026

View reviewed changes

syntactically reviewed Mar 5, 2026

View reviewed changes

fix: address code review comments

bf6be7d

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the libc-takeover branch from 0de7e24 to bf6be7d Compare March 5, 2026 14:17

fix: fmt with different rust version

2fbc721

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the libc-takeover branch from cacc739 to 2fbc721 Compare March 5, 2026 14:36

andreiltd added 2 commits March 5, 2026 15:50

fix: apply clippy fixes

94df68a

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: clarify usage of no-red-zone flag

3680b58

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

dblnz approved these changes Mar 10, 2026

View reviewed changes


		extern void _current_time(uint64_t *ts);

		int gettimeofday(struct timeval __restrict tv, void __restrict __tz) {

Conversation

andreiltd commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Aug 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dblnz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syntactically commented Mar 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

andreiltd commented Aug 28, 2025 •

edited

Loading

andreiltd Mar 5, 2026 •

edited

Loading