(A)synchronous cancellation guarantees

[jellevandenhooff]

After working on a javascript-based component model runtime and a port of the full go standard library to wasip3, I would like to propose making stream and future cancellation semantics guaranteed synchronous, and I would also like to raise some concerns about subtask cancellation guarantees.

For context, the Javascript runtime optionally targets non-JSPI browsers, meaning none of the component model intrinsics can block, and the go guest successfully runs in that mode. To my surprise, the rust witgen guest did not because it relied on blocking stream and task cancellations. Non-JSPI mode should become irrelevant when all browsers support it later this year thanks to interop, but I still wonder if blocking semantics should change, because I think waiting-on-implicit-cancel is confusing behavior. I also asked about these semantics on this Zulip thread.

Synchronous cancellations for streams and futures

My examples here are the rust bindings, but I think this applies to all languages that want to support cancelling reads or writes. With the component model, if you want to read or write from a stream you pass a buffer to the host, and you get a handle back to the operation, which in rust becomes a future. For safety, the bindings must invoke synchronous cancellation when cancelling (dropping) the future. Since dropping the future is something that happens implicitly, this means the component might pause at surprising times. Today, the wasmtime implementation (and wcjs) cancel quickly, but nothing guarantees it. What if the spec enforced it, and made hosts responsible for ensuring quick cancellation?

The tension, which Alex mentioned in #617, comes from the combination of readiness-based IO (rust) and completion based IO (component model). But I do not think that tension is required, in practice, because hosts today using epoll or whatever are using readiness-based IO and so they can implement synchronous cancellation at (close to) no cost, codifying the guarantee that the rust guest today already relies on.

I think, stepping back a bit, almost all languages today have bindings that fit nicely on readiness-based IO because that is what operating systems offer, and completion based API (like the bindings in eg. tokio-uring) have a different API to handle the tension between the language and underlying API semantics. I think the component model should offer an API that is close to what languages expect today. If, in the future, hosts adopt completion based IO and the cancellation overhead or semantics become a problem for wasm hosts, then the component model could introduce asynchronous-cancellable operations. But that complexity does not seem necessary today.

A counter argument is, perhaps, that completion based IO is the future, and relying in the guest on fast-synchronous-cancel is making some assumptions about host behavior, but those are accurate assumptions. I would prefer that to be guaranteed by the spec.

Concretely, I would propose dropping the async option from the stream and future cancel operations, and guaranteeing that after cancel returns the buffers can be used by the guest. I think that code-wise nothing changes in wasmtime -- it just follows the blocking code path -- but any future host bindings that would require (long) blocking cancellation must instead use some buffering or copying.

Synchronous cancellation for sub-tasks

All the same concerns from above also apply to cancelling subtasks. I think there is a similar tension between what languages model handle today and what the component model exposes. If you lend out a handle to some sub-task, you must wait for that sub-task to return before you can, say, drop that handle. But there is no way to enforce an explicit wait on the future in eg. the rust bindings, and so instead the drop must await implicitly. I think this will cause issues in, for example, any code that races two futures where one does not handle cancellation gracefully. The wasmtime wasip3 host bindings behave nicely and do support cancelling network connects or file reads, so waiting for cancellation on drop works out fine, but once again there are no guarantees.

I think there a fewer concrete patterns that exist in languages or APIs today that I can map this to. Cancellation for long-running operations in most languages requires some kind of external interrupt (an AbortController in javascript, a context.Context in go, a CancellationToken in rust+tokio), and you have to explicitly wait for those to complete. Nothing forces you wait (a goroutine will keep running unless explicitly in a waitgroup; rust and javascript require explicit await). In javascript forgetting the promise means it keep going, while in rust dropping the future cancels the task and cancellation safety is a real issue. In many programs explicitly waiting on tasks has given me issues because it is easy to forget to check a cancellation token in all paths... but at least it was explicit that I would have to wait.

None of this really applies to the operations exposed by wasip3 because they are safely cancellable, will cancel quickly on all implementations, and the language bindings do not show up in the same way because cancellation semantics are mostly around eg. connecting a socket which already comes with a timeout in most languages.

I am not quite sure what I would advocate here. I think implicit and unexpected blocking is going to cause problems, but for safety I am also convinced the caller must wait if you have these borrow semantics. The one thing I could think of is then getting rid of borrowing in some way or another. As long as you do not borrow anything from the calling stack, you can have the operation keep going in the background. You could instead just give ownership of the handle; or make a copy; or make a revocable handle that you can cancel instantaneously. None of those seem like perfect or small changes.

Perhaps revocable handles is what I would find the most clean way out, mostly because it maps to what I have used in go, where you can close a socket or file and any pending reads fail after that. But it would need some host binding representation, and it would introduce an API difference between an owned resource and a borrowed resource. The copying alternative is similar to today's wasip3 TCP read and write stream semantics that keep a socket alive even if the socket is dropped. It would change the surprising behavior on cancelling from blocking to a task still keeping going with the resource you shared.

Curious to hear what people think. Thanks for reading.

[lukewagner]

Hi @jellevandenhooff, Thanks for all the implementation work and these are great questions!

Synchronous stream cancellation is the more tempting option of the two, for the reasons you mentioned. However, it does seem like we are moving to a world where we at least want to allow future hosts to implement stream I/O in terms of io_uring-style APIs (or ye olde Windows Overlapped I/O), and this does seem to require async cancellation. IIRC, there were also some Go and .NET stdlib I/O operations (which might be used to implement stream I/O in a Go/.NET host) that also didn't guarantee synchronous cancellation (I'm guessing b/c they run on Windows and want to use Overlapped I/O).

Even more acutely, we might even start allowing wasm guest code to cause stream write cancellation to block by adding a stream.read-complete built-in, both for the original reasons mentioned here but also by necessity for lazy lowering to indicate when all lazy values passed via stream have been dropped/lowered.

So for both host and guest reasons, I think stream/future read/write cancellation need to stay asynchronous.

Subtask cancellation has a different set of reasons that are more fundamental and guest-code-motivated: I don't think it's possible to safely force a guest wasm task to stop executing without the guest's cooperation without some heavier hammer like, in the native world, killing the containing process or, in a future component world, blast zones. For reference, look for all the reasons that folks vigorously advise to never use TerminateThread(). One can attempt to define "half-way" solutions (like "the guest gets to run in response to cancellation, but it's not allowed to block"), but this pretty quickly runs into the same sorts of issues for less-obvious reasons.

[jellevandenhooff]

Thanks for your response and the reference links. I think everything you say makes a lot of sense, and in some ways I think comes to different priorities or trade-offs. I think I was (am?) arguing for closer compatibility with today's languages, while the component model today prioritizes a cleaner future. I don't know if that trade-off is the right one to make; I think friction with languages today is a pain we can already identify (eg. blocking on future drop), while the future gains are more theoretical.

For completion based IO (be it io_uring or stream.read-complete, and thanks again for that reference), I can see the benefits of those APIs. But do they need to be supported today? Even if today's cancel-read guarantees synchronous and fast cancellation, a future read-non-cancellable (or whatever) does not have to follow that contract and could give you the performance then, but it'd be opt-in.

I think the trade-off space is, roughly, implementation complexity (future knobs); performance (no completion based IO); guest friction (mandatory completion based IO). I believe that guest friction is a big deal and perhaps worth sacrificing implementation complexity for.

For subtask cancellation I think I would end up making a similar argument. Handles borrow are a powerful abstraction, and I genuinely love the power and guarantees they give, but I do not see them fit in with languages today. Most handles in languages today are essentially freely copyable, and the idea of having to give up a handle doesn't really exist. If you truly need one in eg. javascript, you create a revocable handle or a proxy or something like that. If that is the solution with semantics that make sense in languages today, that is what I would argue for in a design today.

[lukewagner]

I think you're right that we have to care a lot about minimizing friction in guest language bindings. However, in the case of cancellation, I think we're not sacrificing guest language experience because guest code can call {stream,future}.cancel-{read,write} synchronously (by not setting the async immediate). With this, the caller simply blocks until the cancellation is done (which is what the host would've had to do anyways if we made synchrony the only option, so not really a regression if the host was going to block in the first place).

There is the fact that performing a synchronous cancellation counts as "blocking" which means you can't do it when called from a sync export. However most code will from an async export call (e.g., currently all WASI entry points in 0.3), so this shouldn't be a problem for ordinary guest code in practice. For the more advanced (virtualization) cases, see this presentation to the WASI SG made when async was made a trapping effect (from a hint), but again, these shouldn't be introducing friction in the common case.

[lukewagner]

I forgot to mention: another import case where the host would need cancellation to be async is, in the hopefully-not-to-distant-future, the browser. In particular, if whatwg/streams/#757 is added (to allow the browser to read directly to/from a wasm linear memory without the intermediate ArrayBuffer and copy), the cancel() method is async.

[jellevandenhooff]

With this, the caller simply blocks until the cancellation is done (which is what the host would've had to do anyways if we made synchrony the only option, so not really a regression if the host was going to block in the first place).

I agree that today there is no friction and the bindings are pleasant, but that is true only because cancellation is fast. The spec doesn't guarantee that, and it could change in the future, breaking an assumption for today's bindings; I think that is a real regression.

When I imagine "making synchrony the only option" I also imagine an explicit guarantee that the cancel would be fast, in the worst case requiring host work with a copy or something like that. With epoll bindings it is free, and if it becomes a cost in the future, the component model could at that point add new intrinsics for asynchronously-cancellable IO that map to guest bindings that look io_uring.

if whatwg/streams#757 is added (to allow the browser to read directly to/from a wasm linear memory without the intermediate ArrayBuffer and copy), the cancel() method is async

Thanks for that link, that's a fun API. I don't think that is incompatible with synchronous cancellation on the component model side. If that async stream cancel is "fast", then the host could (behind the scenes) await the cancel and still expose a synchronous API to the guest (but I agree it would need JSPI). If it is slow, then I would want an explicit io_uring style API in the guest forcing to handle those waiting points.

The concrete scenario I worry about is a rust guest awaiting two futures, and returning as soon as one finishes. If the wait-on-cancellation of the second future blocks that would make the guest block unexpectedly. Today this code is fine because all cancellation (stream and host wasip3 bindings) is fast, but that is not guaranteed, and I think it should be.

[lukewagner]

I'm not sure what you're proposing really changes the underlying calculus: if we need async cancellation due to the host not being able to cancel "quickly", and if we don't want the guest to synchronously block on this non-quick cancellation, then we'll need the guest code to cooperate in some way or another which may introduce friction.

As you said in your earlier post, there's an inherent tension between all these competing concerns and I think we have a pretty reasonable compromise now that allows guest bindings to be simple while preserving hosts the flexibility they need (these async-cancelled system APIs exist today, even if our first-draft wasip3 implementations aren't using them yet) and, in the worst case, we just lose some concurrency in a way that can be incrementally improved over time.

[jellevandenhooff]

I think what I am stuck on is that guest code like

let future = stream.write(&buf).await;
select! {
    _ = future => {},
    _ = timeout(Duration::from_secs(1)) => {},
}

behaves like

let future = stream.write(&buf).await;
select! {
    _ = future => {},
    _ = timeout(Duration::from_secs(1)) => {},
}
future.cancel().await;

without indicating that in the code. Today these snippets are the same because the cancel is fast, but at some point they might not. If that were opt-in, like explicitly deciding to use tokio-uring, I'd be much less worried.

[jellevandenhooff]

Is it true that first doing a 0-sized write and then doing a non 0-sized write avoids this problem?

[lukewagner]

Is it true that first doing a 0-sized write and then doing a non 0-sized write avoids this problem?

Yes, with the caveat that after a zero-length write succeeds, there is no guarantee that the next non-zero-length write will succeed without blocking. However, this (less common) occurrence can be worked around by writer-side buffering as discussed in this section. Coincidentally #617 just-recently pointed out a limitation with the currently-described approach in that section (which uses stream.cancel-write) which suggests adding some stream.try-{read,write} built-ins (previously considered just as an optimization).

[jellevandenhooff]

Yes, with the caveat that after a zero-length write succeeds, there is no guarantee that the next non-zero-length write will succeed without blocking.

Ah, so tricky! I think the completion-based vs readiness-based framing is very helpful in understanding the options here for me. It seems like 0-sized writes (or reads) combined with try-write is essentially readiness-based IO, which you (almost?) need to implement libc-style / tokio-style IO without risk of blocking. Which then becomes (almost?) isomorphic to instantaneous-cancel like I asked for here, which also gives you readiness-based IO, but without the option of doing io_uring behind the scenes. Or, well, you could do io_uring with instant cancel if you block internally in the cancel.

What do you imagine for try-read behind the scenes? If a guest can support both read and try-read efficiently, should it prefer read? In other words, is read the future and try-read the past? It seems that with read and try-read the stream API combines completion and readiness IO, that's cool, but perhaps inefficient? Happy to take this elsewhere, also.

[lukewagner]

What do you imagine for try-read behind the scenes? If a guest can support both read and try-read efficiently, should it prefer read?

My impression is that try-read/try-write should only be used for libc O_NONBLOCK (and any other situation where it's forced by the API design) and read/write would be used by default otherwise. This helps to avoid the "streaming between two components and both sides are doing non-blocking I/O and so now we have to do an intermediate buffering copy" case mentioned in the above link.

[syntactically]

these async-cancelled system APIs exist today, even if our first-draft wasip3 implementations aren't using them yet

I'll chime in here to say that we have near-ish term plans in hyperlight-wasm for p3 support, and our p3 support will be based on a very fundamentally completion-based primitive (where host calls end up flowing over virtio queues---very io_uring-like). Synchronous cancel would likely be difficult for us to implement with good performance, so something like the try-{read,write} APIs that at some level let the guest detect a completion-based host and provide its own buffering for readiness-based APIs would be nicer for us.

[jellevandenhooff]

I'll chime in here to say that we have near-ish term plans in hyperlight-wasm for p3 support

That's exciting news!

Synchronous cancel would likely be difficult for us to implement with good performance, so something like the try-{read,write} APIs that at some level let the guest detect a completion-based host and provide its own buffering for readiness-based APIs would be nicer for us.

How are you thinking about implementing this? This makes me also think of function coloring, but instead it is IO paradigm coloring, and the desire to hide it. Do you want to look at imports? Fall back on first call to try-read? Both the 0-sized read and try-read triggering a buffer are tricky behaviors that a guest (might) have to reason or know about.

[syntactically]

How are you thinking about implementing this? This makes me also think of function coloring, but instead it is IO paradigm coloring, and the desire to hide it. Do you want to look at imports? Fall back on first call to try-read? Both the 0-sized read and try-read triggering a buffer are tricky behaviors that a guest (might) have to reason or know about.

I think the idea I had right now at least is that going off the behaviours that Luke mentioned above w/r/t stream readiness, we could just have the wasm host (/ vm guest) implement try-read/try-write to return wouldblock always, and have 0-byte reads return success always, which combined should signal to guest code (e.g. language bindings) that it's in a runtime that doesn't implement readiness-based I/O and it should fall back to copies---would that make sense to you? We could also detect that particular pattern and try to provide the copy layer in the wasm host, but I worry about the possibility of accidentally triggering it and putting extra copies in for guests that don't strictly need them.

This whole approach also maybe only makes sense for stream read/write futures---are there any cases where there are other futures (either provided by the host or by other guests) where you anticipate that the (in)ability to cancel them synchronously when dropped will cause problems?