yield await

[plajjan]

The idea with yield await is to be able to turn programs performing something callback based, like I/O, to have a more natural form in source code. It should be easy to reason about these programs!

Today the only pattern we have are state machines. State machines are always ready to react to events, but it can be hard to capture the "intent" of a state machine, like procedural programs execute top down and have an "intent", after X we want to do Y etc. In state machine form, this intent gets hidden and spread out across various event callbacks. With yield await, we hope we can once again gather this intent in a single place.

Something like:

actor main(env):
    yield await c = http.Client("google.com")
    yield await r = c.get("/")
    print(r.status)

After each statement we await the result and meanwhile we yield control of the local actor, thus "yield await". This allows us to express a procedural program with an intent but for callbacks.

This is not yet a concrete proposal, we need to work out the finer details!

[plajjan]

I think it is natural that all things, primarily I/O actors, that use callbacks, should be designed in such a way that each method call has a primary action and a single callback that gets called after the primary action has been performed.

Thus far, we have gravitated towards using multiple callbacks, like in the process module we have a on_stdout, on_stderr, on_exit & on_error. There is a glaring gap here as well, we aren't informed about when a process has started.

I think all I/O modules should be shaped in a new way. Each method or actor instantiation has a primary action, like starting a process, and there should thus be a primary callback, like on_start(process, ?error) which is notified when the process has started or if it errors out. The user will need to check if error != None and act accordingly. Similarly for network based things, we can meld together the on_connect & on_error callbacks since the primary action is "connect" when instantiating a client actor and the response is either positive on_connect or negative on_error(). That is better expressed as on_connect(conn: c, err: ?str)

To fit with yield await, I suppose this on_connect() callback needs to go either first or last among arguments. How's this?

# http module
actor Client(
    on_connect: action(Client, ?str) -> None,
    cap: net.TCPConnectCap,
    address: str,
    scheme: str="https",
    port: ?int=None,
    tls_verify: bool=True,
    log_handler: ?logging.Handler):

actor GoogleSearch(on_result(res: list[str], err: ?str) -> None, query_string: str):
    tcpc_cap = net.TCPConnectCap(net.TCPCap(net.NetCap(env.cap)))

    go()    

    def go():
        yield await c, err = http.Client(tcpc_cap, "google.com")
        if err != None:
            on_result(None, "Got an error when connecting:" + err)
            return

        yield await r, r_err = c.get("/q=%s" % query_string)
        if r_err != None:
            on_result(None, "Got an error when searching:" + r_err)
            return
        res = parse_result(r)
        on_response(res)

    def parse_result(res: str) -> list[str]:
        """Parse search result and chop up"""
        ...


actor main(env):
    yield await res, err = GoogleSearch(tcpc_cap, "bananas")
    if err != None:
        print("Some error happened:", err)
        env.exit(1)
    print("Result:", res)
    env.exit(0)

Can / should we prefer exceptions rather than some err value? This is starting to look like the err-lang (Go).

It seems to me that CPS transformation of the go() function should be possible just based on this, the yield await callback is automatically computed and sent as first argument to the http.Client (init / connect) and .get() method. yield await c, err unpacks two values, which map to the two arguments passed in the callback. Obviously the go method is CPS transformed into multiple continuations and so the local state needs to be passed from one continuation to the next, but isn't this doable with like a closure of the local env?

This means http.Client() can still be called with classic callback style in case we actually want to write more of an explicit state machine.

@nordlander you have mentioned suspend & resume functions. I am not sure how that would look here. I think it is really valuable to keep the http.Client() code, which has underpinnings written in C, just the way it is, except that the callbacks must match our expectation, like cb arg is first and number of args is correct etc. I think it is crucial to not fragment code into red / blue functions (classic sync vs async) but with our version being "classic callbacks" vs "yield await" style.