feat(runtime): schedule object fields concurrently

[vito]

I kicked around a few ideas for concurrency, and I think this one strikes the sweet spot for Dang's focus.

We really just need a way to schedule work in parallel and group the results and/or fail-fast when any of them fail. In Go you might model this with goroutines, wait groups, context cancellation, and storing results in a map using locks to synchronize writes (or sync.Map but you get the point). But that's all low-level plumbing: tons of boilerplate to write, and tons of independent language features that would dramatically increase Dang's scope.

This PR instead enhances a single pre-existing feature: object literals - quick reminder example of those:

dang> {{foo: {{bar: 1}}, fizz: foo.bar}}
=> module {fizz: Int!, foo: {bar: Int!}!}
dang> toJSON({{foo: {{bar: 1}}, fizz: foo.bar}})
=> {"fizz":1,"foo":{"bar":1}}

Currently object literals instantiate a new scope and evaluate + assigns slots within it, serially.

With this PR, they are instead evaluated in DAG order. Independent slots are evaluted in parallel, and inter-dependent slots are evaluated in dependency order.

This seems pretty elegant to me, because it means {{}} literals that make API queries are now just as parallel as GraphQL multi-field selection syntax - i.e. {{bar: foo.bar, baz: foo.baz}} is just as parallel as foo.{{bar, baz}} (except it sends two requests instead of one).

Take this snippet, from one of Dang's tests:

pub complex_query = {{
  server: serverInfo.{{ version, platform }},
  users: users.{{ name, emails }},
  posts: posts.{{ title, author.{{ name }} }},
  titles: postTitles
}}

Previously this would send 3 queries one after another. Now it sends them all at once.

With the DAG evaluation order, this also means you can hypothetically define entire pipelines as single objects, and let Dang figure out how to evaluate everything optimally:

{{
  images: {{
    linux: build("linux")
    windows: build("windows")
    darwin: build("darwin")
  }}
  test: test
  integration: integration(images.linux)
}}
# evaluates:
# 1. build("linux"), build("windows"), build("darwin"), test
# 2. integration(images.linux)

Failing fast (or not)

Evaluation fails as soon as anything fails, i.e. it fails fast. I think this is the best default, since the alternative would mean returning a mixed report of successful results and failures, and seems like it could be represented by composing with something like try to have fields typed as something like Either foo Error.

[fire-ant]

I've been hacking on CI with Dagger/Dang for its pure speed and neatness. This would be great to see in the next few dagger releases as i work on a monorepo and need to rebuild multiple deps from time to time :)

[vito]

@fire-ant Good to know! I've just been sitting on this because no one actually asked for it yet, and I didn't want to theorycraft too hard. :)

Do you happen to have a code snippet demonstrating how you'd use it, or are you mostly guessing that it would help in some way?

[fire-ant]

today i do this for some python build pipeline:

pub all(Token: Secret, cachebust: String! = "repeat"): String! {
  lint                                                   # ruff check    (container)
  fmt                                                    # ruff format   (container)
  let testOut = test(Token: Token, cachebust: cachebust)  # pytest (container)
  let pkgOut  = package                                  # uv build      (container)
  comp.name + " all stages:\n  lint: ok\n  fmt: ok\n  test: " + testOut + "\n  package: " + pkgOut
}

where I think the above would allow:

pub all(Token: Secret, cachebust: String! = "repeat"): String! {
  let r = {{
    lint:    lint,
    fmt:     fmt,
    test:    test(Token: Token, cachebust: cachebust),
    package: package,
  }}
  comp.name + " all stages:\n  lint: ok\n  fmt: ok\n  test: " + r.test + "\n  package: " + r.package
}

Which would be a better expression, especially when I have monorepo of all the things where I'm effectively replacing githubs path filtering with a thin dang orchestration module detecting where paths have diverged and running many builds in parallel. I previously did this in go-sdk with goroutine's and that worked great, but everything is so fast with dang Im trying to move all my modules to dang. A lot of our fan-out is currently list.map { … } where I'd still need to do somethings for maps => object literals, but i think there is a fix even closer to being merged which will resolve this for me

[cloudflare-workers-and-pages]

Deploying dang with    Cloudflare Pages

Latest commit:	b891c17
Status:	✅  Deploy successful!
Preview URL:	https://fa8b3bbc.dang-3kk.pages.dev
Branch Preview URL:	https://concurrency.dang-3kk.pages.dev

View logs