feature request: baml-cli test

[sxlijin]

To run all tests defined in .baml files

[prrao87]

I'd very much love to see this happen, thanks!

[sxlijin]

Implementation is in progress! See #1458 for the first pass (plan is to land it next week) and here are our design notes:

baml-cli test run --include "MyFunction::" --include "AnyMatch" --exclude "::LongTest"

# syntax: <part>::<part>, parts may only be "*\w\d_"

baml-cli test # this will default to "list" b/c it costs $$$ to run tests
baml-cli test list
baml-cli test run
baml-cli test --output-formats [pretty | github | junit=path/to/file]

Product requirements:

• allow users to enforce that baml tests pass in CI
  • users may only want to run a subset of tests in CI
    • filter based on file & name (in the future, maybe some kind of “test tag” attribute)
    • --include
    • --exclude
    • tests for a given function
    • regex
  • --verbose - TBD
  • report timing stats - TBD
  • tests may be flakey and need re-running - TBD
  • for github actions: we can use [output groups](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines)
  • for non-github aka generic integration (TBD) junit xml is a must
    • cf gitlab unit test reports
• debug loop: what are users going to do when tests fail?
  • test failures need to show logs
  • re-run a specific test
    • can go into the vscode playground for this
    • still need to be able to run the same command as in CI for reproducibility's sake though
• happy-ish path
  • show # of tests that passed, timing info, # of tests skipped - take cargo test as inspiration (maybe pytest/mocha too)
  • test execution will be concurrent
    • do we need concurrency limits?
      • need to be able to set concurrency down to 1 (claude basic tier), default concurrency should be... ???
    • definitely need timeouts too

[denizkenan]

I would be also interested in this.