Laundry list of todos

[ivankra]

Sketches of some ideas/improvements that may (or may not) be worth implementing. Moved out of repo's TODO.md.

---

• [high] get rid of console_log, just always probe.
  • Clean up probe.py
  • Only probe for print in run.py
  • warmup_cmd= for jscript, jscript9, starlingmonkey to allow more time for setup during testing -> probe for print first with a longer timeout
• [high] harness/probe.py (and/or test262.py): support a use-case "run all engines against a single test case"
  • harness/probe.py engine... [-- script...] - probe print/console.log/262 against multiple engines; also make it usable for probing against conformance/ tests directly.
• [high] break up update.py into separate scripts build/gen-*.py
  • checkout and update repos, update loc (or move loc into dockerfiles/make dist.py's responsibility)
  • check for broken links
• [high] test262 results viewer app
  • convert json's to a single parquet file and serve it in the browser
  • navbar: engines:[v8 jsc ...] filter:[Temporal] group:[by directory/edition/feature/tag (transposed)]
  • layout: group by dir/edition/feature (rows: treeview of grouped tests, cols: engines); transposed (rows: engines: cols: groups)
  • permalinks: zoo.js.org/test262#v8,jsc&filter=Temporal&group=dir
• [medium] markdown: tags for different engines: embedded, vibe coded, toy, dialect/non-standard
• [medium] riscv64: add build/benchmark workflows on rise; alternatively: benchmark on scaleway
• agents/skills: new engine integration; github agent workflow to automatically fix weekly build
• app: compat-table subpage to serve conformance results from the repo. Original + transposed layout, toggleable loadable details, make it fast
• bench: Support more benchmarks: JetStream, web tooling, quickjs-ng
• bench.yml: merge results from previous releases so long as same rev
• bench.yml: make use of timestamp_output to compute a score for limited engines without build-in date
• build: Wasm builds (dist/wasm/*). Explore how easy is it to build C/C++ engines with WASI SDK / Emscripten and produce something runnable with minimal extra code. Make the extra glue code generalizable/reusable across engines. Try different Wasm runtimes to run them: wazero, wasmtime etc
  • kiesel's wasm build; hermes_sandbox
• build: Fully static builds with musl: generalize build system to target triples, dist/{x86_64,aarch64}-[unknown-]linux-{gnu,musl}/, but DOCKER_ARCH={amd64,arm64}. Also dist/wasm32-[unknown-]wasi/ under native DOCKER_ARCH.
  • Generic ways to pass extra CFLAGS: wrapper, clang's CCC_OVERRIDE_OPTIONS
• build: -march=x86-64-v3 - benchmark builds, add to engines which show improvements (some get broken or already include it, need to go over each engine)
• build: REV=... REPO=... make -> don't use stamp files if these are set or something
• build: COPY --from: fix properly or drop (hako, jscript, apple's container bug). Drop REV pinning logic too, replace by direct --build-arg REV=.
• build: make jit a bool field in metadata
• conformance: align filenames with test262; ES1: this/inheritance/hoisting tests
• conformance: rewrite compat-table/gen and rename tests to align with test262
• conformance/{es3-test262,es5-test262}: subsets of es3 / es5 tests, maybe as symlinks to third_party/test262/test tree? make sure test262.py can resolve them
• conformance.yml: split long-running test262 jobs into several shards and merge; maybe generalize --limit=[start:]end flag and add a small side binary (or add main to reporter.py) to join and reformat .json reports
• engines/*.md: rename "Standard" to "(Target) Specification"
• engines/iv-lv5 vs iv-lv5_clang: investigate crashes
• engines/jsc: build full variant with embedded libicu, use icu 78
  • do something about libicu-dev, COPY build/libicu-static.sh ./ && ./libicu-static.sh? jsz-icu mixin image that installs static libicu system wide?
• engines/jsc: investigate test262 NEW FAIL test/intl402/Locale/constructor-apply-options-canonicalizes-twice.js (default) (Exit code: 3) Exception: Test262Error: Expected SameValue(«"ru-Armn-SU"», «"ru-Armn-AM"») to be true
• engines/narcissus: vendor-in, add console.log() and multiple scripts command-line args
• engines: REPL wrappers: add continuation prompts, check if continuation is needed by a lightweight stack based parser;
• engines: make repl target in engine dirs
• engines: runtimes, parsers and transpilers: expand and add transpilers; test parsers on conformance test suite, report results
• engines: test node_0.x - nvm install 0.x, cjs, gjs
• harness: Support REPRL loop to minimize process creation overhead (https://github.com/googleprojectzero/fuzzilli)
• harness: slow_tests (high priority, start them as soon as discovered)
• harness: split off strict+sloppy tests among process pool workers to speed up runs dominated by timeouts -> first worker does staging, submits second scenario to the pool and continues running the first scenario, reporter or some new intermediary to join both results and cleanup
• harness: ref:$262., - make it filterable -> harness preprocessing needs to cache ref list
• harness: includes: -> transitive? make filterable
• harness/annotator.py: negative tests with multiple errors -> investigate if matching negative expectation against any of them changes results
• harness/assembler.py: test/language/module-code/ambiguous-export-bindings/namespace-unambiguous-if-export-star-as-from-and-import-star-as-and-export.js -> properly fix it. Assembler should preprocess and stage non-fixtures dependencies. Verify with -O stage_dir on full test262 that no conflicts between tests.
• harness/features.yml: stage N proposals; harness features (IsHTMLDDA+host-gc); upstream as comments in features.txt
• harness/reporter.py: verify binary sha256 in the background
• harness/run.py: detect test262 tests and offload to test262.py or warn
• harness/test262.py: implement a better UI. Check engine262 in particular. Add some keyboard controls, p=pause, q=quit, s=skip longest running test
• harness/convert262.py/compare262.py: support generic json-based formats (test262-harness libjs boa), test262-fyi; convert kiesel test262.log
• .github workflows: don't hardcode repo name ivankra