Add zero-shot default and multi-dataset support for AppWorld #38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Change --dataset to --datasets (comma-separated, default: train,dev) - Add --default-few-shot flag (default: zero-shot, no examples) - Update prompts.py to conditionally include few-shot examples - Update output_dir and experiment_name fixtures for multi-dataset - Preserve --start-from functionality from main branch - Update all documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Instead of trying to remove the "examples" intro line via string replacement (which was fragile), remove it from the template and only add it when few-shot mode is enabled. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Remove --experiment-dir from conftest
- Add --appworld-experiment-name for custom experiment names
- Auto-infer as {model}/{datasets} when not specified
- Fix zero-shot prompt (add examples intro only in few-shot mode)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
- AppWorld output_dir now respects --output-dir from main conftest
- Auto-infers as results/{model}/{datasets}/outputs/ when default
- Add --appworld-experiment-name for custom AppWorld experiment names
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
- Add get_datasets_dir() helper to DRY up datasets parsing - Fix validation mode to use --datasets instead of old --dataset - Fix failure_report_dir to derive from output_dir.parent - Import parse_datasets and get_datasets_dir at top of file 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- output_dir now uses results/{experiment_name}/outputs/
- Added get_experiment_name() helper for use in fixtures and pytest_generate_tests
- Validation mode now uses experiment_name for consistent directory lookup
- Both output_dir and experiment_name respect --appworld-experiment-name
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🤖 Generated with Claude Code