Skip to content

fix(config): reject control levels outside 1-3 on framework load (fixes #701) - #742

Open
Daniel-park10 wants to merge 2 commits into
darnitdevorg:mainfrom
Daniel-park10:fix/control-level-range-check
Open

Daniel-park10 wants to merge 2 commits into
darnitdevorg:mainfrom
Daniel-park10:fix/control-level-range-check

Conversation

@Daniel-park10

Copy link
Copy Markdown
Contributor

Summary

When a control has level = 4 or tags = { level = 4 }, darnit audit silently drops it because the level filter is (c.level or 0) <= 3. If that's the only failing control, the audit finishes with 0 results and exits 0 instead of failing. darnit validate already flags level 4, but the audit path never checked the range on load.

This adds a range check (1..3) and integer type check to ControlConfig so invalid levels (including 0, negative numbers, or levels > 3) fail closed when loading the framework or applying composite overrides, rather than getting dropped later during audit.

Fixes #701

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

If this PR modifies the darnit framework (packages/darnit/):

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Control/TOML Changes Checklist

If this PR modifies controls or TOML configuration:

  • Control metadata defined in TOML (not Python code)
  • SARIF fields (description, severity, help_url) included where appropriate
  • Ran validation to confirm TOML schema compliance

Testing

  • Tests pass locally (uv run pytest tests/ -v)
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

AI assistance

  • No AI assistance was used
  • AI assistance was used

Additional Notes

None.

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Daniel-park10. The range check fixes #701: with it, an audit fails with a clear error where it used to drop the control and exit 0. CI fails because of the second change in this PR, the type check, so I'm requesting changes for that.

Tested on main (aae02e4) with this branch merged in. Each audit used a fresh copy of pallets/flask at 49943b5a with no remote, and darnit in a throwaway HOME.

  • No change on a real repository. darnit audit -f openssf-baseline gives the same status and message for all 66 controls on main and here.
  • The #701 case, through operator configuration. I added a custom control at level = 4, with a file_exists step for a missing file:
    • On main, the audit exits 0 with 66 results, and the custom control is missing from them.
    • Here, it exits 1 with one line: Invalid operator configuration <path>: custom_controls.CUSTOM-01: Value error, Invalid control level 4: level must be 1, 2, or 3.
  • Targeted tests. tests/darnit/config/, test_composition.py, test_cli.py and test_orchestrator.py: 724 passed and 4 errors. The 4 errors are the same ones CI reports.

Blocking: the type check. _validate_level_type (framework_schema.py:687-693) does more than #701 asks for. Before, pydantic converted level = "1" to 1; now that value is rejected.

  • The only input I found that uses a string level is the fixture in tests/darnit_baseline/remediation/test_orchestrator.py:34,46, which is why CI fails. No framework TOML, doc or test TOML in the repository does.
  • With an operator custom control at level = "1": main runs it as a level 1 control (FAIL, 67 results). Here the audit exits 1 with level must be an integer, got '1' (str).

Either choice is defensible, since rejecting the string matches what framework-design already requires for tags.level. Please do one of these:

  1. Keep the type check. Change the fixture to level=1, and say in CHANGELOG and in the framework-design bullet that a non-integer top-level level (including "1" and 1.0) now fails loading.
  2. Drop _validate_level_type. The range check alone fixes #701.

Smaller: if you keep the type check, the isinstance(self.level, bool) test at framework_schema.py:732 can never be true, because bools are rejected before that point.


Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

@Daniel-park10

Copy link
Copy Markdown
Contributor Author

@mlieberman85 I've updated the PR with the required changes please take a look when you get a chance!

@Daniel-park10
Daniel-park10 force-pushed the fix/control-level-range-check branch from 21afff5 to 34d0542 Compare November 13, 2026 19:50
mlieberman85
mlieberman85 previously approved these changes Nov 16, 2026

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Daniel-park10, this is what I asked for: the type check stays, the fixture uses level=1, CHANGELOG and framework-design say that "1" and 1.0 now fail loading, and the unreachable isinstance(self.level, bool) check is gone.

Tested on main (c39bb37) with this branch merged in, on fresh copies of pallets/flask at 49943b5 with no remote and darnit in a throwaway HOME:

main here
darnit audit -f openssf-baseline 66 controls the same status and message for all 66
operator custom control at level = 4 exit 0, 66 results, the custom control missing exit 1: custom_controls.CUSTOM-01: Value error, Invalid control level 4: level must be 1, 2, or 3
operator custom control at level = "1" exit 0, 67 results, the custom control FAIL at level 1 exit 1: custom_controls.CUSTOM-01.level: Value error, level must be an integer, got '1' (str)

Full suite on the merged tree: 6002 passed, 11 skipped; ruff check and validate_sync.py pass.


Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

@Marc-cn

Marc-cn commented Nov 16, 2026

Copy link
Copy Markdown
Member

Tested merged with current main on a VM: code merges cleanly, 2504 tests + validate_sync pass. Only CHANGELOG.md conflicts now - please rebase and I will merge.

Assisted by Claude Code

darnitdevorg#701)

ControlConfig permitted control levels outside the valid maturity range
(1, 2, 3), allowing controls declared with level > 3 or tags = { level = >3 }
(or negative / zero levels) to load without error. Because darnit audit
filters controls with (c.level or 0) <= level (defaulting to level 3),
controls with level > 3 silently dropped out of audits, causing an audit
that would otherwise report failures to produce an empty result set and
exit 0.

- In ControlConfig (framework_schema.py), enforce integer type validation
  for top-level level attributes (rejecting booleans, strings, and floats)
  via a before-mode field validator, matching tags.level validation.
- In ControlConfig._sync_level_and_domain_with_tags, validate that declared
  control levels (whether from top-level attributes, tags, or composite
  overrides) fall within the valid maturity range (1, 2, 3), raising a
  validation error on 0, negative integers, or integers above 3 so audits
  fail closed on invalid framework configurations.
- Update docs/architecture/framework-design.md requirement specification.
- Add unit tests covering attribute and tag level range checks, invalid
  level type rejection, composite override rejection, and CLI audit
  fail-closed behavior.

Signed-off-by: Daniel-park10 <dev10.tennyson@gmail.com>
…arnitdevorg#701)

Enforce strict integer validation for top-level control levels in ControlConfig
to match tags.level behavior, rejecting strings, floats, and booleans. Clean up
the redundant boolean check in framework_schema and update the mock framework
fixture in test_orchestrator to declare integer levels.

Update the framework-design specification and CHANGELOG to document that non-integer
control levels fail framework loading.

Signed-off-by: Daniel-park10 <dev10.tennyson@gmail.com>
@Daniel-park10

Copy link
Copy Markdown
Contributor Author

@Marc-cn Rebase done, ready for merge!

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Daniel-park10. The rebase dismissed my approval, so here it is again. Compared with 34d0542, only the CHANGELOG context changed.

Retested on main (714b3e6) with this branch, the same way as before: fresh copies of pallets/flask at 49943b5 with no remote, and darnit in a throwaway HOME.

main here
darnit audit -f openssf-baseline 66 controls the same status and message for all 66
operator custom control at level = 4 exit 0 exit 1: custom_controls.CUSTOM-01: Value error, Invalid control level 4: level must be 1, 2, or 3
operator custom control at level = "1" exit 0 exit 1: custom_controls.CUSTOM-01.level: Value error, level must be an integer, got '1' (str)

scripts/e2e_compare.py (#760), phases 1 and 2 on wtf-frontend at 939e838: the deterministic audit and the MCP tool surface are the same. All 8 registered frameworks still load, including amber and community-spec, which landed after my last review. CI is green.


Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A control level above 3 drops out of the default audit with exit 0

3 participants