Skip to content

docs(nanobench): finalize architecture, governance, and resolve #60 - #62

Open
RONAK-AI647 wants to merge 1 commit into
Nano-Collective:mainfrom
RONAK-AI647:RONAK-AI647/-Final-Address-for-final-version
Open

docs(nanobench): finalize architecture, governance, and resolve #60#62
RONAK-AI647 wants to merge 1 commit into
Nano-Collective:mainfrom
RONAK-AI647:RONAK-AI647/-Final-Address-for-final-version

Conversation

@RONAK-AI647

Copy link
Copy Markdown
Contributor

Related Issues

Resolves #60 (NanoBench: public review closed, request for a final version).
Closes #34, #35, #36, #37.

Description

This PR finalizes the NanoBench whitepaper following the conclusion of the public review window. It systematically addresses the final checklist provided by maintainers, migrating the document from a "proposed architecture" to a "settled protocol."

What this PR does

Specifically, this applies the following requested updates section by section:

  • §1 & §6 (Orchestration): Removed hardcoded Python and (TS) labels. Explicitly recorded that the evaluation core's orchestration language is the maintainer's choice, provided the --json schema validation contract is strictly enforced.
  • §7, §11, & §12 (Contamination & Extraction Gates):
    • Removed unbacked claims regarding "Jaccard similarity checks" and "algorithmic code indexes," moving them to the v2 roadmap.
    • Grounded the v1 contamination defense entirely in highly auditable post-cutoff merge dates and immutable pinned SHAs.
    • Added the missing Flakiness Gate (requiring 3 consecutive deterministic test runs at base_commit) to the extraction methodology.
  • §9.2 (Taxonomy Audit Rules): Documented the Level 4 signal audit protocol. This includes the 90% agreement fallback rule, the mandatory blinding protocol for future reviewers, and the explicit limitation acknowledging the v1 maintainer's unblinded status.
  • §14 (Governance & Curation Decisions): Completely rewrote the former "Open Questions" section. It now formally records the dataset ownership pipeline, the bifurcated validation gates, and the task retirement/dispute protocols.

Type of Change

  • Documentation update

@RONAK-AI647

Copy link
Copy Markdown
Contributor Author

@will-lamerton Take a look , and let me know how it is !!
The PR Nano-Collective/nanocoder#821 dealing with the 'usage' block is also opened !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant