add quality verification for chain system#42
add quality verification for chain system#42arbuzz202 wants to merge 1 commit intoDeevsDeevs:mainfrom
Conversation
…tains quality info for the new sessions by spawning empty context agent and interrogating it whether the provided context is sufficient for continuing hte project
| 1. Detects the file is a chain link | ||
| 2. Runs `chain-system/scripts/chain-verify.sh` — spawns an **isolated CLI process** (`claude --print --bare`) with zero shared context | ||
| 3. The isolated process reads ONLY the chain link content and answers 10 verification questions | ||
| 4. Scores each answer 0-100 |
There was a problem hiding this comment.
явно нужны критерии по которым оценивать ответ, не просто "70-89 = missing some detail, 50-69 = significant gaps, 0-49 = cannot answer"
всегда лучше накидывать баллы, а не снимать
| **If score < 85**: Hook exits 2, **write is blocked**. The gaps are fed back to you as feedback. You must: | ||
| - Read the gap analysis in the hook feedback | ||
| - Re-examine the conversation for the missing information | ||
| - Rewrite the chain link addressing the specific gaps |
| - Read the gap analysis in the hook feedback | ||
| - Re-examine the conversation for the missing information | ||
| - Rewrite the chain link addressing the specific gaps | ||
| - Save again (which re-triggers the hook automatically) |
There was a problem hiding this comment.
which re-triggers the hook automatically
obvious?
| - Rewrite the chain link addressing the specific gaps | ||
| - Save again (which re-triggers the hook automatically) | ||
|
|
||
| This loop continues until the chain link passes or you've addressed all available information from the conversation. |
| # Use a simple, deterministic question set derived from the 9 required sections. | ||
| # These probe whether each section is present and substantive. | ||
| cat > "$QUESTIONS_FILE" << 'EOF' | ||
| 1. What is the user's primary request? Describe the exact goal, not a vague summary. |
There was a problem hiding this comment.
the exact goal, not a vague summary
examples?
| 6. Are there any unresolved issues or blockers? If so, what solutions were attempted and why did they fail? | ||
| 7. What tasks remain to be done? Are they specific enough to act on without guessing? | ||
| 8. What was actively in progress when this chain link was saved? Include file names, line numbers, and code context. | ||
| 9. What is the exact next step? Is it a single, unambiguous action that a new session can execute immediately? |
There was a problem hiding this comment.
single, unambiguous action
harmful for chaining without planning (as before)
that a new session can execute immediately
sounds like to narrow for your exact use-cases @arbuzz202
| 7. What tasks remain to be done? Are they specific enough to act on without guessing? | ||
| 8. What was actively in progress when this chain link was saved? Include file names, line numbers, and code context. | ||
| 9. What is the exact next step? Is it a single, unambiguous action that a new session can execute immediately? | ||
| 10. Could you resume this work right now with ONLY the information in this chain link, without asking any clarifying questions? If not, what would you need to ask? |
| exit 0 | ||
| fi | ||
|
|
||
| # Extract the aggregate score from the verification output |
There was a problem hiding this comment.
comments might be deleted, in fact. Code is clear.
| CHAIN_CONTENT="$(cat "$CHAIN_LINK_FILE")" | ||
| QUESTIONS="$(cat "$QUESTIONS_FILE")" | ||
|
|
||
| SYSTEM_PROMPT="You are a developer resuming work from a chain link. You have NO other context besides what is provided in this prompt. You must answer each question using ONLY the chain link content given. If you cannot answer a question or the answer is incomplete/ambiguous, say INSUFFICIENT and explain what is missing." |
There was a problem hiding this comment.
You have NO other context besides what is provided in this prompt
Makes the use-case when we do /plan after chaining impossible: agents can scavenge the repo additionally.
|
|
||
| ## Instructions | ||
|
|
||
| Answer each question using ONLY the chain link content above. For each question respond with: |
There was a problem hiding this comment.
but the new agents HAVE the new context like wtf?
Maybe something like "using ONLY the context mined from repo and the chain link content above", but this will make the chaining over-complicated.
|
Needs hard re-work: current instructions are vague while being too restrictive for some use-cases. |
2 participants
After creating a chain file spawns an empty context session, loads chain file into it, interrogates the agent for an understanding of a context, then assesses his answers and based off that decides whether to add more info to the chain file in specific places