CodeFRAME Testing Guide

Sprint 1 manual testing checklist; for current test organization and counts see README.md and SPRINTS.md. For the canonical E2E workflows that Sprint 10 tests should exercise, see the E2E section in PRD.md.

Sprint 1 Manual Test Checklist

Setup Requirements

Environment Setup

Clone repository: git clone https://github.com/frankbria/codeframe.git
Navigate to project: cd codeframe
Create virtual environment: python -m venv venv
Activate virtual environment: source venv/bin/activate
Install Python dependencies: pip install -e .
Install Node dependencies: cd web-ui && npm install && cd ..

Configuration

Create .env file in project root
Add ANTHROPIC_API_KEY: ANTHROPIC_API_KEY=your-key-here
Verify .env is in .gitignore

Service Startup

Start Status Server: python -m codeframe.ui.server
- Expected: Server starts on port 8000
- Expected: "Status Server running on http://localhost:8000" message
Start Web UI (new terminal): cd web-ui && npm run dev
- Expected: UI starts on port 3000
- Expected: "Local: http://localhost:3000" message

Test 1: Project Creation (cf-8, cf-11)

1.1 Basic Project Initialization

Run: codeframe init test-project
Verify: Project directory created at ./test-project
Verify: .codeframe/ directory exists in test-project/
Verify: Database file created: .codeframe/state.db
Verify: Success message displayed

1.2 Database Verification

Open database: sqlite3 test-project/.codeframe/state.db
Run: .tables
Verify: All tables exist: projects, tasks, agents, blockers, memory, context_items, checkpoints, changelog
Run: SELECT * FROM projects;
Verify: Project entry exists with name='test-project', status='init'
Exit sqlite: .exit

1.3 Dashboard Integration

Open browser: http://localhost:3000
Verify: "test-project" appears in projects list
Verify: Status shows "init"
Verify: Created timestamp is recent

Test 2: Database CRUD Operations (cf-8)

2.1 Project Operations

Create second project: codeframe init project2
Verify: Both projects appear in dashboard
Verify: Both have separate .codeframe/state.db files
Update project status (via API or direct DB): UPDATE projects SET status='planning' WHERE name='test-project'
Refresh dashboard
Verify: Status updated to "planning"

2.2 Agent Creation

Use Python REPL or script:

from codeframe.persistence.database import Database
db = Database("test-project/.codeframe/state.db")
db.initialize()
agent_id = db.create_agent("lead-1", "lead", "claude", "directive")
agent = db.get_agent("lead-1")
print(agent)

Verify: Agent created successfully
Verify: Agent details correct (type=lead, provider=claude, maturity=directive)

2.3 Memory Storage

Continue in Python REPL:

project = db.get_project(1)
memory_id = db.create_memory(
    project_id=project['id'],
    category='pattern',
    key='auth_pattern',
    value='JWT with refresh tokens'
)
memories = db.get_project_memories(project['id'])
print(memories)

Verify: Memory entry created
Verify: Can retrieve memory by project_id

Test 3: Anthropic Provider Integration (cf-9)

3.1 Provider Initialization

Test provider creation:

from codeframe.agents.providers.anthropic_provider import AnthropicProvider
provider = AnthropicProvider(api_key="test-key")

Verify: Provider initializes without error
Verify: Model defaults to "claude-3-5-sonnet-20241022"

3.2 Message Sending (Mock/Test)

Run unit tests: ANTHROPIC_API_KEY="test-key" pytest tests/test_anthropic_provider.py -v
Verify: All 17 tests pass
Verify: Message formatting tests pass
Verify: Error handling tests pass

Test 4: Lead Agent Lifecycle (cf-9, cf-10)

4.1 Lead Agent Creation

Test Lead Agent initialization:

from codeframe.agents.lead_agent import LeadAgent
from codeframe.persistence.database import Database

db = Database("test-project/.codeframe/state.db")
db.initialize()

lead = LeadAgent(
    agent_id="lead-test-1",
    provider_name="anthropic",
    api_key="test-key",
    database=db
)

Verify: Lead Agent creates successfully
Verify: Agent entry created in database
Verify: Default maturity level is "directive"

4.2 Agent Status Management

Update agent status:

lead.update_status("working")
agent = db.get_agent("lead-test-1")
print(agent['status'])

Verify: Status updates to "working"
Verify: Database reflects change

Test 5: Project Creation API (cf-11)

5.1 API Endpoint Testing

Send POST request to create project:

curl -X POST http://localhost:8000/api/projects \
  -H "Content-Type: application/json" \
  -d '{"name": "api-test-project", "description": "Test via API"}'

Verify: 200 OK response
Verify: Response includes project_id and status
Verify: Project directory created
Verify: Database initialized

5.2 Project Listing API

Send GET request:
```
curl http://localhost:8000/api/projects
```
Verify: Returns JSON array
Verify: Contains all created projects
Verify: Each project has id, name, status, created_at

5.3 WebSocket Real-Time Updates

Open browser developer console on dashboard
Create new project via CLI: codeframe init websocket-test
Verify: Dashboard updates automatically (no refresh needed)
Verify: Console shows WebSocket message received
Check WebSocket connection: Network tab → WS → Messages
Verify: project_created event appears

Test 6: Agent Lifecycle Management (cf-10)

6.1 Agent State Transitions

Test agent lifecycle:

from codeframe.agents.lead_agent import LeadAgent
from codeframe.persistence.database import Database
from codeframe.core.models import AgentMaturity

db = Database("test-project/.codeframe/state.db")
db.initialize()

lead = LeadAgent("lead-lifecycle", "anthropic", "test-key", db)

# Test state transitions
lead.update_status("idle")
lead.update_status("working")
lead.update_status("blocked")
lead.update_status("completed")

# Test maturity progression
lead.update_maturity(AgentMaturity.D2)
lead.update_maturity(AgentMaturity.D3)

Verify: All status transitions succeed
Verify: Maturity level updates correctly
Verify: Database reflects all changes

6.2 Agent Error Handling

Test invalid transitions:

lead.update_status("invalid_status")  # Should handle gracefully

Verify: Error handled without crash
Verify: Agent remains in valid state

Test 7: End-to-End Integration

7.1 Complete Project Workflow

Initialize new project: codeframe init e2e-test
Verify: Project appears in dashboard immediately
Verify: Database created with all tables
Create Lead Agent programmatically
Store memory entry
Update project status to "planning"
Verify: All changes visible in dashboard
Verify: WebSocket events fire for all updates

7.2 Multi-Project Handling

Create 3 projects simultaneously:

codeframe init project-a &
codeframe init project-b &
codeframe init project-c &
wait

Verify: All 3 projects created successfully
Verify: Each has independent database
Verify: Dashboard shows all 3 projects
Verify: No database conflicts

Test 8: Error Conditions & Edge Cases

8.1 Database Errors

Test duplicate project: codeframe init test-project (already exists)
Verify: Error message displayed
Verify: No corruption of existing project

8.2 API Errors

Send invalid JSON to API:

curl -X POST http://localhost:8000/api/projects \
  -H "Content-Type: application/json" \
  -d 'invalid json'

Verify: 400 Bad Request response
Verify: Helpful error message

8.3 Missing Configuration

Remove ANTHROPIC_API_KEY from .env
Try to create Lead Agent
Verify: Clear error about missing API key
Restore API key

Test 9: Performance Verification

9.1 Response Times

Measure API response time:

time curl http://localhost:8000/api/projects

Verify: Response time < 500ms (p95 requirement)

9.2 Database Performance

Create 100 memory entries:

for i in range(100):
    db.create_memory(1, 'test', f'key_{i}', f'value_{i}')

Query all memories: db.get_project_memories(1)
Verify: Query completes in < 1 second

Test 10: Automated Test Suite Verification

10.1 Run All Tests

Run complete test suite:
```
ANTHROPIC_API_KEY="test-key" pytest -v
```
Verify: 111 tests pass (100% pass rate)
Verify: No warnings or errors

10.2 Test Coverage

Run with coverage:

ANTHROPIC_API_KEY="test-key" pytest --cov=codeframe --cov-report=html

Verify: Overall coverage > 90%
Verify: Database module > 92%
Open htmlcov/index.html to review

Definition of Done Verification

Sprint 1 Completion Criteria

✅ All 9 tasks complete (cf-8 through cf-13)
✅ 111 automated tests passing at 100%
✅ Zero mock data in production code
✅ Database operations tested at >80% coverage (actual: 92%)
✅ API response time <500ms (p95)
✅ WebSocket reconnect works automatically
✅ Can run codeframe init and see project in dashboard
✅ Lead Agent can be created with valid API key
✅ No critical bugs blocking sprint review

Code Quality Checks

✅ All code follows Python PEP 8 style guide
✅ No hardcoded API keys or secrets in code
✅ All database operations use parameterized queries
✅ Error handling implemented for all external calls
✅ TDD followed for all features (RED-GREEN-REFACTOR)

Documentation Complete

✅ TESTING.md created with manual test checklist
✅ README.md updated with setup instructions
✅ AGILE_SPRINTS.md reflects actual progress
✅ All API endpoints documented

Test Results Template

## Sprint 1 Manual Test Execution Results
**Date**: YYYY-MM-DD
**Tester**: [Name]
**Environment**: [OS, Python version, Node version]

### Setup
- [ ] All setup steps completed successfully
- Issues found: [None | List issues]

### Test 1: Project Creation
- [ ] Passed | [ ] Failed
- Issues found: [None | List issues]
- Notes:

### Test 2: Database CRUD
- [ ] Passed | [ ] Failed
- Issues found: [None | List issues]
- Notes:

[Continue for all tests...]

### Overall Assessment
- Total Tests Run: X
- Tests Passed: Y
- Tests Failed: Z
- Critical Issues: [List]
- Sprint 1 Ready for Demo: [ ] Yes | [ ] No

Troubleshooting Guide

Common Issues

Issue: Database file not found

Solution: Ensure you're in the correct project directory, run codeframe init first

Issue: WebSocket connection fails

Solution: Check Status Server is running on port 8000, check browser console for errors

Issue: API key error

Solution: Verify .env file exists with valid ANTHROPIC_API_KEY

Issue: Import errors

Solution: Activate virtual environment, reinstall with pip install -e .

Issue: Dashboard doesn't update

Solution: Check WebSocket connection in browser DevTools, restart Status Server

Next Steps (Sprint 2)

After completing Sprint 1 manual testing:

Document any critical bugs and fix before demo
Prepare sprint demo showing working features
Review Sprint 2 tasks: Socratic Discovery phase
Plan Sprint 2 implementation starting with CLI foundation

Sprint 1 Status: COMPLETE ✅ Total Test Cases: 111 automated + 10 manual test scenarios Pass Rate: 100% Ready for Production: Foundation components ready for Sprint 2 integration

FilesExpand file tree

TESTING.md

Latest commit

History