Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
683 lines (567 loc) · 36 KB

ARCHITECTURE.md

File metadata and controls

683 lines (567 loc) · 36 KB
Raw

SchemeAI Architecture

SchemeAI is a Next.js (App Router) application that helps people in India find government schemes, scholarships and welfare programmes they may qualify for. A user fills in a short profile. A deterministic, rule-based eligibility engine checks that profile against a curated reference catalogue of schemes. Every scheme record links back to an official government source and carries transcribed guideline excerpts as evidence.

This document covers how the system fits together. It is based on the current source tree.

1. System overview

The application has five main parts:

  • A single-page client (src/app/page.tsx renders AppShell). It switches views from client state instead of using URL routes.
  • JSON API route handlers under src/app/api.
  • Domain modules in src/lib: the eligibility engine, scheme catalogue, auth, live source checks, the TinyFish client and serializers.
  • A PostgreSQL database accessed through Prisma (prisma/schema.prisma).
  • External services:
    • official government websites (live HTTP checks)
    • the optional TinyFish browser-agent API (live extraction from official pages)
    • an optional LLM reached through z-ai-web-dev-sdk for the assistant
flowchart LR
    subgraph Browser
        UI["AppShell + views"]
        Store["Zustand store (localStorage)"]
    end

    subgraph NextServer["Next.js server"]
        API["API route handlers /api/*"]
        subgraph Lib["src/lib"]
            Elig["eligibility.ts"]
            Cat["schemes catalogue"]
            Auth["auth.ts"]
            Live["live-check.ts"]
            TF["tinyfish.ts"]
            Ser["serialize.ts"]
            DBC["db.ts (Prisma client)"]
        end
        Asst["/api/assistant"]
    end

    PG[("PostgreSQL")]
    Gov["Official .gov.in sites"]
    TinyFish["TinyFish agent API (optional)"]
    LLM["LLM via z-ai-web-dev-sdk (optional)"]

    UI --> Store
    UI -->|"fetch JSON"| API
    API --> Elig
    API --> Auth
    API --> Live
    API --> TF
    API --> Ser
    API --> DBC
    Asst --> DBC
    Asst --> LLM
    Cat -->|"seed via /api/setup or scripts"| DBC
    DBC --> PG
    Live -->|"HEAD / GET"| Gov
    Live --> TF
    TF -->|"POST /v1/automation/run"| TinyFish
    TinyFish -->|"reads public pages"| Gov
Loading

Design principles visible in the code

  • The engine is rule-based and explains every result. eligibility.ts never calls an LLM. Every verdict traces back to a stored rule and a profile field.
  • Missing data is reported, not guessed. If a profile field is absent, the rule's status is UNKNOWN, not a guess. BPL/SECC status (isBpl) is never collected from the profile and always resolves to UNKNOWN.
  • Verification status reflects real checks only. Schemes start as REFERENCE. They move to LIVE_CHECKED only after a real HTTP check of their source domain succeeds or a TinyFish refresh of their official page returns data.
  • Live data is stored as evidence, not merged. TinyFish extractions go to FetchJob.resultJson and LIVE_FETCH evidence rows with the run ID; curated scheme fields are never overwritten.
  • Integrations degrade cleanly. Without TINYFISH_API_KEY the integration reports NOT_CONFIGURED and only direct checks run. When the LLM call fails, the assistant falls back to a deterministic, retrieval-based answer.

2. Layered breakdown

2.1 Frontend: single-page app shell

src/app/page.tsx renders one component, AppShell. Navigation runs on a view value in the Zustand store (src/lib/store.ts), not on Next.js routes. ViewRouter in app-shell.tsx switches on view.name.

View (view.name) Component Purpose API calls
home home.tsx Landing page, category tiles, recent schemes, live source verification section (the run button is shown to admins; other visitors see a note pointing to the scheme-page Verify button) GET /api/schemes, POST /api/discovery (admin)
discover wizard.tsx 4-step profile wizard (basics, education, household, additional) none (writes profile to store)
results results.tsx Ranked match results for the stored profile POST /api/schemes/match
browse browse.tsx Search, filter and paginate the catalogue GET /api/schemes
scheme scheme-detail.tsx Scheme detail, rule-by-rule evaluation, evidence, document checklist, Verify button (source check plus TinyFish refresh, showing the extracted details) GET /api/schemes/:id, GET /api/sources, POST /api/sources/verify (with schemeId)
compare compare.tsx Side-by-side comparison of up to 3 schemes POST /api/schemes/compare
dashboard dashboard.tsx Saved and recent schemes, recommendations, application tracker GET /api/schemes/:id, POST /api/schemes/match
admin admin.tsx System metrics, source registry, per-source checks, Run TinyFish discovery, recent jobs. The nav link is shown only to admins GET /api/admin/stats, POST /api/sources/verify, POST /api/discovery
auth auth-view.tsx Sign in and register /api/auth/*

Components used across views:

  • assistant.tsx: the floating "Ask SchemeAI" widget (POST /api/assistant).
  • bits.tsx: status, verification and category badges; SchemeCard; loading and empty states.
  • i18n.tsx: an English/Hindi dictionary and I18nProvider.

Client state

The schemeai-store key in localStorage holds the persisted Zustand state:

  • profile, hasProfile
  • savedIds, recentIds, compareIds (max 3)
  • tracker and checklists
  • user and lastMatches

The app works for guests. Saved schemes and tracker status are kept locally. For signed-in users they are also sent to /api/saved and /api/tracker. On mount, AppShell calls GET /api/auth/me and merges the server's saved IDs and tracker into local state.

The hero panel on the home page (DEMO_PROFILE, DEMO_MATCHES in home.tsx) contains static example content. It is not computed.

2.2 API layer

All handlers are in src/app/api. They return JSON and validate request bodies with Zod where a body is accepted.

Route Method(s) Auth Description
/api/schemes GET none List with case-insensitive q, category, state, level (national or state), sort (relevance, name, category, recent), page, perPage (max 48)
/api/schemes/[id] GET none Detail by id or slug. An optional ?profile=<url-encoded JSON> adds a match evaluation
/api/schemes/match POST none Evaluates every scheme against a Zod-validated profile. Returns results sorted by relevanceScore plus a status summary
/api/schemes/compare POST none 1 to 3 scheme IDs, with optional profile evaluation
/api/assistant POST none Retrieval-grounded Q&A (see section 7.3)
/api/sources GET none Government source registry with last real check state and TinyFish status
/api/sources/verify POST none with schemeId, otherwise admin Real HTTP check of one source; with schemeId, also a TinyFish refresh of that scheme (6 h cache). maxDuration 180 s
/api/discovery POST, GET POST admin, GET none POST checks up to 5 sources with a direct check and, when configured, a TinyFish extraction in parallel (2 TinyFish runs at a time). maxDuration 300 s. GET returns the last 30 fetch jobs
/api/auth/register POST none Creates a user (password at least 8 characters), starts a session, writes an AuditLog row
/api/auth/login POST none Verifies the password and starts a session
/api/auth/logout POST session Deletes the session row and cookie
/api/auth/me GET session Current user, saved scheme IDs, tracker and stored profile
/api/saved GET, POST, DELETE session Saved schemes for the signed-in user
/api/tracker GET, PUT session Self-reported application status per scheme
/api/admin/stats GET admin Counts, source list, recent jobs, TinyFish status. 401 without a session, 403 for non-admins
/api/health GET none DB connectivity plus scheme and source counts. Returns 503 when degraded
/api/setup POST SETUP_SECRET One-time seed. Needs ?setup=1 and the x-setup-secret header

2.3 Domain layer (src/lib)

Module Responsibility
eligibility.ts Rule evaluation, OR-group logic, scheme-level status, relevance scoring, bilingual explanations, STATUS_META display mapping
schemes/ The static reference catalogue (33 schemes, 22 sources) and the shared seed types, education levels, states and categories
auth.ts scrypt password hashing, DB-backed session tokens (SHA-256 hashed at rest), httpOnly cookie schemeai_session with a 14-day TTL, getSessionUser, requireAdmin
live-check.ts performLiveCheck (HEAD then GET, 12 s timeout, <title> extraction), verifySourceAndRecord (persists a FetchJob, updates the GovernmentSource row, promotes linked schemes) and refreshSchemeWithTinyFish (SCHEME_REFRESH job, LIVE_FETCH evidence, 6 h reuse of successful runs)
tinyfish.ts TinyFish config and status (NOT_CONFIGURED or READY); tinyFishRun (POST /v1/automation/run, X-API-Key, 90 s agent limit, 120 s client timeout, COMPLETED / BLOCKED handling); goal prompts tinyFishExtractSourcePage and tinyFishExtractSchemePage; compactTinyFishResult for storage
serialize.ts Converts Prisma rows to summary or detail JSON. Parses documentsJson and processJson
types.ts Client-safe types that mirror the serializer output
store.ts Client Zustand store (see 2.1)
i18n.tsx English/Hindi dictionary and React context
db.ts Prisma client singleton. Uses POSTGRES_PRISMA_URL if set, otherwise DATABASE_URL

2.4 Data layer

Prisma uses the postgresql provider (prisma/schema.prisma). JSON payloads are stored as String columns and parsed in application code:

  • rule value
  • documentsJson, processJson
  • resultJson, checklistJson

Enumerations are also plain strings, with allowed values listed in schema comments. The data model is in section 4.

2.5 External integrations

Integration Where Behaviour
Official government sites live-check.ts Real read-only HEAD and GET requests to each registered GovernmentSource.domain. No login, form or CAPTCHA automation
TinyFish tinyfish.ts, live-check.ts, /api/discovery, /api/sources/verify Used only when TINYFISH_API_KEY is set. Sends POST {TINYFISH_API_URL}/v1/automation/run (default base https://agent.tinyfish.ai) with { url, goal, browser_profile: "lite", agent_config }. Results are stored in FetchJob.resultJson and, for scheme refreshes, as LIVE_FETCH evidence. Details in TINYFISH.md
LLM /api/assistant Loads z-ai-web-dev-sdk dynamically and calls ZAI.create() then chat.completions.create. If anything fails or the answer is empty, the response is built deterministically (mode: "DETERMINISTIC")

3. Directory structure

.
├── prisma/
│   └── schema.prisma            # PostgreSQL data model (users, schemes, rules, sources, jobs)
├── scripts/
│   ├── seed.ts                  # Idempotent seed of sources + schemes + rules + evidence (npm run db:seed)
│   ├── create-admin.ts          # Creates the demo admin user (npm run db:seed:admin)
│   └── copy-standalone.js       # Copies static/public into .next/standalone (only for standalone output builds)
├── src/
│   ├── app/
│   │   ├── layout.tsx           # Root layout, fonts, metadata, Toaster
│   │   ├── page.tsx             # Renders <AppShell/>, the whole UI
│   │   ├── globals.css          # Tailwind v4 styles and theme tokens
│   │   └── api/
│   │       ├── schemes/route.ts          # List / search / filter / paginate
│   │       ├── schemes/[id]/route.ts     # Detail (+ optional profile match)
│   │       ├── schemes/match/route.ts    # Run eligibility engine over all schemes
│   │       ├── schemes/compare/route.ts  # Compare up to 3 schemes
│   │       ├── assistant/route.ts        # Retrieval-grounded Q&A with LLM fallback
│   │       ├── sources/route.ts          # Source registry
│   │       ├── sources/verify/route.ts   # Live check of one source
│   │       ├── discovery/route.ts        # Batch live discovery + job history
│   │       ├── auth/{register,login,logout,me}/route.ts  # Session auth
│   │       ├── saved/route.ts            # Saved schemes (signed-in)
│   │       ├── tracker/route.ts          # Application tracker (signed-in)
│   │       ├── admin/stats/route.ts      # Admin metrics
│   │       ├── health/route.ts           # Health check
│   │       └── setup/route.ts            # Secret-protected one-time seed
│   ├── components/
│   │   ├── schemeai/            # App views (see section 2.1)
│   │   └── ui/                  # shadcn/ui primitives
│   ├── hooks/                   # use-mobile, use-toast
│   └── lib/
│       ├── eligibility.ts       # Eligibility engine
│       ├── auth.ts              # Password hashing + sessions
│       ├── db.ts                # Prisma client singleton
│       ├── live-check.ts        # Real HTTP source verification + TinyFish scheme refresh
│       ├── tinyfish.ts          # TinyFish API client and goal prompts
│       ├── serialize.ts         # Prisma row to API JSON
│       ├── store.ts             # Zustand client store
│       ├── i18n.tsx             # English/Hindi dictionary
│       ├── types.ts             # Client-side API types
│       └── schemes/
│           ├── index.ts         # ALL_SEED_SCHEMES, ALL_SEED_SOURCES
│           ├── types.ts         # SeedScheme, SeedRule, SeedSource, reference lists
│           └── data/
│               ├── scholarships.ts  # 9 scholarship/education schemes
│               ├── welfare.ts       # 16 welfare/health/housing/insurance schemes
│               ├── livelihood.ts    # 8 agriculture/skills/entrepreneurship schemes
│               └── sources.ts       # 22 official government sources
├── tests/
│   └── eligibility.test.ts      # Assertion script for the engine (npm test)
├── next.config.ts               # Next config (ignoreBuildErrors, no strict mode)
├── vercel.json                  # Vercel build config, region bom1
├── .env.example                 # Environment variable template
├── download/                    # README screenshots
└── DEPLOY-VERCEL.md             # Step-by-step Vercel + Neon deployment guide

4. Data model

This diagram is generated from prisma/schema.prisma.

erDiagram
    User ||--o{ Session : has
    User ||--o| UserProfile : has
    User ||--o{ SavedScheme : saves
    User ||--o{ ApplicationTracker : tracks
    GovernmentSource |o--o{ Scheme : "publishes"
    GovernmentSource |o--o{ FetchJob : "checked by"
    Scheme ||--o{ SchemeEligibilityRule : "has rules"
    Scheme ||--o{ SourceEvidence : "has evidence"
    Scheme ||--o{ SavedScheme : "saved as"
    Scheme ||--o{ ApplicationTracker : "tracked as"

    User {
        String id PK
        String email UK
        String name
        String passwordHash
        String role
        DateTime createdAt
        DateTime updatedAt
    }
    Session {
        String id PK
        String token UK
        String userId FK
        DateTime expiresAt
        DateTime createdAt
    }
    UserProfile {
        String id PK
        String userId FK
        Int age
        String state
        String district
        String gender
        String educationLevel
        String employmentStatus
        String occupation
        Int annualIncome
        String residence
        String socialCategory
        Boolean isFarmer
        Boolean hasDisability
        Int disabilityPercent
        Boolean isMinority
        Boolean isWidow
        Boolean consentGiven
        DateTime updatedAt
    }
    GovernmentSource {
        String id PK
        String name UK
        String domain UK
        String jurisdiction
        String department
        String accessMethod
        String accessStatus
        Boolean automationSupported
        String reliability
        String notes
        DateTime lastCheckedAt
        String lastCheckStatus
        String lastCheckTitle
        DateTime lastSuccessAt
        DateTime lastFailureAt
    }
    Scheme {
        String id PK
        String slug UK
        String name
        String nameHi
        String shortDescription
        String shortDescriptionHi
        String description
        String category
        String ministry
        String department
        String jurisdiction
        String state
        String beneficiaryType
        String benefits
        String benefitAmount
        String applicationUrl
        String infoUrl
        String applicationWindow
        String applicationDeadline
        String documentsJson
        String processJson
        String keywords
        String verificationStatus
        String extractionMethod
        String confidence
        DateTime dataCompiledAt
        DateTime lastLiveCheckedAt
        String sourceId FK
    }
    SchemeEligibilityRule {
        String id PK
        String schemeId FK
        String attribute
        String operator
        String value
        Boolean required
        String label
        String labelHi
        String ruleGroup
    }
    SourceEvidence {
        String id PK
        String schemeId FK
        String sourceUrl
        String excerpt
        String evidenceType
        String note
        DateTime retrievedAt
    }
    FetchJob {
        String id PK
        String type
        String mode
        String status
        String sourceId FK
        String schemeId
        String targetUrl
        String resultJson
        String error
        DateTime startedAt
        DateTime completedAt
    }
    SavedScheme {
        String id PK
        String userId FK
        String schemeId FK
        DateTime createdAt
    }
    ApplicationTracker {
        String id PK
        String userId FK
        String schemeId FK
        String status
        String notes
        String checklistJson
        DateTime updatedAt
    }
    AuditLog {
        String id PK
        String actor
        String action
        String entity
        String detail
        DateTime createdAt
    }
Loading

Notes:

  • SavedScheme and ApplicationTracker are unique on (userId, schemeId). User-owned rows cascade on user delete. Rule, evidence, saved and tracker rows cascade on scheme delete.
  • FetchJob.schemeId is a plain column with no Prisma relation. It is set by SCHEME_REFRESH jobs.
  • AuditLog is standalone. Today it is written only on user registration.
  • Status values stored as strings:
    • Scheme.verificationStatus: REFERENCE, LIVE_CHECKED, STALE
    • FetchJob.type: SOURCE_VERIFY, LIVE_DISCOVERY, SCHEME_REFRESH; FetchJob.mode: DIRECT_FETCH, TINYFISH
    • FetchJob.status: QUEUED, RUNNING, SUCCEEDED, FAILED, SKIPPED_NOT_CONFIGURED
    • SourceEvidence.evidenceType: GUIDELINE_EXCERPT (seeded), LIVE_FETCH (TinyFish scheme refresh, at most one row per scheme)
    • ApplicationTracker.status: INTERESTED, PREPARING_DOCUMENTS, APPLIED_EXTERNALLY, UNDER_REVIEW, COMPLETED
  • Several values defined in the schema are never written by the code:
    • STALE
    • SKIPPED_NOT_CONFIGURED and QUEUED
    • TINYFISH_LIVE and DIRECT_FETCH extraction methods (every scheme keeps SEED_REFERENCE; a TinyFish refresh adds evidence but does not change this field)

5. Eligibility engine

Source: src/lib/eligibility.ts. Callers: /api/schemes/match, /api/schemes/[id] and /api/schemes/compare.

5.1 Rules

Each SchemeEligibilityRule has these fields:

  • attribute: a profile field, or isBpl
  • operator
  • value: JSON-encoded
  • required
  • bilingual labels
  • optional ruleGroup

evaluateRule returns MET, FAILED or UNKNOWN, together with English and Hindi explanations, the user's value and a requirement text.

Operator Semantics
MIN / MAX Numeric comparison user >= value or user <= value. Non-numeric values give UNKNOWN
EQ String equality
IN / NOT_IN Membership in a JSON array
TRUE / FALSE Boolean flag must be true or must be false
other UNKNOWN ("unsupported operator")

Two special cases:

  • A missing profile value always gives UNKNOWN.
  • isBpl always gives UNKNOWN, because BPL/SECC listing must be checked against official records.

5.2 Groups and scheme status

groupEvaluations buckets rules by ruleGroup. A rule without a group is its own bucket.

  • A group's status is MET if any member is MET, FAILED if every member is FAILED, and UNKNOWN otherwise.
  • Groups are ANDed together. Only buckets that contain at least one required rule decide the scheme status. Optional rules only add reasons or "to verify" items.

The scheme status is decided as follows:

Condition Status
No rules at all UNABLE_TO_DETERMINE (score 20, band EXPLORE)
Any required bucket FAILED NOT_ELIGIBLE
All required buckets MET LIKELY_ELIGIBLE
Some required bucket UNKNOWN POSSIBLY_ELIGIBLE
Otherwise UNABLE_TO_DETERMINE

5.3 Relevance score

The relevance score is used for ranking. The UI does not present it as a probability of eligibility. Points are added as follows, and the total is capped at 100:

Component Points
Status base Likely 45, Possibly 25, Unable 8, Not eligible 4
Jurisdiction +20 if the scheme state equals the profile state; +10 for national schemes (state null)
Category affinity +15 when one of these holds: Agriculture with farmer; Scholarship with student; Skill Development with unemployed or age 45 or under; Entrepreneurship with a business, vendor or artisan occupation; Girl Child & Women with female; Social Security with age 60 or over
Info completeness round(evaluable rules / total rules × 15)

The score maps to a band: 68 or more is HIGH_RELEVANCE, 42 or more is POTENTIAL_MATCH, and anything lower is EXPLORE.

5.4 Evidence and explanation output

SchemeMatchResult contains:

  • reasons: explanations from MET buckets, plus a state-residency note when it applies
  • toVerify: explanations from UNKNOWN buckets
  • the full per-rule rules list, from which the UI shows FAILED items
  • infoCompleteness

All text comes in both English and Hindi. Separately from the engine, each scheme's SourceEvidence rows (guideline excerpts transcribed at compile time) and sourceMeta (last real check) are returned by GET /api/schemes/:id and shown next to the evaluation.

flowchart TD
    A["Profile + scheme rules"] --> B{"Any rules?"}
    B -- No --> U0["UNABLE_TO_DETERMINE, score 20"]
    B -- Yes --> C["evaluateRule for each rule"]
    C --> C1{"attribute is isBpl or value missing?"}
    C1 -- Yes --> R1["UNKNOWN"]
    C1 -- No --> C2["Apply operator MIN, MAX, EQ, IN, NOT_IN, TRUE, FALSE"]
    C2 --> R2["MET or FAILED"]
    R1 --> G["Group by ruleGroup: OR within group"]
    R2 --> G
    G --> H["Keep buckets with a required rule"]
    H --> I{"Any bucket FAILED?"}
    I -- Yes --> NE["NOT_ELIGIBLE"]
    I -- No --> J{"All buckets MET?"}
    J -- Yes --> LE["LIKELY_ELIGIBLE"]
    J -- No --> PE["POSSIBLY_ELIGIBLE"]
    NE --> S["Score: base + jurisdiction + affinity + completeness"]
    LE --> S
    PE --> S
    S --> BAND["Band: HIGH_RELEVANCE, POTENTIAL_MATCH or EXPLORE"]
    BAND --> OUT["reasons, toVerify, rules, infoCompleteness"]
Loading

5.5 Matching request flow

sequenceDiagram
    participant W as Wizard view
    participant S as Zustand store
    participant R as Results view
    participant API as Match API route
    participant DB as PostgreSQL
    W->>S: setProfile(draft)
    S->>R: go results
    R->>API: POST profile
    API->>API: Zod validate profile
    API->>DB: findMany schemes with rules and source
    API->>API: evaluateScheme for each scheme
    API-->>R: results sorted by score plus summary
Loading

The tests for the engine are in tests/eligibility.test.ts, a plain assertion script run with npm test (tsx tests/eligibility.test.ts).

6. Scheme catalogue and verification lifecycle

The reference catalogue lives in TypeScript under src/lib/schemes/data. It is the single source of truth for seeding. Each SeedScheme includes:

  • descriptive fields
  • documents and process steps
  • rules
  • evidence excerpts
  • sourceName, which links it to a SeedSource

The catalogue has 33 schemes. 31 are national and 2 are state schemes (Madhya Pradesh and Uttar Pradesh).

Seeding is done by scripts/seed.ts or /api/setup. Both:

  • upsert sources by domain
  • upsert schemes by slug (new schemes use the slug as their id)
  • replace each scheme's rules and GUIDELINE_EXCERPT evidence (LIVE_FETCH evidence is kept)

Source check state (accessStatus, lastChecked*) is never touched by seeding.

stateDiagram-v2
    [*] --> REFERENCE : seeded
    REFERENCE --> LIVE_CHECKED : source HTTP check succeeds
    REFERENCE --> LIVE_CHECKED : TinyFish scheme refresh succeeds
    LIVE_CHECKED --> LIVE_CHECKED : later checks or refreshes
Loading

A failed check sets the source's accessStatus to RESTRICTED but does not downgrade schemes. Nothing in the code moves a scheme to STALE.

7. Cross-cutting flows

7.1 Authentication

The flow, implemented in auth.ts:

  1. Register or login creates a random 32-byte token.
  2. The SHA-256 hash of the token is stored in Session.token.
  3. The raw token goes into the schemeai_session cookie (httpOnly, sameSite=lax, secure in production, 14 days).
  4. getSessionUser looks up the hashed token and deletes expired sessions.

Passwords are stored as salt:scrypt(64). They are compared with timingSafeEqual. next-auth is listed as a dependency but is not used.

7.2 Live source verification and discovery

  • Single source (POST /api/sources/verify, verifySourceAndRecord), admin only unless a schemeId is sent:
    1. creates a FetchJob (SOURCE_VERIFY, DIRECT_FETCH)
    2. runs performLiveCheck
    3. updates the job and the source
    4. on success, sets linked schemes to LIVE_CHECKED
  • Scheme refresh (same route with schemeId, open to everyone, TinyFish READY only), run in parallel with the source check by refreshSchemeWithTinyFish:
    1. reuses the last SUCCEEDED SCHEME_REFRESH job if it finished within 6 hours
    2. otherwise creates a SCHEME_REFRESH / TINYFISH job and runs tinyFishExtractSchemePage against the scheme's infoUrl
    3. on success, replaces the scheme's LIVE_FETCH evidence and sets it to LIVE_CHECKED
  • Batch discovery (POST /api/discovery, admin only): takes up to 5 sources, either the given sourceIds or the least-recently-checked ones. For each source it runs the direct HTTP check and, when TinyFish is READY, tinyFishExtractSourcePage in parallel (at most 2 sources at a time). The job succeeds if either check succeeds; a TinyFish failure never hides the direct-check result. The extraction (scheme names, announcements, deadlines) is stored in FetchJob.resultJson and returned to the admin UI.

7.3 Ask SchemeAI assistant

/api/assistant works in three steps:

  1. Retrieval. Loads all schemes and scores each one by keyword overlap with the question, plus a large boost for a context schemeId. It keeps the top 5.
  2. Prompting. Builds a grounded context (benefits, rule labels, official URL) and a strict system prompt: answer only from context, cite sources, optionally reply in Hindi.
  3. Answer. Tries the LLM through z-ai-web-dev-sdk. If that fails or the answer is empty, it returns a deterministic bulleted list of the retrieved schemes with their source URLs.

The response includes mode (AI_GROUNDED or DETERMINISTIC), cited schemes and a disclaimer.

8. Configuration and environment

Variables, from .env.example and their uses in code. The assistant has no variable of its own; its LLM access is configured by z-ai-web-dev-sdk itself.

Variable Required Used by Purpose
DATABASE_URL Yes prisma/schema.prisma, db.ts PostgreSQL connection string (pooled Neon URL recommended)
POSTGRES_PRISMA_URL No db.ts If set, used at runtime instead of DATABASE_URL (Vercel Postgres convention). Not in .env.example
SETUP_SECRET For /api/setup /api/setup Shared secret checked against the x-setup-secret header
TINYFISH_API_KEY No tinyfish.ts Enables the TinyFish integration (READY)
TINYFISH_API_URL No tinyfish.ts API base URL (default https://agent.tinyfish.ai)
NODE_ENV Set by Next auth.ts, db.ts Secure cookies in production. Prisma singleton caching outside production

Build and runtime configuration:

  • next.config.ts: typescript.ignoreBuildErrors: true, reactStrictMode: false. No output mode is set.
  • vercel.json: framework nextjs, npm run build, region bom1 (Mumbai).
  • package.json (name schemeai): build runs prisma generate && next build, and postinstall runs prisma generate. Seeds run with tsx. npm test runs the engine tests and npm run typecheck runs tsc --noEmit.
  • Route segment config: /api/sources/verify exports maxDuration = 180 and /api/discovery exports maxDuration = 300, so TinyFish runs are not cut off on serverless hosts whose plan allows those limits.

9. Deployment topology

The documented target (DEPLOY-VERCEL.md) is Vercel serverless functions in the bom1 region, backed by a Neon PostgreSQL database.

flowchart LR
    Dev["Developer"] -->|"git push"| GH["GitHub repo"]
    GH -->|"auto build: prisma generate, next build"| V["Vercel project, region bom1"]
    User["Browser"] -->|"HTTPS"| V
    V -->|"Prisma, pooled URL"| Neon[("Neon PostgreSQL")]
    V -->|"live checks"| Gov["Official .gov.in sites"]
    V -.->|"optional"| TF["TinyFish API (agent.tinyfish.ai)"]
    TF -.-> Gov
    V -.->|"optional"| LLM["LLM provider via SDK"]
    Op["Operator"] -->|"POST /api/setup?setup=1 with x-setup-secret"| V
    Op -.->|"or: prisma db push + npm run db:seed"| Neon
Loading

First-time database setup

Two options:

  1. Local CLI against the production URL: npx prisma db push, then npm run db:seed, then npm run db:seed:admin.
  2. HTTP: POST /api/setup?setup=1 with the header x-setup-secret: <SETUP_SECRET>. This upserts sources, schemes, rules and evidence, and creates the demo admin account. It does not create tables, so the schema must already exist (via prisma db push).

Afterwards, GET /api/health reports DB connectivity and the scheme and source counts. The build never needs a database connection, because only prisma generate runs at build time.

10. Tech stack

Area Technology
Framework Next.js 16 (App Router, route handlers), React 19
Language TypeScript 5
Styling Tailwind CSS v4, tw-animate-css, shadcn/ui (Radix primitives), lucide-react icons
Client state Zustand 5 with persist (localStorage)
Validation Zod 4
ORM / DB Prisma 6, PostgreSQL (Neon recommended)
Auth Custom: Node crypto scrypt + DB sessions + httpOnly cookie
i18n Custom English/Hindi dictionary (i18n.tsx)
LLM (optional) z-ai-web-dev-sdk chat completions
Browser automation (optional) TinyFish agent HTTP API
Fonts next/font/google: Plus Jakarta Sans, Inter
Hosting Vercel (bom1)
Tooling ESLint 9 (eslint-config-next), tsx for scripts

package.json also lists several dependencies that the application code under src/app, src/lib and src/components/schemeai does not import, for example next-auth, next-intl, @tanstack/react-query, @mdxeditor/editor and @dnd-kit/*. They are inherited from the project template.

11. Known limitations

These follow directly from the current code:

  • No rate limiting. No endpoint is rate limited. Scheme refreshes (POST /api/sources/verify with schemeId) are open to everyone; TinyFish usage per scheme is bounded only by the 6-hour reuse of successful runs, and failed runs are not cached.
  • Source-level promotion is coarse. A successful direct check of a source's domain root marks every scheme linked to that source LIVE_CHECKED, although the check does not read scheme content.
  • Demo admin credentials. /api/setup and scripts/create-admin.ts create admin@schemeai.in / Admin@1234, and the sign-in view has a button that submits them. Change or remove that account on public deployments.
  • Type errors do not fail the build. next.config.ts sets typescript.ignoreBuildErrors: true; npm run typecheck is the check.
  • Assistant provider is fixed. The LLM path uses z-ai-web-dev-sdk with SDK defaults; there is no provider or model setting in the repo.
  • Profiles stay in the browser. The UserProfile table is read by /api/auth/me but never written. Document checklists are also local only, and ApplicationTracker.checklistJson is never written.
  • Never-checked sources are picked last. Discovery orders by lastCheckedAt ascending, and PostgreSQL sorts nulls last in ascending order.
  • The CONTAINS operator is not implemented. It is listed in the schema comment but falls through to UNKNOWN in the engine. The seed data does not use it.