QuizMate - AI-Powered Quiz Generator (Proof-of-concept)

An intelligent RAG (Retrieval-Augmented Generation) application that generates contextual quiz questions from your documents using vector search and LLM capabilities.

Project Overview

QuizMate is a full-stack application designed to help users create personalized quizzes from their own documents (PDFs and Markdown files). The system uses a RAG pipeline to:

1. Ingest documents - Upload and process PDF or Markdown files
2. Store embeddings - Convert document chunks into vector embeddings stored in Qdrant
3. Generate quizzes - Query relevant content and use OpenAI to create contextual quiz questions
4. Interactive testing - Take quizzes with multiple-choice, true/false, and open-ended questions
5. Evaluation - Built-in LLM-as-a-judge evaluation system to assess RAG quality

Use Cases:

• Students preparing for exams from textbooks
• Teachers creating assessments from course materials
• Professionals testing knowledge from documentation
• Self-learners validating understanding of technical content

Dataset & Document Processing

Supported Formats

• PDF files - Textbooks, research papers, documentation
• Markdown files - Technical docs, notes, articles

Chunking Strategy

The application uses a hybrid chunking approach that balances semantic coherence with size constraints:

Current Implementation:

• Sentence-aware chunking using Apache OpenNLP
• Max tokens per chunk: 512 tokens (configurable via quizmate.chunking.max-tokens)
• Overlap: 50 tokens between chunks (configurable via quizmate.chunking.overlap-tokens)
• Text cleaning: Removes extra whitespace, normalizes characters

How it works:

1. Split text into sentences using OpenNLP sentence detector
2. Group sentences until reaching token limit (512 tokens)
3. Add overlap from previous chunk (50 tokens) for context continuity
4. Store each chunk with source metadata in Qdrant

Known Limitations

The current chunking strategy is functional but has room for improvement:

1. Fixed-size chunking - Doesn't consider semantic boundaries beyond sentences
2. No hierarchical structure - Headers, sections, and document structure are lost
3. Limited text cleaning - Basic normalization only

🛠️ Tech Stack

Backend

• Java 25 - Core application language
• Spring Boot 3.5.6 - Application framework
• Spring Data JPA - Database access
• H2 Database - Lightweight embedded database for source tracking

Vector Database & Embeddings

• Qdrant - Vector database for semantic search
• Deep Java Library (DJL) - ML framework for Java
• sentence-transformers/all-MiniLM-L6-v2 - Embedding model (384 dimensions)

LLM Integration

• OpenAI GPT - Quiz generation and evaluation
• GPT-3.5-turbo - Default model for quiz generation
• GPT-4 - Evaluation judge model

Document Processing

• Spring AI PDF Reader - PDF text extraction
• Apache OpenNLP - Sentence detection and tokenization

Preparation

Prerequisites

1. Java 25 (or compatible JDK)

   java --version

2. Maven (for building)

   mvn --version

3. Qdrant Vector Database

   Option A: Docker (Recommended)

   docker run -p 6334:6334 -p 6333:6333 \
       -v $(pwd)/qdrant_storage:/qdrant/storage \
       qdrant/qdrant

   Option B: Local Installation

   • Download from Qdrant releases
   • Follow installation instructions for your OS

4. OpenAI API Key

   Get your API key from OpenAI Platform

Configuration

1. Set OpenAI API Key (required)

   # Linux/Mac
   export OPENAI_API_KEY="sk-your-api-key-here"
   
   # Windows
   set OPENAI_API_KEY=sk-your-api-key-here

2. Configure Application (optional)

   Edit src/main/resources/application.properties:

   # Qdrant Configuration (if not using defaults)
   quizmate.qdrant.host=localhost
   quizmate.qdrant.port=6334
   quizmate.qdrant.collection-name=cs-textbooks
   
   # LLM Configuration
   quizmate.llm.model=gpt-3.5-turbo
   quizmate.evaluation.judge-model=gpt-4
   
   # Chunking Strategy
   quizmate.chunking.max-tokens=512
   quizmate.chunking.overlap-tokens=50

Running the Application

1. Start Qdrant

# If using Docker
docker run -p 6334:6334 qdrant/qdrant

2. Build and Run QuizMate

# Clone the repository
git clone <your-repo-url>
cd quizmate

# Build with Maven
./mvnw clean install

# Run the application
./mvnw spring-boot:run

3. Access the Application

Open your browser and navigate to:

http://localhost:8080

4. Using QuizMate

Step 1: Upload Documents

• Click "Select File" and choose a PDF or Markdown file
• Enter a source name (e.g., "ReactDocs", "SystemDesignBook")
• Click "Upload and Ingest"

Step 2: Generate Quiz

• Enter a topic/query (e.g., "database normalization")
• Select the source from the dropdown
• Choose number of questions (1-20)
• Click "Generate Quiz"

Step 3: Take the Quiz

• Answer each question (multiple-choice, true/false, or open-ended)
• Click "Submit Quiz"
• View your score and correct answers

📁 Code Structure Overview

Core Components

src/main/java/ai/quizmate/
├── config/                          # Configuration classes
│   ├── QdrantProperties.java       # Qdrant connection settings
│   ├── ChunkingProperties.java     # Chunking strategy config
│   ├── LlmProperties.java          # LLM model settings
│   └── EvaluationProperties.java   # Evaluation system config
│
├── ingestion/                       # Document ingestion pipeline
│   └── IngestionPipeline.java      # Orchestrates PDF/MD processing
│
├── service/                         # Business logic
│   ├── HybridChunkingService.java  # Sentence-aware chunking
│   ├── EmbeddingService.java       # Text → vectors (DJL)
│   ├── RetrievalService.java       # Vector search in Qdrant
│   ├── AugmentationService.java    # Prompt building
│   ├── LlmService.java             # OpenAI API calls
│   └── SourceService.java          # Source management
│
├── repository/                      # Data access
│   ├── QdrantRepository.java       # Vector DB operations
│   └── SourceRepository.java       # H2 database access
│
├── evaluation/                      # RAG quality evaluation
│   ├── EvaluationService.java      # LLM-as-a-judge logic
│   ├── EvaluationController.java   # Evaluation endpoints
│   └── model/                       # Evaluation data models
│
├── facade/                          # REST controllers
│   ├── IngestionRestController.java # File upload & sources API
│   └── QuizRestController.java      # Quiz generation API
│
└── model/                           # Data models
    ├── entity/Source.java           # Source entity (H2)
    ├── request/                     # API request models
    └── response/                    # API response models

Key Classes Explained

1. IngestionPipeline

Orchestrates the document ingestion process:

PDF/Markdown → Text Extraction → Cleaning → Chunking → Embedding → Qdrant Storage

2. HybridChunkingService

Implements sentence-aware chunking:

• Uses OpenNLP for sentence detection
• Respects token limits
• Maintains overlap between chunks

3. EmbeddingService

Converts text to 384-dimensional vectors:

• Uses sentence-transformers/all-MiniLM-L6-v2
• Automatically detects vector dimensions

4. RetrievalService

Performs semantic search:

• Embeds query
• Searches Qdrant with source filtering
• Returns top-K most relevant chunks

5. AugmentationService

Builds prompts for LLM:

• Combines query + retrieved context
• Instructs LLM on quiz format
• Defines question types (multiple-choice, true/false, open-ended)

6. EvaluationService

LLM-as-a-judge for quality assessment:

• Faithfulness: No hallucinations
• Answer Relevance: Addresses the query
• Context Relevance: Good retrieval
• Answer Quality: Well-structured responses

API Endpoints

Document Management

• POST /ingestion/upload - Upload and ingest document
• GET /ingestion/sources - List available sources

Quiz Operations

• POST /api/quiz - Generate quiz from query

Evaluation

• POST /api/evaluation/evaluate-query - Evaluate single query
• POST /api/evaluation/evaluate-batch - Batch evaluation with aggregate stats

🔧 Future Enhancements

1. Improved Chunking Strategy

Current limitations:

• Fixed-size chunks ignore document structure
• No semantic boundary detection beyond sentences
• Tables and code blocks poorly handled

2. Advanced Text Cleaning

Current: Basic whitespace normalization

3. Qdrant Filtering & Boosting

Current: Simple source-based filtering

4. Query Enhancement with LLM

Current: User query sent directly to retrieval

Proposed pipeline:

User Query → LLM Enhancement → Better Retrieval → Better Quiz

Enhancement strategies:

• Query expansion:

  Input: "React hooks"
  Enhanced: "React hooks including useState, useEffect, useContext, custom hooks"
  

• Hypothetical Document Embeddings (HyDE):

  Input: "What is database normalization?"
  Generate hypothetical answer → Embed → Search with that embedding
  

• Multi-query generation:

  Input: "Explain microservices"
  Generate: [
    "What are microservices?",
    "Benefits of microservices architecture",
    "Microservices vs monolithic architecture"
  ]
  Search all → Deduplicate results
  

5. Monitoring & Observability

6. Enhanced Evaluation

Current: LLM-as-a-judge on 4 metrics