FinanceGPT — Retrieval-Augmented Generation System

BSc Data Science for Responsible Business — Centrale Lyon, 2025-2026
Deep Learning — Project: Implementing a RAG System using ChromaDB and Hugging Face LLMs

---

Overview

FinanceGPT is a Retrieval-Augmented Generation (RAG) system designed for financial question answering. Instead of relying solely on the language model's internal knowledge, the system first retrieves relevant documents from an external knowledge base (ChromaDB) and injects them into the prompt before generating an answer. This grounds responses in factual, domain-specific data.

User question
     │
     ▼
Embedding model → semantic vector
     │
     ▼
ChromaDB → top-3 most relevant chunks
     │
     ▼
Prompt: <context> chunks </context> + <question> ... </question>
     │
     ▼
Hugging Face LLM → answer

---

Dataset

Finance Alpaca (gbharti/finance-alpaca, available on Hugging Face Datasets)

• ~50 000 financial Q&A pairs covering stocks, bonds, crypto, macroeconomics, personal finance, and more
• Each entry has an instruction (question), an optional input (context), and an output (answer)
• We index the first 10 000 documents as the knowledge base
• Documents indexed at positions 10 000+ are reserved as a held-out evaluation set (never seen during indexing)

Document format after preprocessing:

Q: What is a stock?
A: A stock represents a share of ownership in a company...

Or, when a context field is present:

Q: What is the P/E ratio of Apple?
Context: Apple reported earnings of $6.11 per share...
A: The P/E ratio is calculated by dividing...

---

Technical Stack

Component	Tool	Source
Embedding model	all-MiniLM-L6-v2 (Sentence Transformers)	Hugging Face
Vector database	ChromaDB (PersistentClient)	local
Language model	Qwen/Qwen2.5-3B-Instruct	Hugging Face
Interface	Gradio	—
Runtime	Google Colab (T4 GPU)	—

---

Repository Structure

.
├── RAG_Colab.ipynb      # Main notebook — run on Google Colab (T4 GPU)
├── requirements.txt     # Python dependencies
├── Project_RAG.pdf      # Course project guidelines
└── finance_db/          # ChromaDB persistent store (auto-created on first run)

---

Implementation

1. Data Preparation

The Finance Alpaca dataset is loaded via datasets and preprocessed into plain-text documents. Each document combines the question and answer into a single string, making it self-contained for retrieval.

2. Embedding Generation

Following the course PDF (Section 2), we use SentenceTransformer("all-MiniLM-L6-v2") from Hugging Face to convert each document into a normalized semantic vector:

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

def emb_text(text):
    return embedding_model.encode([text], normalize_embeddings=True).tolist()[0]

3. Vector Database — ChromaDB

Following the course PDF (Section 3), we create a persistent ChromaDB collection. On first run, all 10 000 documents are embedded and stored. On subsequent runs, the existing collection is reloaded directly:

client = PersistentClient(path="./finance_db")

try:
    collection = client.get_collection(name="finance_alpaca")
except Exception:
    collection = client.create_collection(name="finance_alpaca")
    embeddings = [emb_text(doc) for doc in tqdm(documents)]
    collection.add(documents=documents, embeddings=embeddings, ids=[str(i) for i in range(len(documents))])

4. Retrieval

Following the course PDF (Section 4), a user question is embedded and queried against ChromaDB to retrieve the 3 most semantically similar documents:

def retrieve(question, n=3):
    query_embedding = emb_text(question)
    results = collection.query(query_embeddings=[query_embedding], n_results=n)
    context = "\n".join([line for line in results['documents'][0]])
    return context

5. LLM Integration

Following Session 6 of the course, we load Qwen/Qwen2.5-3B-Instruct from Hugging Face using AutoTokenizer and AutoModelForCausalLM. torch_dtype="auto" is used as shown in Session 6 to automatically select float16 on GPU:

# model_name = "gpt2"
# model_name = "mistralai/Mistral-7B-Instruct-v0.2"  # too large for T4 (~14 GB)
# model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
# model_name = "meta-llama/Llama-3.2-3B-Instruct"
model_name = "Qwen/Qwen2.5-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
model = model.to(device)

Qwen2.5-3B-Instruct is fully open access and fits comfortably on a T4 GPU (~6 GB in float16). Mistral-7B-Instruct-v0.2 was tested but uses ~14 GB, which saturates the T4. TinyLlama-1.1B-Chat was also tested but produced poor-quality responses due to its small size.

Generation follows the exact pattern from Session 6:

encoded_input = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    input_ids=encoded_input["input_ids"],
    attention_mask=encoded_input["attention_mask"],
    max_new_tokens=200,
    temperature=0.9,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)

6. Prompt Engineering

Following the XML template from the course PDF (Section 5):

Use the information enclosed in <context> tags to provide an answer
to the question enclosed in <question> tags.
<context>
{retrieved chunks}
</context>
<question>
{user question}
</question>

---

Evaluation & Optimization

All evaluation is performed on held-out samples (positions 10 000+ in Finance Alpaca, never indexed).

Metrics

Metric	Description
Retrieval keyword score	Fraction of expected-answer keywords found in the retrieved context
ROUGE-1 F1	Word-overlap F1 between extracted answer and expected answer
Retrieval latency	ChromaDB query time per question (seconds)

Embedding Model Comparison

We compare two Sentence Transformer models on retrieval quality over 20 held-out samples:

Model	Description
all-MiniLM-L6-v2	Used in course PDF — balanced speed and quality
paraphrase-MiniLM-L3-v2	Lighter and faster

Each model is evaluated on its own temporary ChromaDB collection built from 1 000 documents.

Prompt Template Comparison

We compare three prompt strategies and measure ROUGE-1 F1 on 3 test questions:

Template	Description
1 - Basic	Plain Context: ... Question: ... Answer:
2 - XML (PDF)	XML tags from course PDF: <context>, <question>
3 - XML + Instructions	XML tags with an explicit system instruction

---

How to Run

Google Colab (recommended)

1. Open RAG_Colab.ipynb in Google Colab
2. Go to Runtime → Change runtime type → T4 GPU
3. Run all cells in order — no token required

---

Requirements

transformers
chromadb
sentence-transformers
huggingface-hub
datasets
gradio
tqdm
accelerate
torch

Install with:

pip install -r requirements.txt

---

Example Questions

• What is a stock?
• How does inflation affect interest rates?
• What is the difference between a bond and a stock?
• What is dollar cost averaging?
• What is a P/E ratio?
• How does compound interest work?

---

References

• Course guidelines: Project_RAG.pdf — Emmanuel Dellandréa, Centrale Lyon 2025-2026
• Session 6 notebook: Deep Learning — Large Language Models, Centrale Lyon
• Dataset: gbharti/finance-alpaca
• ChromaDB: trychroma.com
• Sentence Transformers: sbert.net
• Qwen2.5-3B-Instruct: Qwen/Qwen2.5-3B-Instruct