This repository offers a comprehensive suite of resources and scripts for working with Large Language Models (LLMs). It encompasses tools for training, fine-tuning, pretraining, and inference using models like litGPT, Hugging Face Transformers, and custom GPT implementations. Leveraging frameworks such as PyTorch and PyTorch Lightning, it also supports Low-Rank Adaptation (LoRA) for efficient fine-tuning of large models.
-
Introduction_to_LLMs
Foundational notebooks to understand LLM architectures, tokenization, and attention mechanisms. -
LLMs_with_Hugging_Face
Scripts for pretraining and inference using Hugging Face Transformers, including training models from scratch and utilizing pretrained models. -
Finetune_LLMs_with_Litgpt
Resources for fine-tuning litGPT models using PyTorch Lightning, incorporating LoRA for parameter-efficient training. -
GPT_from_scratch
Implementations for building and training GPT models from the ground up using PyTorch.
- Comprehensive Workflow: Covers the entire LLM pipeline from foundational understanding to deployment.
- Framework Integration: Utilizes Hugging Face, litGPT, and PyTorch for versatile model development.
- Efficient Fine-Tuning: Implements LoRA for resource-effective model adaptation.
- Modular Design: Scripts are organized for easy navigation and customization.
- Practical Applications: Includes examples for real-world inference and deployment scenarios.
- Clone the Repository:
git clone https://github.com/dineshsoudagar/LLM-Lab-From-Scratch-to-Fine-Tuning.git
cd LLM-Lab-From-Scratch-to-Fine-Tuning- Install Dependencies:
Ensure you have Python 3.8+ installed. Then, install the required packages:
pip install -r requirements.txt- Explore the Modules:
Navigate through the directories to explore different aspects of LLM development and fine-tuning.
- Modularity: Each module is self-contained, allowing you to focus on specific areas of interest.
- Customization: Scripts can be adapted to suit different datasets and model configurations.
- Community Support: Contributions and feedback are welcome to enhance the repository's value.