Gemma2 Fine-Tuning Demo

This project was forked from https://github.com/google-gemini/gemma-cookbook

This repository demonstrates how to fine-tune the Gemma2 2B model for specific tasks in the Chavacano language, such as answering online inquiries and processing orders. The project leverages LoRA (Low-Rank Adaptation) for efficient fine-tuning and integrates with HuggingFace for model hosting and deployment.

The dataset used to fine-tune the model consists of 20 email inquiries and their corresponding responses from an imaginary handicrafts store, Zambo Artesanía. The inquiries range from product availability and pricing to shipping and order follow-ups, providing a diverse set of scenarios to train the model on.

Features

• Fine-tune the Gemma2 2B model for Chavacano-based NLP tasks.
• Seamless integration with HuggingFace for model hosting and sharing.
• Lightweight Flask-based demo app to test model inference in real-time.

Pre-requisites:

Kaggle account (for downloading base Gemma2 model)

• KAGGLE_USERNAME
• KAGGLE_KEY

HuggingFace account (for uploading finetuned model)

• HF_USERNAME
• HF_TOKEN

Google Cloud VM instance

• GPU Type: NVidia L4
• CPU Type: g2-standar-4 Custom (4 vCPU, 2 core, 32 GB)
• Boot Disk Image: Deep Learning VM for TensorFlow 2.17 with CUDA 12.3 M125 (c1-deeplearning-tf-2-17-cu123-v20240922-debian-11-py310)

Highlights:

• Base Model: gemma2_instruct_2b_en
• Data Set: 98Labs/chavacano-small-dataset
• Finetuned Model: 98Labs/Gemma-2-CBK-2B-Instruct
• Finetuned Model size: 10.5GB

Finetuning Configuration

token_limit = 256
batch_size_value = 1
lora_rank = 4
learning_rate_value = 1e-4
train_epochs = 20

Finetuning Duration

• Finetuning using LoRA: 5.5min
• Upload to huggingface: 7min
• Total Duration: 12.5min

Project Setup

1. Clone the Project and Install Dependencies

Start by cloning the repository and running the installation scripts:

git clone https://github.com/98labs/gemma2-finetuning-demo
cd gemma2-finetuning-demo
./installation.sh
./setup_python.sh

2. Configure Environment Variables

Create a .env file with the following keys and populate them with your credentials:

KAGGLE_USERNAME=your_kaggle_username
KAGGLE_KEY=your_kaggle_api_key
HF_USERNAME=your_huggingface_username
HF_TOKEN=your_huggingface_api_token

Save the .env file in the following directories:

• gemma2-finetuning-demo/k-gemma-it/.env
• gemma2-finetuning-demo/k-mail-replier/.env

This configuration ensures that the scripts have access to required APIs and authentication tokens.

Fine-Tuning the Model

Navigate to the fine-tuning directory and run the script to train the model:

cd gemma2-finetuning-demo/k-gemma-it/
./tune_model.sh

The script will load the dataset, fine-tune the Gemma2 2B model using LoRA, and generate a weights file (*.h5) for deployment.

Uploading the Fine-Tuned Model to HuggingFace

Once fine-tuning is complete, upload the model to HuggingFace for hosting and sharing:

cd gemma2-finetuning-demo/k-gemma-it/
./upload_model_to_huggingface.sh

This step ensures the model is accessible online and can be integrated into other applications or shared with collaborators.

Deploying the Fine-Tuned Model for Inference

Copy the weights file from the fine-tuning directory to the demo app directory for inference:

cd gemma2-finetuning-demo/k-gemma-it/
./deploy_weights.sh

This script transfers the trained model weights to the k-mail-replier project.

Running the Demo Application

The demo application is a Python Flask-based website that allows you to test the fine-tuned model with real-time inputs.

Start the Flask App

Run the demo application:

cd gemma2-finetuning-demo/k-mail-replier/
./run_flask_app.sh

Access the Application

Open a web browser and go to:

http://finetuning.with98labs.com:5000/

The application allows you to input email inquiries, which are processed by the fine-tuned model to generate contextually relevant responses.

Troubleshooting

• Missing Dependencies: Ensure that all required Python packages are installed. You can re-run ./setup_python.sh if necessary.
• API Authentication Errors: Verify that your .env files are correctly configured and saved in the appropriate directories.
• Model Deployment Issues: Check that the *.h5 weights file was successfully copied to the k-mail-replier directory.

Contributions

Contributions to this project are welcome! Feel free to open an issue or submit a pull request for improvements or new features.

License

This project is licensed under Apache License, Version 2.0. See the LICENSE file for details.