7.1 Usecase.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af366c49-944d-4ad3-9bf9-cf0b5e386cc6",
   "metadata": {},
   "source": [
    "# Why Pretraining?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bb03d167-6ebc-4da9-87c8-11ce1bd4eeaa",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "# Install any packages if it does not exist\n",
    "# !pip install -q -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f3d6b90b-7f1f-4f1f-a5fb-47bea82d3896",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "# Ignore insignificant warnings (ex: deprecations)\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8b23a1a8-096f-4563-ab87-bbf3c2e699a2",
   "metadata": {
    "height": 183
   },
   "outputs": [],
   "source": [
    "# Set a seed for reproducibility\n",
    "import torch\n",
    "\n",
    "def fix_torch_seed(seed=42):\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed(seed)\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.backends.cudnn.benchmark = False\n",
    "\n",
    "fix_torch_seed()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95414776-4bd9-4f87-93ce-2e4d1ec449b3",
   "metadata": {},
   "source": [
    "## 2. Load a general pretrained model\n",
    "\n",
    "We will work with small models that fit within the memory of the learning platform. TinySolar-248m-4k is a small decoder-only model with 248M parameters (similar in scale to GPT2) and a 4096 token context window. You can find the model on the Hugging Face model library at [this link](https://huggingface.co/upstage/TinySolar-248m-4k).\n",
    "\n",
    "Load the model in three steps:\n",
    "1. Specify the path to the model in the Hugging Face model library\n",
    "2. Load the model using `AutoModelforCausalLM` in the `transformers` library\n",
    "3. Load the tokenizer for the model from the same model path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8405c72c-21d5-4a12-bbad-08a6e25be383",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "model_path_or_name = \"./models/upstage/TinySolar-248m-4k\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1cfe44f2-7376-4647-a0b3-719794316812",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "from transformers import AutoModelForCausalLM\n",
    "tiny_general_model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_path_or_name,\n",
    "    device_map=\"cpu\", # change to auto if you have access to a GPU\n",
    "    torch_dtype=torch.bfloat16\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ecf36736-ba12-4c73-bedb-1aaab9007220",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "tiny_general_tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_path_or_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d4d6924-b8a0-4e5b-a66d-e316e14a564d",
   "metadata": {},
   "source": [
    "## 3. Generate text samples\n",
    "\n",
    "Here we'll try generating some text with the model. You'll set a prompt, instantiate a text streamer, and then have the model complete the prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "804a5efc-45b1-4d93-8fde-df8a102d08e1",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "prompt = \"I am an engineer. I love\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "400eeca5-67ea-437c-a8fe-98d4a66d90e4",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "inputs = tiny_general_tokenizer(prompt, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1c1a9c24-5b1a-482c-a14a-90fc4334ceb1",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "from transformers import TextStreamer\n",
    "streamer = TextStreamer(\n",
    "    tiny_general_tokenizer,\n",
    "    skip_prompt=True, # If you set to false, the model will first return the prompt and then the generated text\n",
    "    skip_special_tokens=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f41d5707-4ee1-4d82-a515-431235a3d775",
   "metadata": {
    "height": 166
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to travel and have a great time, but I'm not sure if I can do it all again.\n",
      "I've been working on my first book for the last 10 years, and I've always wanted to write about books that are more than just a collection of short stories. I've also written a few short stories, which I hope will be published in a future anthology.\n",
      "I've been writing fiction since I was 12, and I've read many different genres over the years. I've also written a number of non-fiction books, including a novel called\n"
     ]
    }
   ],
   "source": [
    "outputs = tiny_general_model.generate(\n",
    "    **inputs, \n",
    "    streamer=streamer, \n",
    "    use_cache=True,\n",
    "    max_new_tokens=128,\n",
    "    do_sample=False, \n",
    "    temperature=0.0,\n",
    "    repetition_penalty=1.1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29640cf-3429-4e0c-be9c-a74833ddc29b",
   "metadata": {},
   "source": [
    "## 4. Generate Python samples with pretrained general model\n",
    "\n",
    "Use the model to write a python function called `find_max()` that finds the maximum value in a list of numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e87d6a02-ec21-4352-8dec-203bf2cacf16",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "prompt =  \"def find_max(numbers):\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "859db93d-c214-450b-8da3-57abfabf55b3",
   "metadata": {
    "height": 166
   },
   "outputs": [],
   "source": [
    "inputs = tiny_general_tokenizer(\n",
    "    prompt, return_tensors=\"pt\"\n",
    ").to(tiny_general_model.device)\n",
    "\n",
    "streamer = TextStreamer(\n",
    "    tiny_general_tokenizer, \n",
    "    skip_prompt=True, # Set to false to include the prompt in the output\n",
    "    skip_special_tokens=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "de83f196-0118-4797-8894-c781a53c43a4",
   "metadata": {
    "height": 166
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "       \"\"\"\n",
      "       Returns the number of times a user has been added to the list.\n",
      "       \"\"\"\n",
      "       return num_users() + 1\n",
      "\n",
      "   def get_user_id(self, id):\n",
      "       \"\"\"\n",
      "       Returns the number of users that have been added to the list.\n",
      "       \"\"\"\n",
      "       return self._get_user_id(id)\n",
      "\n",
      "   def get_user_name(self, name):\n",
      "       \"\"\"\n",
      "       Returns the name of the user that has been added to the list.\n",
      "       \"\"\"\n",
      "       return self._get_user_name(name\n"
     ]
    }
   ],
   "source": [
    "outputs = tiny_general_model.generate(\n",
    "    **inputs, \n",
    "    streamer=streamer, \n",
    "    use_cache=True, \n",
    "    max_new_tokens=128, \n",
    "    do_sample=False, \n",
    "    temperature=0.0, \n",
    "    repetition_penalty=1.1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78517b2d",
   "metadata": {
    "height": 30
   },
   "source": [
    "The output is very random the reason is the model was not trained on python code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf5d693-9eb0-4394-8ad6-262ac5a48656",
   "metadata": {},
   "source": [
    "## 5. Generate Python samples with finetuned Python model\n",
    "\n",
    "This model has been fine-tuned on instruction code examples. You can find the model and information about the fine-tuning datasets on the Hugging Face model library at [this link](https://huggingface.co/upstage/TinySolar-248m-4k-code-instruct)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2c8669a4-384b-4de5-a4e3-7fecc81fd065",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "model_path_or_name = \"./models/upstage/TinySolar-248m-4k-code-instruct\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1aaf4752-c21d-44bd-8772-b1826d8f3bea",
   "metadata": {
    "height": 166
   },
   "outputs": [],
   "source": [
    "tiny_finetuned_model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_path_or_name,\n",
    "    device_map=\"cpu\",\n",
    "    torch_dtype=torch.bfloat16,\n",
    ")\n",
    "\n",
    "tiny_finetuned_tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_path_or_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4a9b9233-c384-45ee-9fbb-cf39611b83bb",
   "metadata": {
    "height": 370
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "   if len(numbers) == 0:\n",
      "       return \"Invalid input\"\n",
      "   else:\n",
      "       return numbers[i]\n",
      "```\n",
      "\n",
      "In this solution, the `find_max` function takes a list of numbers as input and returns the maximum value in that list. It then iterates through each number in the list and checks if it is greater than or equal to 1. If it is, it adds it to the `max` list. Finally, it returns the maximum value found so far.\n"
     ]
    }
   ],
   "source": [
    "prompt =  \"def find_max(numbers):\"\n",
    "\n",
    "inputs = tiny_finetuned_tokenizer(\n",
    "    prompt, return_tensors=\"pt\"\n",
    ").to(tiny_finetuned_model.device)\n",
    "\n",
    "streamer = TextStreamer(\n",
    "    tiny_finetuned_tokenizer,\n",
    "    skip_prompt=True,\n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "outputs = tiny_finetuned_model.generate(\n",
    "    **inputs,\n",
    "    streamer=streamer,\n",
    "    use_cache=True,\n",
    "    max_new_tokens=128,\n",
    "    do_sample=False,\n",
    "    temperature=0.0,\n",
    "    repetition_penalty=1.1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee8f283-71da-4736-9ac1-ba4ce9d5367f",
   "metadata": {},
   "source": [
    "## 6. Generate Python samples with pretrained Python model\n",
    "\n",
    "Here we'll use a version of TinySolar-248m-4k that has been further pretrained (a process called **continued pretraining**) on a large selection of python code samples. You can find the model on Hugging Face at [this link](https://huggingface.co/upstage/TinySolar-248m-4k-py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "30a3d3a3-1b57-4e56-98ad-58631485a58c",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "model_path_or_name = \"./models/upstage/TinySolar-248m-4k-py\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1df7a061-ca41-41c3-8ffa-533a3f557d16",
   "metadata": {
    "height": 166
   },
   "outputs": [],
   "source": [
    "tiny_custom_model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_path_or_name,\n",
    "    device_map=\"cpu\",\n",
    "    torch_dtype=torch.bfloat16,    \n",
    ")\n",
    "\n",
    "tiny_custom_tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_path_or_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "db2240fd-19d6-4f44-89dc-9325f4fdc6b0",
   "metadata": {
    "height": 336
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "   \"\"\"Find the maximum number of numbers in a list.\"\"\"\n",
      "   max = 0\n",
      "   for num in numbers:\n",
      "       if num > max:\n",
      "           max = num\n",
      "   return max\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompt = \"def find_max(numbers):\"\n",
    "\n",
    "inputs = tiny_custom_tokenizer(\n",
    "    prompt, return_tensors=\"pt\"\n",
    ").to(tiny_custom_model.device)\n",
    "\n",
    "streamer = TextStreamer(\n",
    "    tiny_custom_tokenizer,\n",
    "    skip_prompt=True, \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "outputs = tiny_custom_model.generate(\n",
    "    **inputs, streamer=streamer,\n",
    "    use_cache=True, \n",
    "    max_new_tokens=128, \n",
    "    do_sample=False, \n",
    "    repetition_penalty=1.1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ec94120",
   "metadata": {},
   "source": [
    "Try running the python code the model generated above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d109e788-2128-470d-8099-0a641938e062",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "def find_max(numbers):\n",
    "   max = 0\n",
    "   for num in numbers:\n",
    "       if num > max:\n",
    "           max = num\n",
    "   return max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "868a767b-b5a1-4986-bef5-156a7e5a7acb",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_max([1,3,5,1,6,7,2])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}