fast-test-of-llama-v2-pre-quantized-with-llama-cpp.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/gpreda/fast-test-of-llama-v2-pre-quantized-with-llama-cpp?scriptVersionId=144643266\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"# Introduction\n\n\nLet's test now the Llama v2 model, quantized using llama.cpp, and load in GGUF format from the output of a previously run Notebook (https://www.kaggle.com/code/gpreda/test-llama-2-quantized-with-llama-cpp), where we performed the conversion of the LLama v2 (Hugging Face / Chat) model.\n\nHere we will use GPU T4 x2 to run the quantized model.","metadata":{}},{"cell_type":"markdown","source":"# Let's run it","metadata":{}},{"cell_type":"markdown","source":"Let's first pip install llama.cpp.python.","metadata":{}},{"cell_type":"code","source":"!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" pip install llama-cpp-python\n!git clone https://github.com/ggerganov/llama.cpp.git","metadata":{"_kg_hide-input":true,"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we import from the llama_cpp package the Llama module.","metadata":{}},{"cell_type":"code","source":"from llama_cpp import Llama\nfrom time import time","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load the quantized model.","metadata":{}},{"cell_type":"code","source":"t1 = time()\nllm = Llama(model_path=\"/kaggle/input/test-llama-2-quantized-with-llama-cpp/llama-7b.gguf\")\nt2 = time()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Time to load the model: {round(t2 - t1, 3)} sec.\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And now, let's just try a prompt.","metadata":{}},{"cell_type":"code","source":"t1 = time()\noutput = llm(\"Q: Name three capital cities in Europe? A: \", \n             max_tokens=38, \n             stop=[\"Q:\", \"\\n\"], \n             echo=True)\nt2 = time()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Time to run the query: {round(t2 - t1, 3)} sec.\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And now let's inspect the output.","metadata":{}},{"cell_type":"code","source":"output","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's all. Try it out yourself.","metadata":{}}]}