File tree Expand file tree Collapse file tree 1 file changed +50
-1
lines changed Expand file tree Collapse file tree 1 file changed +50
-1
lines changed Original file line number Diff line number Diff line change 22
33## Ollama
44
5- For installation see: https://ollama.com/
5+ For installation see: https://ollama.com/ . Then start
6+ [ ollama server] ( https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama ) with
7+ ` ollama serve ` command.
68
79``` python
810from langchain_community.chat_models import ChatOllama
911
1012llm = ChatOllama(model = ' llava' )
1113```
1214
15+ ### Configure ollama wth OpenAI compatible API
16+
17+ Ollama supports OpenAI compatible APIs (see [ details] ( https://ollama.com/blog/openai-compatibility ) ).
18+
19+ > [ !TIP]
20+ > Such setup might be more convenient if you frequently switch between OpenAI API and
21+ > local models.
22+
23+ To configure ollama through OpenAI API in ` rai ` :
24+
25+ 1 . Add ` base_url ` to [ config.toml] ( ../config.toml )
26+
27+ ``` toml
28+ [openai ]
29+ simple_model = " llama3.2"
30+ complex_model = " llama3.2"
31+ ...
32+ base_url = " http://localhost:11434/v1"
33+ ```
34+
35+ ### Example of setting up vision models with tool calling
36+
37+ In this example ` llama3.2-vision ` will be used.
38+
39+ 1 . Create a custom ollama ` Modelfile ` and load the model
40+
41+ > [ !NOTE]
42+ > Such setup is not officially supported by Ollama and it's not guaranteed to be
43+ > working in all cases.
44+
45+ ``` shell
46+ ollama pull llama3.2
47+ echo FROM llama3.2-vision > Modelfile
48+ echo ' TEMPLATE """' " $( ollama show --template llama3.2) " ' """' >> Modelfile
49+ ollama create llama3.2-vision-tools
50+ ```
51+
52+ 3 . Configure the model through OpenAI compatible API in [ config.toml] ( ../config.toml )
53+
54+ ``` toml
55+ [openai ]
56+ simple_model = " llama3.2-vision-tools"
57+ complex_model = " llama3.2-vision-tools"
58+ ...
59+ base_url = " http://localhost:11434/v1"
60+ ```
61+
1362## OpenAI
1463
1564``` bash
You can’t perform that action at this time.
0 commit comments