File tree Expand file tree Collapse file tree 1 file changed +50
-1
lines changed Expand file tree Collapse file tree 1 file changed +50
-1
lines changed Original file line number Diff line number Diff line change 22
33## Ollama  
44
5- For installation see: https://ollama.com/ 
5+ For installation see: https://ollama.com/ . Then start
6+ [ ollama server] ( https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama )  with
7+ ` ollama serve `  command.
68
79``` python 
810from  langchain_community.chat_models import  ChatOllama
911
1012llm =  ChatOllama(model = ' llava' 
1113``` 
1214
15+ ### Configure ollama wth OpenAI compatible API  
16+ 
17+ Ollama supports OpenAI compatible APIs (see [ details] ( https://ollama.com/blog/openai-compatibility ) ).
18+ 
19+ >  [ !TIP] 
20+ >  Such setup might be more convenient if you frequently switch between OpenAI API and
21+ >  local models.
22+ 
23+ To configure ollama through OpenAI API in ` rai ` :
24+ 
25+ 1 .  Add ` base_url `  to [ config.toml] ( ../config.toml ) 
26+ 
27+ ``` toml 
28+ [openai ]
29+ simple_model  = " llama3.2" 
30+ complex_model  = " llama3.2" 
31+ ... 
32+ base_url  = " http://localhost:11434/v1" 
33+ ``` 
34+ 
35+ ### Example of setting up vision models with tool calling  
36+ 
37+ In this example ` llama3.2-vision `  will be used.
38+ 
39+ 1 .  Create a custom ollama ` Modelfile `  and load the model
40+ 
41+ >  [ !NOTE] 
42+ >  Such setup is not officially supported by Ollama and it's not guaranteed to be
43+ >  working in all cases.
44+ 
45+ ``` shell 
46+ ollama pull llama3.2
47+ echo  FROM llama3.2-vision >  Modelfile
48+ echo  ' TEMPLATE """' " $( ollama show --template llama3.2) " ' """' >>  Modelfile
49+ ollama create llama3.2-vision-tools
50+ ``` 
51+ 
52+ 3 .  Configure the model through OpenAI compatible API in [ config.toml] ( ../config.toml ) 
53+ 
54+ ``` toml 
55+ [openai ]
56+ simple_model  = " llama3.2-vision-tools" 
57+ complex_model  = " llama3.2-vision-tools" 
58+ ... 
59+ base_url  = " http://localhost:11434/v1" 
60+ ``` 
61+ 
1362## OpenAI  
1463
1564``` bash 
 
 
   
 
     
   
   
          
    
    
     
    
      
     
     
    You can’t perform that action at this time.
  
 
    
  
    
      
        
     
       
      
     
   
 
    
    
  
 
  
 
     
    
0 commit comments