You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Out of the box, the LLM xpack provides wrappers for text generation and embedding LLMs. For text generation, you can use native wrappers for the OpenAI chat model and HuggingFace models running locally. Many other popular models, including Azure OpenAI, HuggingFace (when using their API) or Gemini can be used with the [`LiteLLM`](/developers/user-guide/llm-xpack/llm-chats#litellm) wrapper. To check the full list of providers supported by LiteLLM check [LiteLLM documentation](https://docs.litellm.ai/docs/providers). Currently, Pathway provides wrappers for the following LLMs:
12
+
Out of the box, the LLM xpack provides wrappers for text generation and embedding LLMs. For text generation, you can use native wrappers for the OpenAI chat model and HuggingFace models running locally. Many other popular models, including Azure OpenAI, HuggingFace (when using their API) or Gemini can be used with the [`LiteLLM`](/developers/user-guide/llm-xpack/llm-chats#litellm) wrapper. To check the full list of providers supported by LiteLLM check [LiteLLM documentation](https://docs.litellm.ai/docs/providers).
13
+
::if{path="/llm-xpack/"}
14
+
Currently, Pathway provides wrappers for the following LLMs:
-[Hugging Face Pipeline](/developers/user-guide/llm-xpack/llm-chats#hugging-face-pipeline)
25
+
::
26
+
17
27
28
+
::if{path="/llm-xpack/"}
18
29
To use a wrapper, first create an instance of the wrapper, which you can then apply to a column containing prompts.
19
30
20
31
We create a Pathway table to be used in the examples below:
@@ -28,17 +39,21 @@ How many 'r' there are in 'strawberry'? | 400
28
39
split_on_whitespace=False,
29
40
)
30
41
```
42
+
::
31
43
44
+
::if{path="/llm-xpack/"}
32
45
## UDFs
33
46
34
47
Each wrapper is a [UDF](/developers/api-docs/pathway#pathway.UDF) (User Defined Function), which allows users to define their own functions to interact with Pathway objects. A UDF, in general, is any function that takes some input, processes it, and returns an output. In the context of the Pathway library, UDFs enable seamless integration of custom logic, such as invoking LLMs for specific tasks.
35
48
36
49
In particular a UDF can serve as a wrapper for LLM calls, allowing users to pass prompts or other inputs to a model and retrieve the corresponding outputs. This design makes it easy to interact with Pathway tables and columns while incorporating the power of LLMs.
50
+
::
37
51
38
52
## OpenAIChat
39
53
40
54
For OpenAI, you create a wrapper using the [`OpenAIChat` class](/developers/api-docs/pathway-xpacks-llm/llms#pathway.xpacks.llm.llms.OpenAIChat).
# Run the computations (including sending requests to OpenAI) and print the output table
52
67
pw.debug.compute_and_print(responses)
53
68
```
69
+
::
70
+
::if{path="/ai-pipelines/"}
71
+
```yaml
72
+
chat: !pw.xpacks.llm.llms.OpenAIChat
73
+
model: "gpt-4o-mini
74
+
api_key: $OPENAI_API_KEY
75
+
```
76
+
::
77
+
54
78
79
+
::if{path="/llm-xpack/"}
55
80
### Message format
56
81
`OpenAIChat`expects messages to be in the format required by [OpenAI API](https://platform.openai.com/docs/api-reference/chat/create) - that is a list of dictionaries, where each dictionary is one message in the conversation so far. For asking a single question, you can use [`pw.xpacks.llm.llm.prompt_chat_single_qa`](/developers/api-docs/pathway-xpacks-llm/llms#pathway.xpacks.llm.llms.prompt_chat_single_qa) to wrap a string so that it matches the format expected by OpenAI API. Our example above presents that use case.
Pathway has a wrapper for LiteLLM - [`LiteLLMChat`](/developers/api-docs/pathway-xpacks-llm/llms#pathway.xpacks.llm.llms.LiteLLMChat). For example, to use Gemini with LiteLLM, create an instance of `LiteLLMChat` and then apply it to the column with messages to be sent over API.
model: "gemini/gemini-pro", # Choose the model you want
138
+
```
139
+
::
106
140
107
141
With the wrapper for LiteLLM, Pathway allows you to use many popular LLMs.
108
142
109
143
## Hugging Face pipeline
110
144
For models from Hugging Face that you want to run locally, Pathway gives a separate wrapper called `HFPipelineChat` (for calling HuggingFace through API, use LiteLLM wrapper). When an instance of this wrapper is created, it initializes a HuggingFace `pipeline`, so any [arguments to the `pipeline`](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.pipeline) - including the name of the model - must be set during the initialization of `HFPipelineChat`. Any parameters to `pipeline.__call__` can be as before set during initialization or overridden during application.
111
145
146
+
::if{path="/llm-xpack/"}
112
147
```python
113
148
from pathway.xpacks.llm import llms
114
149
@@ -118,10 +153,21 @@ model = llms.HFPipelineChat(
model: "TinyLlama/TinyLlama-1.1B-Chat-v1.0", # Choose the model you want
161
+
```
162
+
::
121
163
122
164
Note that format of questions used in Hugging Face pipeline depends on the model. Some models, like [`gpt2`](https://huggingface.co/openai-community/gpt2), expect a prompt string, whereas conversation models also accept messages as a list of dicts. The model's prompt template will be used if a conversation with a list of dicts is passed.
165
+
::if{path="/ai-pipelines/"}
166
+
Note that Pathway AI pipelines expect conversation models, so models like `gpt2` cannot be used.
167
+
::
123
168
For more information, see [pipeline docs](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline.__call__.text_inputs).
124
169
170
+
::if{path="/llm-xpack/"}
125
171
For example for model [`TinyLlama/TinyLlama-1.1B-Chat-v1.0`](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0), you can use it with:
Pathway has a wrapper for the [`Cohere Chat Services`](https://docs.cohere.com/docs/command-beta). The wrapper allows for augmenting the query with documents. The result contains cited documents along with the response.
Wrapper for OpenAI and LiteLLM, both for chat and embedding, are asynchronous, and Pathway allows you to set three parameters to set their behavior. These are:
@@ -172,6 +220,7 @@ Wrapper for OpenAI and LiteLLM, both for chat and embedding, are asynchronous, a
172
220
173
221
These three parameters need to be set during the initialization of the wrapper. You can read more about them in the [UDFs guide](/developers/user-guide/data-transformation/user-defined-functions#asyncexecutor).
Copy file name to clipboardExpand all lines: docs/2.developers/4.user-guide/50.llm-xpack/.embedders/embedders.md
+34-2
Original file line number
Diff line number
Diff line change
@@ -21,6 +21,7 @@ The following embedding wrappers are available through the Pathway xpack:
21
21
## OpenAIEmbedder
22
22
The default model for [`OpenAIEmbedder`](/developers/api-docs/pathway-xpacks-llm/embedders/#pathway.xpacks.llm.embedders.OpenAIEmbedder) is `text-embedding-3-small`.
The model for [`LiteLLMEmbedder`](/developers/api-docs/pathway-xpacks-llm/embedders/#pathway.xpacks.llm.embedders.LiteLLMEmbedder) has to be specified during initialization. No default is provided.
This [`SentenceTransformerEmbedder`](/developers/api-docs/pathway-xpacks-llm/embedders/#pathway.xpacks.llm.embedders.SentenceTransformerEmbedder) embedder allows you to use the models from the Hugging Face Sentence Transformer models.
69
85
70
86
The model is specified during initialization. Here is a list of [`available models`](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html).
[`GemeniEmbedder`](/developers/api-docs/pathway-xpacks-llm/embedders/#pathway.xpacks.llm.embedders.GeminiEmbedder) is the embedder for Google's Gemeni Embedding Services. Available models can be found [`here`](https://ai.google.dev/gemini-api/docs/models/gemini#text-embedding-and-embedding).
112
+
## GeminiEmbedder
113
+
[`GeminiEmbedder`](/developers/api-docs/pathway-xpacks-llm/embedders/#pathway.xpacks.llm.embedders.GeminiEmbedder) is the embedder for Google's Gemini Embedding Services. Available models can be found [`here`](https://ai.google.dev/gemini-api/docs/models/gemini#text-embedding-and-embedding).
Copy file name to clipboardExpand all lines: docs/2.developers/4.user-guide/50.llm-xpack/.splitters/splitters.md
+25
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,7 @@ A better method is to chunk the text by tokens, ensuring each chunk makes sense
18
18
## TokenCountSplitter
19
19
Pathway offers a [`TokenCountSplitter`](/developers/api-docs/pathway-xpacks-llm/splitters#pathway.xpacks.llm.splitters.TokenCountSplitter) for token-based chunking. Here's how to use it:
20
20
21
+
::if{path="/llm-xpack/"}
21
22
```python
22
23
from pathway.xpacks.llm.splitters import TokenCountSplitter
This configuration creates chunks of 100–500 tokens using the `cl100k_base` tokenizer, compatible with OpenAI's embedding models.
32
42
@@ -42,6 +52,7 @@ However, the way it determines split points differs.
42
52
The splitter continues this process until all chunks are smaller than `chunk_size`.
43
53
Additionally, you can introduce overlapping chunks by setting the `chunk_overlap` parameter. This is particularly useful if you want to capture different contexts in your chunks. However, keep in mind that enabling overlap increases the total number of chunks retrieved, which could impact performance.
0 commit comments