|
8 | 8 |
|
9 | 9 | * The server creates the following dedicated collections,
|
10 | 10 | which contain documents that reference the embedding attachments:
|
11 |
| - * **Embeddings Collection** |
| 11 | + * **Embeddings Collections** |
12 | 12 | * **Embeddings Cache Collection**
|
13 | 13 |
|
14 | 14 | * This article describes these custom-designed collections.
|
15 |
| - It is recommended to first refer to this [overview](../../ai-integration/generating-embeddings/overview#embeddings-generation---overview) |
| 15 | + It is recommended to first refer to this [Overview](../../ai-integration/generating-embeddings/overview#embeddings-generation---overview) |
16 | 16 | to understand the embeddings generation process flow.
|
17 | 17 |
|
18 | 18 | ---
|
19 | 19 |
|
20 | 20 | * In this article:
|
21 |
| - * [The embeddings collection](../../ai-integration/generating-embeddings/embedding-collections#the-embeddings-collection) |
| 21 | + * [The embeddings collections](../../ai-integration/generating-embeddings/embedding-collections#the-embeddings-collections) |
22 | 22 | * [The embeddings cache collection](../../ai-integration/generating-embeddings/embedding-collections#the-embeddings-cache-collection)
|
23 | 23 |
|
24 | 24 | {NOTE/}
|
25 | 25 |
|
26 | 26 | ---
|
27 | 27 |
|
28 |
| -{PANEL: The embeddings collection} |
| 28 | +{PANEL: The embeddings collections} |
29 | 29 |
|
30 | 30 | * RavenDB creates a separate embeddings collection for each source collection from which embeddings are generated.
|
31 | 31 | The naming format for these collections is: `<source-collection-name>/embeddings`.
|
32 | 32 |
|
33 |
| -* Each document in the embeddings collection references ALL embeddings generated from the content of the corresponding document in the source collection by any defined embeddings generation task. |
| 33 | +* Each document in the embeddings collection references ALL embeddings generated from |
| 34 | + the content of the corresponding document in the source collection by any defined embeddings generation task. |
34 | 35 |
|
35 | 36 | * The document structure in the embeddings collection is:
|
36 | 37 |
|
37 | 38 | {CODE-BLOCK:json}
|
38 | 39 | {
|
39 |
| - "identifier-of-task-1": { |
40 |
| - "@quantization": "<quantization-type>", // Shown only if embeddings are Quantized |
41 |
| - "Property1": [ |
42 |
| - "A hash created from the 1st text chunk of Property1's content", |
43 |
| - "A hash created from the 2nd text chunk of Property1's content", |
44 |
| - "A hash created from the 3rd text chunk of Property1's content", |
45 |
| - "..." |
46 |
| - ], |
47 |
| - "Property2": [ |
48 |
| - "A hash created from the 1st text chunk of Property2's content", |
49 |
| - "..." |
50 |
| - ] |
51 |
| - }, |
52 |
| - "identifier-of-task-2": { |
53 |
| - "Property3": [ |
54 |
| - "A hash created from the 1st text chunk of Property3's content", |
55 |
| - "..." |
56 |
| - ] |
57 |
| - }, |
58 |
| - "Other-tasks...": { |
59 |
| - ... |
60 |
| - }, |
61 |
| - "@metadata": { |
62 |
| - "@collection": "<source-collection-name>/embeddings", |
63 |
| - "@flags": "HasAttachments" |
64 |
| - } |
| 40 | + "identifier-of-task-1": { |
| 41 | + "@quantization": "<quantization-type>", |
| 42 | + "Property1": [ |
| 43 | + "Hash of the embedding vector generated for 1st text chunk of Property1's content", |
| 44 | + "Hash of the embedding vector generated for 2nd text chunk of Property1's content", |
| 45 | + "Hash of the embedding vector generated for 3rd text chunk of Property1's content", |
| 46 | + "..." |
| 47 | + ], |
| 48 | + "Property2": [ |
| 49 | + "Hash of the embedding vector generated for 1st text chunk of Property2's content", |
| 50 | + "..." |
| 51 | + ] |
| 52 | + }, |
| 53 | + "identifier-of-task-2": { |
| 54 | + "Property3": [ |
| 55 | + "Hash of the embedding vector generated for 1st text chunk of Property3's content", |
| 56 | + "..." |
| 57 | + ] |
| 58 | + }, |
| 59 | + "Other-tasks...": { |
| 60 | + ... |
| 61 | + }, |
| 62 | + "@metadata": { |
| 63 | + "@collection": "<source-collection-name>/embeddings", |
| 64 | + "@flags": "HasAttachments" |
| 65 | + } |
65 | 66 | }
|
66 | 67 | {CODE-BLOCK/}
|
67 | 68 |
|
|
79 | 80 | 2. **Document ID**
|
80 | 81 | Each document ID in this collection follows the format: `<source-document-name>/embeddings`
|
81 | 82 | 3. **Task identifier**
|
82 |
| - The identifier of the task that generated the embeddings for the listed properties. |
83 |
| - 4. **Source properties & their hash**: |
| 83 | + The identifier of the task that generated the embeddings for the listed properties. |
| 84 | + 4. **Quantization type** |
| 85 | + The quantization method applied by the task when generating the embeddings. |
| 86 | + 5. **Source properties & their hashes**: |
84 | 87 | This section contains properties from the source document whose content was converted into embeddings.
|
85 |
| - Each property contains an array of hashes derived from text chunks created from the property's content: |
86 |
| - `<property-name>: [<hash-created-from-text-chunk-1>, <hash-created-from-text-chunk-2>, ...]` |
87 |
| - 5. **Attachment flag** |
| 88 | + Each property holds an array of Base64 hashes. |
| 89 | + Each hash is derived from the content of an embedding vector generated for a text chunk from the property's content: |
| 90 | + `<property-name>: [` |
| 91 | + `<hash-of-the-embedding-vector-for-text-chunk-1>,` |
| 92 | + `<hash-of-the-embedding-vector-for-text-chunk-2>,` |
| 93 | + `...` |
| 94 | + `]` |
| 95 | + 6. **Attachment flag** |
88 | 96 | Indicates that the document includes attachments, which store the embeddings.
|
89 | 97 | The next image shows the embedding attachments in the document's properties pane.
|
90 | 98 |
|
91 | 99 | 
|
92 | 100 |
|
93 |
| - * Each attachment contains a single embedding. |
94 |
| - |
95 |
| - * The attachment name follows this format: |
96 |
| - `<task-identifier>_<property-name>_<hash-of-text-chunk-from-property-content>` |
97 |
| - |
98 |
| - * If the embeddings were [Quantized](../../ai-integration/vector-search/vector-search-using-dynamic-query#what-is-quantization) by the task, |
99 |
| - the format includes the quantization type: |
100 |
| - `<task-identifier>_<property-name>_<hash-of-text-chunk-from-property-content>_<quantization-type>` |
| 101 | + * Each attachment contains a **single embedding**. |
| 102 | + |
| 103 | + * The **attachment name** is the Base64 hash derived from the content of the embedding vector stored in the attachment: |
| 104 | + `<hash-of-the-embedding-vector-content>` |
101 | 105 |
|
102 | 106 | {PANEL/}
|
103 | 107 |
|
104 | 108 | {PANEL: The embeddings cache collection}
|
105 | 109 |
|
106 | 110 | {CONTENT-FRAME: }
|
107 | 111 |
|
| 112 | +#### Cache contents |
| 113 | +--- |
| 114 | + |
108 | 115 | * In addition to creating embeddings collections for each source collection,
|
109 | 116 | RavenDB creates and maintains a single **embeddings cache collection** named: `@embeddings-cache`.
|
110 | 117 |
|
111 |
| -* This collection references all embeddings generated by all providers for all source collections. |
112 |
| - It includes embeddings generated from source documents as well as embeddings generated from search terms in vector search queries. |
| 118 | +* This cache collection contains embeddings generated by all providers, |
| 119 | + both from source documents and from search terms used in vector search queries. |
113 | 120 |
|
114 |
| -* Each document in the `@embeddings-cache` collection references a **single attachment** containing an embedding vector. |
| 121 | +* Each document in the `@embeddings-cache` collection references a **single attachment** that contains a single embedding vector. |
115 | 122 | **The document ID includes**:
|
116 | 123 | * The [connection string identifier](../../ai-integration/connection-strings/connection-strings-overview#creating-an-ai-connection-string),
|
117 | 124 | which specifies the provider and model that generated the embedding.
|
118 |
| - * A hash generated from a text chunk - either from a source document property or from a search term. |
| 125 | + * A Base64 hash generated from a text chunk value - either from a source document property or from a search term. |
119 | 126 | * If the embedding was quantized by the task, the document ID also includes the quantization type.
|
120 | 127 |
|
121 | 128 | {CONTENT-FRAME/}
|
|
158 | 165 | #### Expiration policy
|
159 | 166 | ---
|
160 | 167 |
|
161 |
| -* Each document in this cache collection has an expiration date. |
| 168 | +* **The expiration date**: |
| 169 | + Each document in this cache collection is created with an expiration date, which is set according to the expiration period defined in the embeddings generation task. |
162 | 170 | Once the expiration date is reached, the document is automatically deleted (provided that [document expiration](../../studio/database/settings/document-expiration) is enabled).
|
163 | 171 |
|
164 |
| -* When a source document (from which embeddings were generated) is modified, |
165 |
| - RavenDB extends the expiration date of the relevant document in this cache if the remaining time is less than one-third of the original duration. |
| 172 | +* **Extending the expiration period**: |
| 173 | + * When a source document (from which embeddings were generated) is modified - even if the change is not to a property used for embeddings - |
| 174 | + RavenDB checks the expiration of the matching document in the cache collection. |
| 175 | + If the remaining time is less than half of the original duration, RavenDB extends the expiration by the period defined in the task. |
| 176 | + * When you make a vector search query and an embedding generated from a chunk of the search term already exists in the cache, |
| 177 | + RavenDB also extends the expiration of the matching document by the period defined in the query settings of the embeddings generation task. |
166 | 178 |
|
167 | 179 | {CONTENT-FRAME/}
|
168 | 180 |
|
|
179 | 191 | The document ID includes the connection string identifier, which specifies the provider that generated the embedding.
|
180 | 192 |
|
181 | 193 | 3. **Hash**
|
182 |
| - The document ID includes a hash created from a text chunk - |
| 194 | + The document ID includes a Base64 hash created from a text chunk - |
183 | 195 | either from a source document property or from a search term in a vector search query.
|
184 | 196 |
|
185 | 197 | ---
|
|
192 | 204 | The document ID follows this format:
|
193 | 205 | `embeddings-cache/<connection-string-identifier>/<hash-of-text-chunk-from-property-or-search-term>`
|
194 | 206 |
|
195 |
| - If the embedding was [Quantized](../../ai-integration/vector-search/vector-search-using-dynamic-query#what-is-quantization) by the task, |
196 |
| - the format includes the quantization type: |
| 207 | + If the embedding was [quantized](../../ai-integration/vector-search/vector-search-using-dynamic-query#what-is-quantization) by the task |
| 208 | + using a type other than _Single_ (e.g., _Int8_ or _Binary_), |
| 209 | + the ID format includes the quantization type: |
197 | 210 | `embeddings-cache/<connection-string-identifier>/<hash-of-text-chunk-from-property-or-search-term>/<quantization-type>`
|
198 | 211 |
|
199 | 212 | 2. **Expiration time**
|
|
205 | 218 |
|
206 | 219 | 
|
207 | 220 |
|
208 |
| - * In the cache collection, the attachment name consists only of the hash: |
| 221 | + * The name of the attachment is the hash string: |
209 | 222 | `<hash-of-text-chunk-from-property-or-search-term>`
|
210 | 223 |
|
211 | 224 | {PANEL/}
|
|
0 commit comments