Support for nomic-embed-vision-v1 and nomic-embed-vision-v1.5

[simonw]

https://www.nomic.ai/blog/posts/nomic-embed-vision

https://docs.nomic.ai/reference/api/embed-image-v-1-embedding-image-post

[simonw]

Their documentation is missing crucial details for getting this to work. After much experimentation the recipe is this:

# items is an array of bytes, each one representing an image
response = httpx.post(
    "https://api-atlas.nomic.ai/v1/embedding/image",
    headers={"Authorization": f"Bearer {key}"},
    data={"model": "nomic-embed-vision-v1.5"},
    files=[("images", item) for item in items],
)

Crucial detail is that you use a multipart/form-data submission and send images as the name of the form field that contains the binary file data - and you can send images multiple times.

[simonw]

Tested like this:

llm embed-multi -d emb.db images --files ../llm '*.png' --binary --model nomic-embed-vision-v1.5

[simonw]

Testing this was hard due to this bug:

• [BUG] HTTPX Binary upload data unsuported kevin1024/vcrpy#656

I applied a nasty monkeypatch workaround based on this:

• fix(stubs/httpx): fix decode error on utf-8 response with file kevin1024/vcrpy#820