ElasticSearch doSimilaritySearch broken after recent changes in Document

[jcgouveia]

After update to the most recent version the similarity search gives an exception because the Document is no longer compatible to the data return by the search hits

1. Document (org.springframework.ai.document.Document) is directly serialized from the ES response using standard jackson map to object serialization

doSimilaritySearch(SearchRequest searchRequest)
...
			SearchResponse<Document> res = this.elasticsearchClient.search(
					sr -> sr.index(this.options.getIndexName())
						.knn(knn -> knn.queryVector(EmbeddingUtils.toList(vectors))
							.similarity(finalThreshold)
							.k((long) searchRequest.getTopK())
							.field("embedding")
							.numCandidates((long) (1.5 * searchRequest.getTopK()))
							.filter(fl -> fl.queryString(
									qs -> qs.query(getElasticsearchQueryString(searchRequest.getFilterExpression()))))),
			---->		Document.class);

			return res.hits().hits().stream().map(this::toDocument).collect(Collectors.toList());

2. The previous Document class was compatible with ES responses

"_source":{
"embedding":[…],
"content": "...",
"media":[],
"metadata":{ .... }

Now, it isn´t anymore in the Document class, after #1794.
context has been renamed to text, media is no longer an array

3. Therefore I get the error

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `org.springframework.ai.model.Media` from Array value (token `JsonToken.START_ARRAY`)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 8428] (through reference chain: org.springframework.ai.document.Document["media"])

I'm not able to use ElasticSearch queries through the Vector Store.

[jcgouveia]

@ThomasVitale , your change in Document broke the ElasticSearch doSimilaritySearch method because the Document class was used directly for deserializing responses and it is not compatible anymore (see above). I think a compatible ESDocument should be used and then mapped to the new one in the toDocument method.
Why didn't anyone address this issue ? It´s not causing problems to other users of ElasticSearch queries ?

[ThomasVitale]

@jcgouveia thanks for reporting this issue. The change of mine you referenced (#1794) introduces a new score field in the Document class, which maintains the backward compatibility for the Elasticsearch vector store integration.

The Jackson exception you got seems to be due to a different change that happened in the Document class: the type of the media field was changed from a Collection<Media> to a Media (in #1883). During deserialisation, Jackson fails turning a media JSON array into a something that is not an array/collection.

[jcgouveia]

You're right @ThomasVitale , sorry for the mistake. Indeed the breaking change was introduced by @markpollack change you refer. But this is preventing me to move on to M5 or the SNAPSHOT version. How can I proceed? Should I solve the situation and ask for a pull request, or should it be done by the original developpers ?

[jcgouveia]

@markpollack , any ideia when this could be solved ? I cannot use ElasticSearch queries with the current version.

[stillmoon]

I'm having the same problem. Still no solution?

[dev-jonghoonpark]

@jcgouveia @stillmoon
Could you provide a reproducible example repository?
This would help others to better understand and fix the problem.

[jcgouveia]

@dev-jonghoonpark, @ThomasVitale

I don´t see where is the dificulty in understading the problem. The read and write in the elastic search store has to be symetric. The fields stored in the fragment source has to match. That should be assured by using the ElasticSearchDocument that represents the schema to store, as you can see in the doAdd method with the line

ElasticSearchDocument doc = new ElasticSearchDocument(document.getId(), document.getText(),

	public void doAdd(List<Document> documents) {
		// For the index to be present, either it must be pre-created or set the
		// initializeSchema to true.
		if (!indexExists()) {
			throw new IllegalArgumentException("Index not found");
		}
		BulkRequest.Builder bulkRequestBuilder = new BulkRequest.Builder();

		List<float[]> embeddings = this.embeddingModel.embed(documents, EmbeddingOptionsBuilder.builder().build(),
				this.batchingStrategy);

		for (Document document : documents) {
===>		ElasticSearchDocument doc = new ElasticSearchDocument(document.getId(), document.getText(),
===>					document.getMetadata(), embeddings.get(documents.indexOf(document)));
			bulkRequestBuilder.operations(
					op -> op.index(idx -> idx.index(this.options.getIndexName()).id(document.getId()).document(doc)));
		}

Therefore, the read must use the same Document. Up to the breaking change they were compatible, but not anymore (context --> text)!
The document to parameterize the search must be the same, ElasticSearchDocument and not Document, in doSimilaritySearch

SearchResponse<ElasticSearchDocument > res = this.elasticsearchClient.search(sr -> sr.index(this.options.getIndexName())
...
.size(searchRequest.getTopK()), ElasticSearchDocument.class);

I solved the problem building my own extension of ElasticsearchVectorStore, with this modification, and a new toDocument to convert between those documents. It is working correctly as expected, reading and writing (id, content)

	private Document toDocument(Hit<ElasticSearchDocument> hit) {

		ElasticSearchDocument doc = hit.source();
	
		Document.Builder documentBuilder = Document.builder()
				.id(doc.id())
				.text(doc.content())
				.metadata(doc.metadata());
		
		//media does not exist ElasticSearchDocument !
		/*
		if (doc.media() != null && !doc.media().isEmpty()) {
			documentBuilder.media(doc.media.get(0));
		}
		*/
				
		if (hit.score() != null) {
			documentBuilder.metadata(DocumentMetadata.DISTANCE.value(), 1 - normalizeSimilarityScore(hit.score()));
			documentBuilder.score(normalizeSimilarityScore(hit.score()));
		}
		return documentBuilder.build();
	}

Therefore,

• Altough I used an extension, this change has to be made in the ElasticsearchVectorStore
• The extension of the store is very hard and cumbersome because a lot of methods are private and not protected wich requires too many copies on the subclasss (eg normalizeSimilarityScore) (why not protected as default ???)

Notes:

• The problem was not in media, but in the change from context to text
• The media does not exist in ElasticSearchDocument, I don´t know if its relevant to have and consider in the toDocument method. I was using some code for that because I was using first my own version of ESDocument and my store had the media field stored as an array (but in fact empty)

[jcgouveia]

@dev-jonghoonpark ,

I noticed that in your last commit you removed ElasticSearchDocument in order to manipulate the fields as a map to change the embedding field name. That will make the toDocument harder to implment. I did that at first : use Object as the parameterized type in the search, and then toDocument receives a map of values, but the the solution is not strongly typed and its based on strings :-(

So I think the ElasticSearchDocument should be reintroduced (at least only for reading), if the configurable name for embeddings field cannot be solved by any Spring technique (like a dynamic json annotation or mixin) or in the Jackson mapper.

[dev-jonghoonpark]

@jcgouveia
I modified it so that custom embedding is possible while still using record.
Does this approach align with what you had in mind?

#2651

[jcgouveia]

@dev-jonghoonpark ,
Going back to the record Document helps solving the problem. But its still needed to:

1. Use that document when getting the search hits

SearchResponse<ElasticSearchDocument > res = this.elasticsearchClient.search(sr -> sr.index(this.options.getIndexName())
...
.size(searchRequest.getTopK()), ElasticSearchDocument.class);

2. Change the toDocument function to receive the correct type

private Document toDocument(Hit<ElasticSearchDocument> hit) {

and build the base Spring Document from the ElasticSearchDocument, similar to what I put in previous comment

[dev-jonghoonpark]

So just to confirm, is the issue that data saved using an old version can't be loaded with the new version?

Also, does the same problem occur even when data is saved and loaded using only the new version?

[jcgouveia]

Both cases, because it is saved with "content" but is read to "text" if serialised into the new version of the spring Document, since the breaking change is in that field.
To make sure it works the Document type should be the same (the record ElasticSearchDocument)

[dev-jonghoonpark]

For the first case
migrating the data seems like a reasonable solution. Have you considered data migration as an option?

For the second case
could you provide an example of code where an error would occur?

As you mentioned, the data is stored with the content field, but looking at the Document class:

@JsonCreator(mode = JsonCreator.Mode.PROPERTIES)
public Document(@JsonProperty("content") String content) {
    this(content, new HashMap<>());
}

This constructor ensures that content is properly converted to text, so everything seems to be functioning as expected in the current version.

[jcgouveia]

I was indeed testing on version M6, where the bug exists.
In the develop/snapshot with the definition you pointed on Document in fact now it works because context => text conversion is ocurring. 👍

But data miration is something that will be necessary, for those we have ES indexes created with previous versions, because in that case the exception is relatede with the Media field (was stored as an array, now its a single field)
I will have to create some procedures for that migration in my environment.