Repository navigation
re-serialize JSON metadata written into the ES and OpenSearch _bulk body - #3603
Open
mrdevel2002-ui wants to merge 1 commit into
Open
mrdevel2002-ui wants to merge 1 commit into
mrdevel2002-ui wants to merge 1 commit into
Conversation
The OpenSearch and Elasticsearch emitters wrote JSON-typed metadata values such as the reserved tk:chunks key into the NDJSON _bulk body with writeRawValue(vals[0]). That value is attacker influenced: a tuple's metadata block submitted to /pipes or /async reaches the emitter unescaped. OpenSearch's isValidJson used readTree, which ignores content after the first value, so a value like [1]} followed by newline-separated bulk actions closed the source document early and injected arbitrary delete/index actions against any index the credentials allow. ES was already strict about trailing content but still passed the value raw, so an embedded newline split the NDJSON line there too. Make OpenSearch strict like ES, and in both clients re-serialize the parsed node so trailing content is dropped and insignificant whitespace is stripped; a valid chunks array still indexes natively. Generated-by: Claude Code (claude-opus-4-8)
Contributor
|
Can you have your bot tone down the description on this issue and going forward? This isn't a security issue. Your bot isn't saying it is, but it sure reads like one, and someone not paying attention will think this is a security issue. This makes sense. Thank you for filing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OpenSearch and Elasticsearch emitters write JSON-typed metadata values (the reserved
tk:chunksvector/locator field in particular) into the NDJSON_bulkrequest body withJsonGenerator.writeRawValue(vals[0]). That value is attacker influenced: aFetchEmitTuple'smetadatablock submitted to tika-server/pipesor/asyncreaches the emitter unescaped, becauseMetadata.reconstructwrites the reserved key throughsetTrustedandEmitHandler.injectUserMetadatacopies it onto the emitted document, andtk:chunksis declaredValueType.JSONsoisJsonFieldroutes it to the raw-write path.OpenSearchClient.isValidJsongated the value withObjectMapper.readTree, which stops at the first JSON value and silently ignores anything after it. Atk:chunksstring of[1]}followed by newline-separated bulk actions therefore passes the check, andwriteRawValueemits it verbatim, so the bulk body becomes:The
}closes the source document early and the embedded newlines add adeleteand anindexaction; the injected actions can name any_index, so one tuple reaches indices the operator never configured this emitter for (bounded only by the cluster'srest.action.multi.allow_explicit_index).ESClient.isValidJsonwas already strict about trailing content (testTrailingContentRejected), so Elasticsearch is not open to that clean injection, but both clients still passed the value through raw, so an embedded newline in an otherwise valid value splits the NDJSON line and corrupts or injects actions there too.The fix brings OpenSearch's
isValidJsonin line with the strict Elasticsearch one and, in both clients, re-serializes the parsed node (readTree(vals[0]).toString()) instead of writing the original string. Re-serialization drops anything after the first value and strips insignificant whitespace, so a crafted value can no longer add or split_bulkaction lines; a legitimate chunks array still indexes natively, since vectors are base64 text and locators round-trip unchanged. The fix lives in the emitter because that is the one place the untrusted value crosses into the bulk protocol.testChunksCannotInjectBulkActions(OpenSearch) andtestChunksWithNewlinesStayOnOneLine(ES) fail onmainand pass with the change. Fulltika-pipes-opensearchandtika-pipes-esbuilds (tests, checkstyle, spotless, forbidden-apis, apache-rat) are green on JDK 21.TIKA-XXXX) in the PR title and commit message: pending the issue above./mvnw clean install(affected modules)mainbranchtika-bom/pom.xml: not applicable