Skip to content

[Streams] Propagate the ingest pipeline access pattern flag to the ingest document #130488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

jbaiera
Copy link
Member

@jbaiera jbaiera commented Jul 2, 2025

Adds the ability for an ingest document to capture the pipeline's access pattern flag when executing the pipeline on the document. The document stores the access pattern flag in a stack to allow for holding and restoring the access pattern flag when running nested pipelines.

@jbaiera jbaiera requested review from masseyke and lukewhiting July 2, 2025 19:23
@jbaiera jbaiera added >non-issue :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v9.2.0 labels Jul 2, 2025
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Jul 2, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

/**
* @return The access pattern for any currently executing pipelines, or null if no pipelines are in progress for this doc
*/
public IngestPipelineFieldAccessPattern getCurrentAccessPattern() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional that this method isn't used above (and seems to only be used in the test)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not used above yet. I'm currently using it in a couple places in the next set of changes I'm working on.

Copy link
Member

@masseyke masseyke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lukewhiting lukewhiting requested a review from Copilot July 3, 2025 07:56
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces tracking of each pipeline’s access pattern within nested pipeline executions by maintaining a stack on the ingest document. Key changes include:

  • Add accessPatternStack field and related push/pop logic in IngestDocument
  • Update Pipeline class to allow mocking by removing final
  • Introduce a comprehensive nested-access-pattern test in IngestDocumentTests

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
server/src/main/java/org/elasticsearch/ingest/IngestDocument.java Added accessPatternStack, push/pop logic, and getCurrentAccessPattern()
server/src/main/java/org/elasticsearch/ingest/Pipeline.java Removed final modifier from Pipeline for testability
server/src/test/java/org/elasticsearch/ingest/IngestDocumentTests.java Added testNestedAccessPatternPropagation with Mockito captures
Comments suppressed due to low confidence (2)

server/src/main/java/org/elasticsearch/ingest/Pipeline.java:28

  • [nitpick] Removing final from Pipeline makes it extendable, which may not be intended for core pipeline behavior; verify that this aligns with the project's API design.
public class Pipeline {

server/src/test/java/org/elasticsearch/ingest/IngestDocumentTests.java:1273

  • The test uses is(empty()) but is is not statically imported; add import static org.hamcrest.Matchers.is; to avoid a compilation error.
        assertThat(document.getPipelineStack(), is(empty()));

Copy link
Contributor

@lukewhiting lukewhiting left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor niggles from CoPilot but otherwise LGTM 👍🏻

@jbaiera jbaiera merged commit 3345062 into elastic:main Aug 12, 2025
33 checks passed
@jbaiera jbaiera deleted the streams-ingest-pipeline-field-access-propagated branch August 12, 2025 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >non-issue Team:Data Management Meta label for data/management team v9.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants