Adding a merge_type parameter to the ingest simulate API #132210

masseyke · 2025-07-30T20:27:31Z

If mapping overrides are given to the ingest simulate API (_ingest/_simulate) in the mapping_addition, index_template_substitutions, or component_template_substitutions, they are currently merged in with existing mappings using MapperService.MergeReason.MAPPING_UPDATE when doing mapping validation. This simulates merging the mappings the way that they would be merged in to an existing index, rather than the way that they would be merged into templates. This can cause problems like #131608, where the mapping overrides cannot be merged in.
This PR adds a new merge_type parameter that allows a user to specify exactly how they would like the mappings merged. The options are index or template. By default, they will be merged using the index strategy to maintain backwards compatibility.
Closes #131608

elasticsearchmachine · 2025-07-30T20:27:55Z

Hi @masseyke, I've created a changelog YAML for you.

elasticsearchmachine · 2025-07-30T20:28:41Z

Hi @masseyke, I've updated the changelog YAML for you.

dakrone · 2025-07-30T21:36:14Z

Drive-by bikeshed :D — how about merge_type?

masseyke · 2025-07-31T21:02:11Z

Drive-by bikeshed :D — how about merge_type?

Think it's still clear that it refers only to mappings though? I didn't want to imply anything about the way settings or pipelines were merged.

elasticsearchmachine · 2025-07-31T21:03:41Z

Pinging @elastic/es-data-management (Team:Data Management)

Copilot

Pull Request Overview

This PR adds a new merge_type parameter to the ingest simulate API to control how mapping overrides are merged with existing mappings. The parameter accepts "index" or "template" values, allowing users to choose between merging mappings as they would be merged into an existing index versus how they would be merged into templates.

Key changes:

Added merge_type parameter to SimulateBulkRequest constructor and related API endpoints
Updated mapping validation logic to use the appropriate MapperService.MergeReason based on the merge type
Added comprehensive test coverage for the new functionality

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`SimulateBulkRequest.java`	Added `mappingMergeType` field and updated constructor to accept the new parameter
`RestSimulateIngestAction.java`	Added extraction of `merge_type` parameter from REST request
`TransportSimulateBulkAction.java`	Updated mapping validation to use the appropriate merge reason based on merge type
`TransportVersions.java`	Added new transport version for the feature
Test files	Updated all test constructors to include the new parameter
API spec	Added documentation for the new `merge_type` parameter
Integration tests	Added test cases to verify the new functionality works correctly

rest-api-spec/src/main/resources/rest-api-spec/api/simulate.ingest.json

server/src/main/java/org/elasticsearch/action/bulk/TransportSimulateBulkAction.java

Co-authored-by: Copilot <[email protected]>

seanzatzdev

lgtm

seanzatzdev · 2025-08-01T18:00:54Z

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -357,6 +357,7 @@ static TransportVersion def(int id) {
    public static final TransportVersion COMPONENT_TEMPLATE_TRACKING_INFO = def(9_132_0_00);
    public static final TransportVersion TO_CHILD_BLOCK_JOIN_QUERY = def(9_133_0_00);
    public static final TransportVersion ML_INFERENCE_AI21_COMPLETION_ADDED = def(9_134_0_00);
+    public static final TransportVersion SIMULATE_INGEST_MAPPING_MERGE_TYPE = def(9_135_0_00);


If i understand correctly, seems we need to update this version number due to the change in the API request? https://github.com/elastic/elasticsearch/blob/21e8bac36e84a73a8c3aa9740d4b92333edca7ef/docs/internal/Versioning.md#transport-protocol

Yeah we have to increment the transport version any time we change serialization.

samxbr · 2025-08-01T18:55:34Z

Drive-by bikeshed :D — how about merge_type?

Think it's still clear that it refers only to mappings though? I didn't want to imply anything about the way settings or pipelines were merged.

A bit late to the party, how about mapping_merge_type? Sounds more explicit on what is being merged.

samxbr

LGTM, just some minor comments.

samxbr · 2025-08-01T19:00:29Z

...all-dependencies/src/yamlRestTest/resources/rest-api-spec/test/ingest/80_ingest_simulate.yml

+            "mapping_addition": {
+              "properties": {
+                "a.b": {
+                  "type": "keyword"
+                }
+              }
+            }


Nit: Is it worth testing with mapping in index_template_substitutions or component_template_substitutions?

Yeah it probably is.

masseyke · 2025-08-01T21:53:23Z

Drive-by bikeshed :D — how about merge_type?

Think it's still clear that it refers only to mappings though? I didn't want to imply anything about the way settings or pipelines were merged.

A bit late to the party, how about mapping_merge_type? Sounds more explicit on what is being merged.

@dakrone any strong preference? Otherwise I'll probably just go with what I've currently got (merge_type).

dakrone · 2025-08-04T17:58:22Z

any strong preference? Otherwise I'll probably just go with what I've currently got (merge_type).

I don't have a strong preference, I think merge_type will be okay.

Adding a mapping_merge_reason parameter to the ingest simulate API

f7c2d7d

masseyke added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v9.2.0 labels Jul 30, 2025

Update docs/changelog/132210.yaml

2d3090d

Update docs/changelog/132210.yaml

5eef172

masseyke mentioned this pull request Jul 30, 2025

Fixing handling of subobject mappings in simulate ingest API #132046

Closed

masseyke added 4 commits July 30, 2025 15:51

Merge branch 'main' into ingest-simulate-mapping-merge-reason

9e41018

fixing a compilation error

e973ce6

fixing a compilation error

d7c3e30

removing forbidden code

5cce105

masseyke marked this pull request as ready for review July 31, 2025 21:03

Merge branch 'main' into ingest-simulate-mapping-merge-reason

0484138

elasticsearchmachine added the Team:Data Management Meta label for data/management team label Jul 31, 2025

masseyke added 2 commits August 1, 2025 09:55

changing parameter name to merge_type

791cdc9

Merge branch 'main' into ingest-simulate-mapping-merge-reason

40a8524

masseyke changed the title ~~Adding a mapping_merge_reason parameter to the ingest simulate API~~ Adding a merge_type parameter to the ingest simulate API Aug 1, 2025

masseyke requested a review from Copilot August 1, 2025 15:04

Copilot AI reviewed Aug 1, 2025

View reviewed changes

rest-api-spec/src/main/resources/rest-api-spec/api/simulate.ingest.json Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/action/bulk/TransportSimulateBulkAction.java Outdated Show resolved Hide resolved

masseyke and others added 3 commits August 1, 2025 10:06

Apply suggestions from code review

be00a10

Co-authored-by: Copilot <[email protected]>

[CI] Auto commit changes from spotless

b447813

Merge branch 'main' into ingest-simulate-mapping-merge-reason

0805181

seanzatzdev approved these changes Aug 1, 2025

View reviewed changes

samxbr reviewed Aug 1, 2025

View reviewed changes

masseyke added 2 commits August 1, 2025 15:00

adding a test for index_template_substitutions

55db16e

Merge branch 'main' into ingest-simulate-mapping-merge-reason

eda4247

Merge branch 'main' into ingest-simulate-mapping-merge-reason

82885d4

masseyke merged commit e223df1 into elastic:main Aug 11, 2025
33 checks passed

masseyke deleted the ingest-simulate-mapping-merge-reason branch August 11, 2025 17:00

This was referenced Aug 13, 2025

Adding the merge_type param to the ingest simulate API elastic/elasticsearch-specification#5139

Closed

Adding the merge_type param to the ingest simulate API elastic/elasticsearch-specification#5140

Open

Adding a merge_type parameter to the ingest simulate API #132210

Adding a merge_type parameter to the ingest simulate API #132210

Uh oh!

Conversation

masseyke commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 30, 2025

Uh oh!

elasticsearchmachine commented Jul 30, 2025

Uh oh!

dakrone commented Jul 30, 2025

Uh oh!

masseyke commented Jul 31, 2025

Uh oh!

elasticsearchmachine commented Jul 31, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

seanzatzdev left a comment

Choose a reason for hiding this comment

Uh oh!

seanzatzdev Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

samxbr commented Aug 1, 2025

Uh oh!

samxbr left a comment

Choose a reason for hiding this comment

Uh oh!

samxbr Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke commented Aug 1, 2025

Uh oh!

dakrone commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

masseyke commented Jul 30, 2025 •

edited

Loading