Ingest simulate: mapping_addition doesn't work for subobjects

[flash1293]

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

macos 15.5

Problem Description

When using the ingest simulate API with mapping additions, the mapping additions are not applied correctly to sub-objects - see steps to reproduce

Steps to Reproduce

Assume an index template like this:

PUT _index_template/myt
{
  "index_patterns": [
    "mytest-*"
  ],
  "composed_of": [],
  "template": {
    "mappings": {
      "properties": {
        "a.b": {
          "type": "match_only_text"
        }
      }
    }
  }
}

I now want to simulate changing a.b. to keyword:

POST _ingest/_simulate
{
  "docs": [
    {
      "_index": "mytest-default",
      "_id": "AZgsHA0B41JjTOmNiBKC",
      "_source": {
        "a.b": "error test"
      }
    }
  ],
  "mapping_addition": {
    "properties": {
      "a.b": {
        "type": "keyword"
      }
    }
  }
}

However, this fails with

{
          "type": "mapper_parsing_exception",
          "reason": "Failed to parse mapping: mapper [a.b] cannot be changed from type [match_only_text] to [keyword]",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "mapper [a.b] cannot be changed from type [match_only_text] to [keyword]"
          }
        }

If I'm mapping ab instead of a.b, everything works as expected.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gbanasiak]

@flash1293 The relevant team will look into this, but the error makes sense to me. The a.b is defined as match_only_text in the template, so mapping addition of keyword conflicts with that. The difference with ab is it's not defined in the template, so any mapping addition is allowed. What would be the expected behavior for a.b you think?

[flash1293]

@gbanasiak What I meant is that this works:

PUT _index_template/myt
{
  "index_patterns": [
    "mytest-*"
  ],
  "composed_of": [],
  "template": {
    "mappings": {
      "properties": {
        "ab": {
          "type": "match_only_text"
        }
      }
    }
  }
}

POST _ingest/_simulate
{
  "docs": [
    {
      "_index": "mytest-default",
      "_id": "AZgsHA0B41JjTOmNiBKC",
      "_source": {
        "ab": "error test"
      }
    }
  ],
  "mapping_addition": {
    "properties": {
      "ab": {
        "type": "keyword"
      }
    }
  }
}

Replacing a.b with ab everywhere (also in the original index template). It seems like the difference is whether the field is defined on the root level or in a sub-object.

What would be the expected behavior for a.b you think?

It should behave as if I changed the mapping of a.b on the index directly - this is also what happens if I use component_template_substitutions for example. It shows me what would happen if I would change the mapping like this and did a rollover.

You could argue that mapping additions only look at the index, not the data streams (which is a lame excuse in the first place), but even then it should be the same error for the ab case.

[gbanasiak]

I see. Can you try with a.b defined as follows?

"properties": {
  "a": {
    "type": "object",
    "properties": {
      "b": {
        "type": "match_only_text"
      }
    }
  }
}

[flash1293]

@gbanasiak I can confirm it does work in that case. Another workaround that works is to use index_template_substitutions and add it there (it works there even in the dotted field style).

[gbanasiak]

Looking at #114742 mapping_addition is expected to take precedence over index mappings, so my intuition from #131608 (comment) does not work here.

[flash1293]

Just as a note - the workaround with "expanding" the object via

"properties": {
  "a": {
    "type": "object",
    "properties": {
      "b": {
        "type": "match_only_text"
      }
    }
  }
}

might work, but is impractical for my use case because it's hard to tell whether this should be done or not - there are other cases (subobjects: false), where doing so would break it.

[tonyghiani]

As a side task for this issue, we also encountered the following behaviour when using the ingest simulation API.

Ingest simulate doesn't consider existing mappings if not specified in the mapping_addition

Ideally, we should not need to retrieve and merge the mappings from the stream index .
However, the ingest simulation API does not validate the mappings correctly unless they are specified in the simulation body.
So we need to merge the mappings from the existing index with the additional new mappings.
This is a workaround we adopted:

    mapping_addition: {
      ...mappings,
      properties: {
        ...(mappings && mappings.properties),
        ...newMappings
      },
    },

Ideally, the ingest simulation API should consider the existing mapping during the ingestion simulation, and just merge the custom mapping addition for the simulation purpose with no need to first retrieve the existing ones.

Replication steps

1. Add a document to a new data stream

POST /logs-dataset-generic/_doc
{
    "num": 3
}

2. Confirm the data stream mapping for num is set to long.

GET logs-dataset-generic/_mapping

# Returns
{
  ".ds-logs-dataset-generic-2025.07.25-000001": {
    "mappings": {
      // Other config...
      "properties": {
        // Other properties...
        "num": {
          "type": "long"
        }
      }
    }
  }
}

3. Try the simulation with a wrong value for num and specifying a new simulated field

POST /_ingest/_simulate
{
  "docs": [
    {
      "_id": "test-id",
      "_index": "logs-dataset-generic",
      "_source": {
        "@timestamp": "2025-07-25T09:06:06.929Z",
        "is_valid": true,
        "num": true
      }
    }
  ],
  "mapping_addition": {
    "properties": {
      "is_valid": {
        "type": "boolean"
      }
    }
  }
}

# This should catch the mapping error, but instead proceeds successfully, returning
{
  "docs": [
    {
      "doc": {
        "_id": "test-id",
        "_index": "logs-dataset-generic",
        "_version": -3,
        "_source": {
          "@timestamp": "2025-07-25T09:06:06.929Z",
          "is_valid": true,
          "num": true
        },
        "executed_pipelines": [
          "logs@default-pipeline"
        ]
      }
    }
  ]
}

4. Given the above doesn't validate the error on num, try the same specifying the mapping addition for num with the original mapping

POST /_ingest/_simulate
{
  "docs": [
    {
      "_id": "test-id",
      "_index": "logs-dataset-generic",
      "_source": {
        "@timestamp": "2025-07-25T09:06:06.929Z",
        "is_valid": true,
        "num": true
      }
    }
  ],
  "mapping_addition": {
    "properties": {
      "is_valid": {
        "type": "boolean"
      },
      "num": {
        "type": "long"
      }
    }
  }
}

# The above fails correctly when `num` is simulated with a mismatching type from the original mapping, returning the error
{
  "docs": [
    {
      "doc": {
        "_id": "test-id",
        "_index": "logs-dataset-generic",
        "_version": -3,
        "_source": {
          "@timestamp": "2025-07-25T09:06:06.929Z",
          "is_valid": true,
          "num": true
        },
        "executed_pipelines": [
          "logs@default-pipeline"
        ],
        "error": {
          "type": "document_parsing_exception",
          "reason": "[1:64] failed to parse field [num] of type [long] in document with id 'test-id'. Preview of field's value: 'true'",
          "caused_by": {
            "type": "x_content_parse_exception",
            "reason": """[1:68] Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
 at [Source: (byte[])"{"@timestamp":"2025-07-25T09:06:06.929Z","is_valid":true,"num":true}"; line: 1, column: 68]""",
            "caused_by": {
              "type": "json_parse_exception",
              "reason": """Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
 at [Source: (byte[])"{"@timestamp":"2025-07-25T09:06:06.929Z","is_valid":true,"num":true}"; line: 1, column: 68]"""
            }
          }
        }
      }
    }
  ]
}

What happens in step 3 seems wrong, as the simulation should report the error on the mismatching num even if not specified in mapping_addition.

[masseyke]

The reason for the behavior in the comment just above is that simulate ingest falls back to templates if you pass in any mapping substitutions (mapping_addition, component_template_substitutions, or index_template_substitutions). The code is https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/bulk/TransportSimulateBulkAction.java#L201-L204. That choice makes sense for component_template_substitutions and index_template_substitutions. Maybe it doesn't for mapping_addition though. I think it's a separate issue from the one this ticket describes though.