[FR] Add support for response_actions in rule validation

[DionAkkerman]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

The rule conversion and validation currently fails with an unknown field error for rules with "response_actions" set.

Desired Solution

Add support for the response_actions field and all subfields.

Considered Alternatives

No response

Additional Context

No response

[eric-forte-elastic]

Hi @DionAkkerman thanks for raising this issue! Could you provide an example of a rule where the error happens? When you say conversion is this from TOML -> JSON, JSON -> TOML?

Here is an example JSON that I am using to test. I do see that the content is not being parsed, but I am not seeing the validation error specifically. Thanks!

rules_export (37).ndjson.txt

[DionAkkerman]

Thanks for looking into it @eric-forte-elastic . I could have been a lot clearer ... we are experiencing the issue when converting from TOML to ndjson.

[metadata]
creation_date = "2025-01-01"
updated_date = "2025-01-01"

[rule]
description = """Example rule"""
from = "now-9m"
author = ["example"]
index = ["*"]
language = "eql"
name = "Example rule"
risk_score = 1
rule_id = "884e87cc-c67b-4c90-a4ed-e1e24a940c83"
severity = "low"
type = "eql"
query = '''
process where host.os.type == "Linux"
'''

[[rule.response_actions]]
action_type_id = ".osquery"
[rule.response_actions.params]


[[rule.response_actions]]
[rule.response_actions.params]
command = "kill-process"
comment = "This was a bad process."
action_type_id = ".endpoint"
[rule.response_actions.params.config]
field = ""
overwrite = true

The commands:
python -m detection_rules validate-rule ./custom_rules/custom_rule.toml
python -m detection_rules view-rule ./custom_rules/custom_rule.toml
python -m detection_rules export-rules-from-repo -f custom_rules/custom_rule.toml
all seem to be very unhappy with the response actions:

marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'response_actions': ['Unknown field.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.'], 'response_actions': ['Unknown field.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.'], 'response_actions': ['Unknown field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.'], 'response_actions': ['Unknown field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.'], 'query': ['Unknown field.'], 'response_actions': ['Unknown field.'], 'language': ['Unknown field.'], 'index': ['Unknown field.']}), ValidationError({'type': ['Must be equal to query.'], 'response_actions': ['Unknown field.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.'], 'response_actions': ['Unknown field.']})]}

Getting rid of the response actions "resolves" the issue which makes sense since I don't think the response_actions fields are defined in the Marshmallow schema (or at least I can't find them). It would be nice if the field was added to the schema and validated the subfields (e.g. .endpoint responses shouldn't be allowed on non-endpoint rules).