crowdstrike: split fdr data stream into three #16211
Description
The plan for scaling CrowdStrike metadata enrichment is to use LOOKUP JOIN, first splitting the current fdr data stream into three: fdr, fdr_aidmaster and fdr_userinfo.
The plan for this is to use routing rules to perform the split based on the log.file.path field. The fdr data would be processed essentially as the data stream currently is, and the other two data streams (fdr_aidmaster and fdr_userinfo) would be passed through largely unaltered (exact changes to be determined).
Initially I think rerouting should be configurable, default: "off" so that this is not a breaking change. This would allow us to roll out the code changes without impacting users in any way. The rerouting configuration would need to be considered in the agent cache-based enrichment logic; if rerouting is turned on, we would be expecting that the user would be doing enrichment downstream with LOOKUP JOIN, so enrichment in the agent seems like pointless work, though this may not be the case and this should be discussed further.