TAXII Collector bot and STIX Parser bot #2611
Conversation
As a bare minimum, TAXII Collector currently collects only the objects of type indicator. These objects contain information about indicators and the detection patterns, e.g. in stix, pcre, sigma, snort, suricata, yara format. The pattern, pattern_type and valid_from properties are required, while confidence, description and labels are only optional properties. However, they are present in several TAXII feeds and could be used to determine classification.taxonomy and classification.type even without processing the relationships of the indicators (e.g. indicator indicates malware) STIX Parser is currently capable of parsing objects of type indicator (usually retrieved from the TAXII Collector). From the indicator objects, it extracts the detection pattern (currently only the single Observation Expressions in STIX format are supported). It supports IP addresses, Domains and URLs indicator values. Moreover, this parser also attempts to extract some optional properties of STIX objects such as description and labels, which can be useful for futher classification of the event with the Expert Bots TAXII Collector tests for missing parameters and mock the simple TAXII server providing minimal collection with simple indicator object STIX Parser tests fo indicator patterns parsing Improvements based on @sebix comments, collection title used as feed.code Fix codestyle in TAXII and STIX bots Fix Python 3.8 support in STIX Parser bot
|
The TAXII and STIX bots are currently tested with the ESET Threat Intelligence (ETI) feeds. I am working on Expert Bot for classification events from ETI and I would like to publish it when it will be ready - together with feeds in |
|
Hello, I have a question regarding the proposal from the last commit. I created ESETExpertBot which can add the proper classification.type and malware.name (if possible) to the events produced by StixParserBot. Ref: https://github.com/laciKE/intelmq/blob/eset/intelmq/bots/experts/eset/expert.py When I wanted to add ESET Threat Intelligence TAXII feeds to feeds.yaml also with the expert bot, too, the tests failed, because it seems that the expert bot is not allowed in feeds.yaml. Especially with the TAXII feeds, three bots will be needed to ingest those feeds:
As far as I understand, two parsers cannot by chained in the pipeline (because the input is Report, and output is the Event). |
|
From what I understand, reading the code, the ESET expert fixes the classification for all events coming from the ESET feed. That logic should be in the Parser instead. Or is the code of ESET expert also useful for other sources other than ESET? |
|
Thank you for your answer. You are right, that expert bot fixes the classification and it is ESET-specific. I will change it to parser bot, which will inherit from the StixParserBot from this pull request. After that, I will add the commits with "EsetStixParserBot" to this pull request. |
|
Ah, I see. That parser also works for multiple sources, other than ESET? |
|
This StixParserBot yes, it should work for any source which provide Threat Intelligence data in STIX 2.1 format. I created it from scratch by reading STIX 2.1 documentation, and it is able to parse Indicators Objects with simple Patterns. StixParserBot (and TaxiiCollectorBot) should be used with any TAXII/STIX 2.1 feed. General parsing of indicators works, but for correct classification, the vendor-specific bot is needed. This is why I asked what is the proper way to do it. Currently I tested TaxiiCollectorBot+StixParserBot only with ESET Threat Intelligence TAXII feeds, because I do not have access to other TAXII 2.1 feeds. For correct classification, I created the ESETExpertBot, which I am going to change to ESETStixParserBot (it will by child a class of generic StixParserBot) |
Parser bot for enriching events from ESET Threat Intelligence, which were collected by TaxiiCollectorBot. It inherits from generic StixParserBot and implement vendor-specific parsing. ESET STIX Parser bot analyzes comment (based on original description of STIX Indicator object) and choose proper classification type and if possible, also fills the malware.name in the event.
ETI feeds with URLs, domains and IP addresses, which can be collected by TaxiiCollectorBot and parsed by ESETStixParserBot
laciKE
force-pushed
the
taxii
branch
2 times, most recently
from
0760f22 to
c177068
Compare
As a bare minimum, TAXII Collector currently collects only the objects of type indicator. These objects contain information about indicators and the detection patterns, e.g. in stix, pcre, sigma, snort, suricata, yara format. The pattern, pattern_type and valid_from properties are required, while confidence, description and labels are only optional properties. However, they are present in several TAXII feeds and could be used to determine classification.taxonomy and classification.type even without processing the relationships of the indicators (e.g. indicator indicates malware)
STIX Parser is currently capable of parsing objects of type indicator (usually retrieved from the TAXII Collector). From the indicator objects, it extracts the detection pattern (currently only the single Observation Expressions in STIX format are supported). It supports IP addresses, Domains and URLs indicator values. Moreover, this parser also attempts to extract some optional properties of STIX objects such as description and labels, which can be useful for futher classification of the event with the Expert Bots
TAXII Collector tests for missing parameters and mock the simple TAXII server providing minimal collection with simple indicator object STIX Parser tests fo indicator patterns parsing
Improvements based on @sebix comments, collection title used as feed.code Fix codestyle in TAXII and STIX bots
Fix Python 3.8 support in STIX Parser bot.