Index and search on HDFS log dataset

[fmassot]

Dataset description

Dataset containing 40 millions of hdfs log entries, available here.
Size: 13GB.

Example of a log entry formatted for quickwit

{
  "timestamp": 1460530013,
  "severity_text": "INFO",
  "body": "PacketResponder: BP-108841162-10.10.34.11-1440074360971:blk_1074072698_331874, type=HAS_DOWNSTREAM_IN_PIPELINE terminating",
  "resource": {
    "service": "datanode/01"
  },
  "attributes": {
    "class": "org.apache.hadoop.hdfs.server.datanode.DataNode"
  }
}

Indexing performance

Documents Read: 40069452 Parse Errors: 0 Published Splits: 9 Dataset Size: 14003MB Throughput: 45.32MB/s
Indexed 40069452 documents in 5.15min

Split generated: 8 with 5 million docs and 1 with 69452 docs on apple M1.

Typically, for 1 split, we have:

• .term: 11MB
• .fast: 13MB
• hotcache: 129MB
• .store: 413MB
• .idx: 108MB

Local search performance

TODO

Configuration

Doc mapper

{
    "store_source": true,
    "default_search_fields": ["body", "severity_text"],
    "timestamp_field": "timestamp",
    "field_mappings": [
        {
            "name": "timestamp",
            "type": "i64",
            "fast": true
        },
        {
            "name": "severity_text",
            "type": "text"
        },
        {
            "name": "body",
            "type": "text"
        },
        {
            "name": "resource",
            "type": "object",
            "field_mappings": [
                {
                    "name": "service",
                    "type": "text"
                }
            ]
        },
        {
            "name": "attributes",
            "type": "object",
            "field_mappings": [
                {
                    "name": "class",
                    "type": "text"
                }
            ]
        }
    ]
}