autodetect fails to restore categorizer

[valeriy42]

Recently we observe bursts for error with the following logs:

autodetect/283 process stopped unexpectedly: JSON state must be object at root10 Cannot restore categorizer - a element expected but found = JSON restore failed Failed to restore state ML anomaly detector job failed
[autodetect/192] [CCmdSkeleton.cc@49] Failed to restore state
[autodetect/192] [CJsonStateRestoreTraverser.cc@317] JSON state must be object at root10

It's unclear what causes this error. The first step is to enhance logging to get more information for further investigation.

[valeriy42]

I added #2867 for enhanced logging.

[valeriy42]

There was a new failure with the following log message:

autodetect/512 process stopped unexpectedly:
JSON state must be object at root. Found token type: array_end (value: 10), isArrayOfObjects: true, isEof: false, stream state - bad: false, fail: true, eof: true, bytes remaining: 0, buffer position: 2
Cannot restore categorizer - a element expected but found =
JSON restore failed
Failed to restore state
ML anomaly detector job failed