Best practices for use as benchmark on datasets with a train/test split

[xun468]

Hi! I'm new to using matrix profiles and would like to use them for anomaly detection, ideally on multivariate and single variate data but only single variate is also acceptable. What would you consider the best practice for evaluation on datasets designed for the standard machine learning training paradigm (train/test split, test set windows are each shown to the model independently etc etc)?

[seanlaw]

@xun468 Thank you for your question and welcome to the STUMPY community!

While matrix profile isn't a machine learning model, per se (it is really just a brute force pairwise Euclidean distance calculation), you may find an example use of matrix profiles for building a classifier here:

https://stumpy.readthedocs.io/en/latest/Tutorial_Shapelet_Discovery.html

[xun468]

Indeed! I am currently working on a ML anomaly detection paper and was hoping to use matrix profile as one of the baseline approaches, I am hoping to avoid accidentally "kneecapping" the method unfairly by using it inappropriately or unusually.

Would you consider for example, using matrix profile on the test set only to be unfair? Since the test sets are usually not guaranteed to a continuous extension from the train set, I believe effectively concatenating them using stumpi is not right either (if I understood how stumpi works correctly that is!)

[seanlaw]

I am currently working on a ML anomaly detection paper and was hoping to use matrix profile as one of the baseline approaches, I am hoping to avoid accidentally "kneecapping" the method unfairly by using it inappropriately or unusually.

For anomaly detection, the original matrix profile researchers have developed MADRID algorithm but it isn't implemented in STUMPY yet (hopefully, soon). You may also be interested in a related paper on anomaly detection by the same group.

Thus, I would tend to agree with you that, unless you are comparing to MADRID, it's probably unfair to compare basic matrix profile computation to other models. It really depends on what your model is looking for. Z-normalized matrix profiles can perform well for subsequence shapes and non-normalized matrix profiles can perform well for time series with spikes. The (second) related paper above really emphasizes how (time series) anomaly detection metrics tend to be flawed and I'm inclined to agree.

Would you consider for example, using matrix profile on the test set only to be unfair? Since the test sets are usually not guaranteed to a continuous extension from the train set, I believe effectively concatenating them using stumpi is not right either (if I understood how stumpi works correctly that is!)

I think your intuition is probably correct and you're probably better off comparing your model with a different approach. Again, it really depends on what your data looks like. There's no one-size-fits-all solution