How enforce SequenceModel to be causal

[jaak-s]

Hi, I am trying out your model for timeseries problems from the Informer paper, e.g. experiment=s4-informer-ecl. And it looks like the SequenceModel is not causal, meaning that the future values of x affect the output of the model at time 0. This means it is not a filtering, in the terminology of signal processing.

For reference, this is the command I use.

python3 -m train wandb=null experiment=s4-informer-ecl model.layer.liquid_kernel=polyb

I would be interested to make the model causal (i.e., a filtering). Is that possible?

PS to test for the causal property, I ran the SequenceModel with two sequences in eval mode, where the first 40 steps had the same data and after 40 steps had different values. If the model is causal then the output for the first 40 steps for the two sequences must be equal (at numerical noise level, e.g.,. 1e-5) but the norm of the difference was quite far from 0, around 0.01 to 0.1. And the bigger I make the difference in the later values (40+ steps) between the two sequences the bigger difference I get in the first 40 steps. Clearly the default model has access to the future values.

[mlech26l]

Hi @jaak-s

Yes, this implementation of the liquid-S4 is noncausal, i.e., it can only be used to process entire sequence classification/regression.
I think the polyb kernel might be made causal by removing the flip operation in https://github.com/raminmh/liquid-s4/blob/main/src/models/sequence/ss/s4.py#L326, but I am not 100% sure there is no other dependency on future values.