Skip to content

Commit

Permalink
update visuals and readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@khairulislam
khairulislam committed Dec 5, 2024
1 parent f99889e commit 58526b5
Show file tree
Hide file tree
Showing 11 changed files with 186 additions and 121 deletions.
21 changes: 11 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ TSlib is an open-source library for deep learning researchers, especially deep t
## Interpretation Methods

The following local intepretation methods are supported at present:
<details>
<details open>

1. *WinTSR* - proposed new method
2. *Feature Ablation* [[2017]](https://arxiv.org/abs/1705.08498)
Expand All @@ -37,14 +37,23 @@ The following local intepretation methods are supported at present:

</details>

## Foundation Models

We support the following time series LLM models

1. GPT4TS - [One Fits All (OFA) : Power General Time Series Analysis by Pretrained LM](https://arxiv.org/abs/2302.11939) (NeurIPS 2023 Spotlight)
2. CALF - [CALF - Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning.](https://arxiv.org/abs/2403.07300) (Under review 2024)
3. TimeLLM - [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://arxiv.org/pdf/2310.01728) (ICLR 2024)

## Time Series Models

This repository currently supports the following models:

<details>
<details open>

- [x] **TimeMixer** - TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting [[ICLR 2024]](https://openreview.net/pdf?id=7oLshfEIC2) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/TimeMixer.py).
- [x] **TSMixer** - TSMixer: An All-MLP Architecture for Time Series Forecasting [[arXiv 2023]](https://arxiv.org/pdf/2303.06053.pdf) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/TSMixer.py)
- [x] **iTransformer** - iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [[ICLR 2024]](https://arxiv.org/abs/2310.06625) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/iTransformer.py).
- [x] **TimesNet** - TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis [[ICLR 2023]](https://openreview.net/pdf?id=ju_Uqw384Oq) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/TimesNet.py)
- [x] **DLinear** - Are Transformers Effective for Time Series Forecasting? [[AAAI 2023]](https://arxiv.org/pdf/2205.13504.pdf) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/DLinear.py)
- [x] **PatchTST** - A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. [[ICLR 2023]](https://openreview.net/pdf?id=Jbdc0vTOcol) [[Code]](https://github.com/thuml/Time-Series-Library/blob/main/models/PatchTST.py)
Expand All @@ -67,14 +76,6 @@ This repository currently supports the following models:

</details>

## Foundation Models

We also support the following time series LLM models

1. GPT4TS - [One Fits All (OFA) : Power General Time Series Analysis by Pretrained LM](https://arxiv.org/abs/2302.11939) (NeurIPS 2023 Spotlight)
2. CALF - [CALF - Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning.](https://arxiv.org/abs/2403.07300) (Under review 2024)
3. TimeLLM - [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://arxiv.org/pdf/2310.01728) (ICLR 2024)

## Train & Test

Use the [run.py](/run.py) script to train and test the time series models. Check the [scripts](/scripts/) and [slurm](/slurm/) folder to see sample scripts. Make sure you have the datasets downloaded in the `dataset` folder following the `Datasets` section. Following is a sample code to train the electricity dataset using the DLinear model. To test an already trained model, just remove the `--train` parameter.
Expand Down
78 changes: 34 additions & 44 deletions dual_interpret.ipynb
View file

Large diffs are not rendered by default.

Binary file modified results/electricity_iTransformer/1/heatmap_2587_normed.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified results/mimic_iii_iTransformer/1/heatmap_2_normed.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
72 changes: 72 additions & 0 deletions results/results.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,18 @@ electricity,dyna_mask,mae,iTransformer,3,0.05,2.626,21.176
electricity,dyna_mask,mae,iTransformer,3,0.075,3.553,20.82
electricity,dyna_mask,mae,iTransformer,3,0.1,4.249,20.59
electricity,dyna_mask,mae,iTransformer,3,0.15,5.52,19.972
electricity,dyna_mask,mae,CALF,1,0.05,2.239,21.033
electricity,dyna_mask,mae,CALF,1,0.075,3.433,20.436
electricity,dyna_mask,mae,CALF,1,0.1,4.289,20.0
electricity,dyna_mask,mae,CALF,1,0.15,6.432,18.841
electricity,dyna_mask,mae,CALF,2,0.05,2.249,21.681
electricity,dyna_mask,mae,CALF,2,0.075,3.542,21.068
electricity,dyna_mask,mae,CALF,2,0.1,4.374,20.606
electricity,dyna_mask,mae,CALF,2,0.15,6.737,19.301
electricity,dyna_mask,mae,CALF,3,0.05,2.188,20.922
electricity,dyna_mask,mae,CALF,3,0.075,3.372,20.413
electricity,dyna_mask,mae,CALF,3,0.1,4.133,20.037
electricity,dyna_mask,mae,CALF,3,0.15,6.192,18.86
electricity,extremal_mask,mae,DLinear,1,0.05,5.577,15.355
electricity,extremal_mask,mae,DLinear,1,0.075,8.154,14.1
electricity,extremal_mask,mae,DLinear,1,0.1,9.559,13.436
Expand Down Expand Up @@ -395,6 +407,18 @@ electricity,extremal_mask,mae,iTransformer,3,0.05,3.121,20.505
electricity,extremal_mask,mae,iTransformer,3,0.075,4.693,19.69
electricity,extremal_mask,mae,iTransformer,3,0.1,5.829,19.185
electricity,extremal_mask,mae,iTransformer,3,0.15,7.906,18.028
electricity,extremal_mask,mae,CALF,1,0.05,6.242,18.809
electricity,extremal_mask,mae,CALF,1,0.075,7.859,18.071
electricity,extremal_mask,mae,CALF,1,0.1,8.648,17.651
electricity,extremal_mask,mae,CALF,1,0.15,10.159,16.691
electricity,extremal_mask,mae,CALF,2,0.05,6.603,19.284
electricity,extremal_mask,mae,CALF,2,0.075,8.181,18.615
electricity,extremal_mask,mae,CALF,2,0.1,8.935,18.138
electricity,extremal_mask,mae,CALF,2,0.15,10.39,17.063
electricity,extremal_mask,mae,CALF,3,0.05,6.152,18.833
electricity,extremal_mask,mae,CALF,3,0.075,7.85,18.082
electricity,extremal_mask,mae,CALF,3,0.1,8.618,17.63
electricity,extremal_mask,mae,CALF,3,0.15,10.064,16.632
electricity,winIT,mae,DLinear,1,0.05,10.058,14.177
electricity,winIT,mae,DLinear,1,0.075,10.479,13.783
electricity,winIT,mae,DLinear,1,0.1,10.653,13.55
Expand Down Expand Up @@ -971,6 +995,18 @@ traffic,dyna_mask,mae,iTransformer,3,0.05,2.61,29.46
traffic,dyna_mask,mae,iTransformer,3,0.075,3.418,28.629
traffic,dyna_mask,mae,iTransformer,3,0.1,3.956,28.022
traffic,dyna_mask,mae,iTransformer,3,0.15,5.037,26.397
traffic,dyna_mask,mae,CALF,1,0.05,2.283,29.43
traffic,dyna_mask,mae,CALF,1,0.075,3.543,28.415
traffic,dyna_mask,mae,CALF,1,0.1,4.367,27.676
traffic,dyna_mask,mae,CALF,1,0.15,6.439,25.783
traffic,dyna_mask,mae,CALF,2,0.05,2.159,29.73
traffic,dyna_mask,mae,CALF,2,0.075,3.394,28.69
traffic,dyna_mask,mae,CALF,2,0.1,4.214,27.975
traffic,dyna_mask,mae,CALF,2,0.15,6.216,26.175
traffic,dyna_mask,mae,CALF,3,0.05,2.045,30.389
traffic,dyna_mask,mae,CALF,3,0.075,3.188,29.108
traffic,dyna_mask,mae,CALF,3,0.1,3.954,28.183
traffic,dyna_mask,mae,CALF,3,0.15,5.954,25.861
traffic,extremal_mask,mae,DLinear,1,0.05,3.195,25.827
traffic,extremal_mask,mae,DLinear,1,0.075,4.975,24.511
traffic,extremal_mask,mae,DLinear,1,0.1,6.026,23.732
Expand Down Expand Up @@ -1019,6 +1055,18 @@ traffic,extremal_mask,mae,iTransformer,3,0.05,2.684,28.826
traffic,extremal_mask,mae,iTransformer,3,0.075,3.675,27.637
traffic,extremal_mask,mae,iTransformer,3,0.1,4.364,26.79
traffic,extremal_mask,mae,iTransformer,3,0.15,5.903,24.883
traffic,extremal_mask,mae,CALF,1,0.05,5.022,27.649
traffic,extremal_mask,mae,CALF,1,0.075,7.759,25.86
traffic,extremal_mask,mae,CALF,1,0.1,9.221,24.744
traffic,extremal_mask,mae,CALF,1,0.15,11.835,22.432
traffic,extremal_mask,mae,CALF,2,0.05,4.836,28.193
traffic,extremal_mask,mae,CALF,2,0.075,7.28,26.49
traffic,extremal_mask,mae,CALF,2,0.1,8.59,25.367
traffic,extremal_mask,mae,CALF,2,0.15,11.155,23.004
traffic,extremal_mask,mae,CALF,3,0.05,4.935,27.831
traffic,extremal_mask,mae,CALF,3,0.075,7.615,25.475
traffic,extremal_mask,mae,CALF,3,0.1,9.081,24.215
traffic,extremal_mask,mae,CALF,3,0.15,11.833,21.68
traffic,winIT,mae,DLinear,1,0.05,6.655,26.383
traffic,winIT,mae,DLinear,1,0.075,7.083,25.671
traffic,winIT,mae,DLinear,1,0.1,7.343,25.184
Expand Down Expand Up @@ -1595,6 +1643,18 @@ mimic_iii,dyna_mask,auc,iTransformer,3,0.05,0.435,0.607
mimic_iii,dyna_mask,auc,iTransformer,3,0.075,0.437,0.576
mimic_iii,dyna_mask,auc,iTransformer,3,0.1,0.44,0.558
mimic_iii,dyna_mask,auc,iTransformer,3,0.15,0.45,0.526
mimic_iii,dyna_mask,auc,CALF,1,0.05,0.931,0.957
mimic_iii,dyna_mask,auc,CALF,1,0.075,0.931,0.957
mimic_iii,dyna_mask,auc,CALF,1,0.1,0.931,0.955
mimic_iii,dyna_mask,auc,CALF,1,0.15,0.931,0.946
mimic_iii,dyna_mask,auc,CALF,2,0.05,0.627,0.744
mimic_iii,dyna_mask,auc,CALF,2,0.075,0.628,0.715
mimic_iii,dyna_mask,auc,CALF,2,0.1,0.629,0.686
mimic_iii,dyna_mask,auc,CALF,2,0.15,0.634,0.661
mimic_iii,dyna_mask,auc,CALF,3,0.05,0.599,0.726
mimic_iii,dyna_mask,auc,CALF,3,0.075,0.598,0.697
mimic_iii,dyna_mask,auc,CALF,3,0.1,0.599,0.673
mimic_iii,dyna_mask,auc,CALF,3,0.15,0.602,0.635
mimic_iii,extremal_mask,auc,DLinear,1,0.05,0.614,0.646
mimic_iii,extremal_mask,auc,DLinear,1,0.075,0.632,0.643
mimic_iii,extremal_mask,auc,DLinear,1,0.1,0.651,0.64
Expand Down Expand Up @@ -1643,6 +1703,18 @@ mimic_iii,extremal_mask,auc,iTransformer,3,0.05,0.424,0.606
mimic_iii,extremal_mask,auc,iTransformer,3,0.075,0.431,0.575
mimic_iii,extremal_mask,auc,iTransformer,3,0.1,0.439,0.555
mimic_iii,extremal_mask,auc,iTransformer,3,0.15,0.462,0.531
mimic_iii,extremal_mask,auc,CALF,1,0.05,0.931,0.963
mimic_iii,extremal_mask,auc,CALF,1,0.075,0.931,0.96
mimic_iii,extremal_mask,auc,CALF,1,0.1,0.931,0.958
mimic_iii,extremal_mask,auc,CALF,1,0.15,0.931,0.951
mimic_iii,extremal_mask,auc,CALF,2,0.05,0.632,0.652
mimic_iii,extremal_mask,auc,CALF,2,0.075,0.637,0.642
mimic_iii,extremal_mask,auc,CALF,2,0.1,0.647,0.638
mimic_iii,extremal_mask,auc,CALF,2,0.15,0.662,0.633
mimic_iii,extremal_mask,auc,CALF,3,0.05,0.601,0.65
mimic_iii,extremal_mask,auc,CALF,3,0.075,0.605,0.635
mimic_iii,extremal_mask,auc,CALF,3,0.1,0.614,0.628
mimic_iii,extremal_mask,auc,CALF,3,0.15,0.631,0.624
mimic_iii,winIT,auc,DLinear,1,0.05,0.581,0.618
mimic_iii,winIT,auc,DLinear,1,0.075,0.59,0.6
mimic_iii,winIT,auc,DLinear,1,0.1,0.599,0.59
Expand Down
37 changes: 19 additions & 18 deletions results/time_efficiency.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -563,21 +563,22 @@ traffic,CALF,1,gradient_shap,2024-12-02 15:45:17.878299,0:07:14.477001,CALF_traf
traffic,CALF,2,integrated_gradients,2024-12-02 15:57:59.322569,0:12:38.741828,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,2,gradient_shap,2024-12-02 16:05:37.362382,0:07:37.839306,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,integrated_gradients,2024-12-02 16:17:51.232809,0:12:11.118725,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,gradient_shap,2024-12-02 16:24:55.338855,0:07:03.926698,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4electricity,CALF,3,extremal_mask,2024-12-03 23:41:13.886709,0:21:49.050907,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,2,extremal_mask,2024-12-03 23:41:32.657732,0:22:07.608213,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,1,extremal_mask,2024-12-03 23:41:40.839411,0:22:16.311136,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,extremal_mask,2024-12-03 23:56:35.221809,0:14:38.608498,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,2,extremal_mask,2024-12-03 23:56:36.343006,0:14:35.989732,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,1,extremal_mask,2024-12-03 23:57:01.464861,0:14:45.565032,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
mimic_iii,CALF,1,extremal_mask,2024-12-04 00:36:08.582614,0:38:39.760414,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,2,extremal_mask,2024-12-04 00:36:13.438178,0:39:04.463946,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,3,extremal_mask,2024-12-04 00:36:14.097345,0:39:04.778743,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
electricity,CALF,2,dyna_mask,2024-12-04 08:43:36.876473,0:17:17.764725,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,3,dyna_mask,2024-12-04 08:43:46.235815,0:17:27.592070,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,1,dyna_mask,2024-12-04 08:46:23.432281,0:20:08.503292,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,dyna_mask,2024-12-04 08:55:50.408682,0:11:36.040150,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,2,dyna_mask,2024-12-04 08:55:51.417164,0:11:39.634293,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,1,dyna_mask,2024-12-04 09:00:11.392176,0:13:22.953987,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
mimic_iii,CALF,3,dyna_mask,2024-12-04 09:22:17.200907,0:25:49.005921,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,2,dyna_mask,2024-12-04 09:22:40.642313,0:26:12.531972,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,1,dyna_mask,2024-12-04 09:26:35.831008,0:25:58.921301,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
traffic,CALF,3,gradient_shap,2024-12-02 16:24:55.338855,0:07:03.926698,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,3,extremal_mask,2024-12-03 23:41:13.886709,0:21:49.050907,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,2,extremal_mask,2024-12-03 23:41:32.657732,0:22:07.608213,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,1,extremal_mask,2024-12-03 23:41:40.839411,0:22:16.311136,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,extremal_mask,2024-12-03 23:56:35.221809,0:14:38.608498,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,2,extremal_mask,2024-12-03 23:56:36.343006,0:14:35.989732,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,1,extremal_mask,2024-12-03 23:57:01.464861,0:14:45.565032,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
mimic_iii,CALF,1,extremal_mask,2024-12-04 00:36:08.582614,0:38:39.760414,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,2,extremal_mask,2024-12-04 00:36:13.438178,0:39:04.463946,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,3,extremal_mask,2024-12-04 00:36:14.097345,0:39:04.778743,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
electricity,CALF,2,dyna_mask,2024-12-04 08:43:36.876473,0:17:17.764725,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,3,dyna_mask,2024-12-04 08:43:46.235815,0:17:27.592070,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
electricity,CALF,1,dyna_mask,2024-12-04 08:46:23.432281,0:20:08.503292,CALF_electricity_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,3,dyna_mask,2024-12-04 08:55:50.408682,0:11:36.040150,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,2,dyna_mask,2024-12-04 08:55:51.417164,0:11:39.634293,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
traffic,CALF,1,dyna_mask,2024-12-04 09:00:11.392176,0:13:22.953987,CALF_traffic_ftS_sl96_ll12_pl24_dm768_nh4
mimic_iii,CALF,3,dyna_mask,2024-12-04 09:22:17.200907,0:25:49.005921,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,2,dyna_mask,2024-12-04 09:22:40.642313,0:26:12.531972,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
mimic_iii,CALF,1,dyna_mask,2024-12-04 09:26:35.831008,0:25:58.921301,CALF_mimic_iii_ftMS_sl48_ll48_pl0_dm768_nh4
Binary file added results/traffic_iTransformer/1/heatmap_1530_normed.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified results/traffic_iTransformer/1/heatmap_1672_normed.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added results/traffic_iTransformer/1/heatmap_608_normed.jpg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 58526b5

Please sign in to comment.