Running with tutorial data gives issues...

[mojoee]

I managed to run the tutorial data. There are a few things that I ran into:

first, the data of the csv file of the nike sales is in a different format than expected. I needed to run a script to change the format in the csv file since the wrapper doesn't allow to set the format afaik.

Second, below error occurs after having the right format and running the following:

cwd = pathlib.Path.cwd() data_dir = cwd.joinpath('./docs/source/tutorials/tutorial_data') save_dir = data_dir model = 'xgboost' ForeTiS.optim_pipeline.run(data_dir=data_dir, data='nike_sales', config_file_section='nike_sales', save_dir=data_dir, models=[model], n_trials=10, periodical_refit_frequency=['complete'])

Still figuring it out how to solve this at this point. This is the error message:

---Start preprocessing data---
--Adding calendar dataset--
--Added calendar dataset--
-Adding statistical dataset-
No seasonal lags defined!
-one-hot-encoding the data-
-one-hot-encoded the data-
/home/mojoee/Research/01_ForeTis/ForeTiS/venv/lib/python3.10/site-packages/tables/path.py:137: NaturalNameWarning: object name is not a valid Python identifier: 'featureset-full'; it does not match the pattern ^[a-zA-Z_][a-zA-Z0-9_]*$; you will not be able to use natural naming to access this object; using getattr() will still work, though
check_attribute_name(name)
/home/mojoee/Research/01_ForeTis/ForeTiS/ForeTiS/optimization/optuna_optim.py:126: ExperimentalWarning: RetryFailedTrialCallback is experimental (supported from v2.8.0). The interface can change in the future.
failed_trial_callback=optuna.storages.RetryFailedTrialCallback(max_retry=3))
---Data preprocessed---

Dataset is loaded

Starting Optuna Optimization for model xgboost and featureset featureset-full

Starting Optimization

[I 2025-01-24 14:43:34,259] A new study created in RDB with name: 2025-01-24_14-43-33_-MODELxgboost-TRIALS10-FEATURESETfeatureset-full
/home/mojoee/Research/01_ForeTis/ForeTiS/venv/lib/python3.10/site-packages/optuna/study/_optimize.py:186: ExperimentalWarning: fail_stale_trials is experimental (supported from v2.9.0). The interface can change in the future.
optuna.storages.fail_stale_trials(study)
Params for Trial 0
{'n_estimators': 700, 'max_depth': 10, 'learning_rate': 0.225, 'gamma': 600, 'subsample': 0.2, 'colsample_bytree': 0.2, 'reg_lambda': 58.0, 'reg_alpha': 867.0}
[W 2025-01-24 14:43:34,855] Trial 0 failed with parameters: {'n_estimators': 700, 'max_depth': 10, 'learning_rate': 0.225, 'gamma': 600, 'subsample': 0.2, 'colsample_bytree': 0.2, 'reg_lambda': 58.0, 'reg_alpha': 867.0} because of the following error: AttributeError("'float' object has no attribute 'flatten'").
Traceback (most recent call last):
File "/home/mojoee/Research/01_ForeTis/ForeTiS/venv/lib/python3.10/site-packages/optuna/study/_optimize.py", line 197, in _run_trial
value_or_values = func(trial)
File "/home/mojoee/Research/01_ForeTis/ForeTiS/ForeTiS/optimization/optuna_optim.py", line 747, in
self.study.optimize(lambda trial: self.objective(trial=trial), n_trials=self.user_input_params["n_trials"])
File "/home/mojoee/Research/01_ForeTis/ForeTiS/ForeTiS/optimization/optuna_optim.py", line 271, in objective
y_pred = model.train_val_loop(train=train, val=val)[0]
File "/home/mojoee/Research/01_ForeTis/ForeTiS/ForeTiS/model/_sklearn_model.py", line 99, in train_val_loop
return self.predict(X_in=val)
File "/home/mojoee/Research/01_ForeTis/ForeTiS/ForeTiS/model/_sklearn_model.py", line 88, in predict
return self.prediction.flatten(), self.var.flatten()
AttributeError: 'float' object has no attribute 'flatten'
[W 2025-01-24 14:43:34,857] Trial 0 failed with value None.

AttributeError Traceback (most recent call last)
Cell In[10], line 1
----> 1 ForeTiS.optim_pipeline.run(data_dir=data_dir, data='nike_sales', config_file_section='nike_sales', save_dir=data_dir, models=[model], n_trials=10, periodical_refit_frequency=['complete'])

File ~/Research/01_ForeTis/ForeTiS/ForeTiS/optim_pipeline.py:130, in run(data_dir, save_dir, datasplit, test_set_size_percentage, val_set_size_percentage, n_splits, imputation_method, windowsize_current_statistics, windowsize_lagged_statistics, models, n_trials, pca_transform, save_final_model, periodical_refit_frequency, refit_drops, data, config_file_path, config_file_section, refit_window, intermediate_results_interval, batch_size, n_epochs, event_lags, optimize_featureset, scale_thr, scale_seasons, cf_thr_perc, scale_window_factor, cf_r, cf_order, cf_smooth, scale_window_minimum, max_samples_factor, valtest_seasons, seasonal_valtest)
110 optuna_run = optuna_optim.OptunaOptim(save_dir=save_dir, data=data, config_file_section=config_file_section,
111 featureset_name=featureset_name, datasplit=datasplit,
112 n_trials=n_trials, test_set_size_percentage=test_set_size_percentage,
(...)
126 seasonal_valtest=seasonal_valtest,
127 config_model_featureset=config_model_featureset)
128 print('### Starting Optuna Optimization for model ' + current_model_name + ' and featureset ' +
129 featureset_name + ' ###')
--> 130 overall_results = optuna_run.run_optuna_optimization
131 print('### Finished Optuna Optimization for ' + current_model_name + ' and featureset ' + featureset_name
132 + ' ###')
133 featureset_overview[featureset_name] = overall_results

File ~/Research/01_ForeTis/ForeTiS/ForeTiS/optimization/optuna_optim.py:747, in OptunaOptim.run_optuna_optimization(self)
745 self.current_best_val_result = None
746 # Start optimization run
--> 747 self.study.optimize(lambda trial: self.objective(trial=trial), n_trials=self.user_input_params["n_trials"])
748 self.study.best_trial_copy = self.study.best_trial
749 helper_functions.set_all_seeds()
...
86 return self.prediction.flatten(), self.var.flatten(), conf.flatten()
87 else:
---> 88 return self.prediction.flatten(), self.var.flatten()

AttributeError: 'float' object has no attribute 'flatten'
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[jeiglsperger]

Hey @mojoee, does the error also occur with the other example datasets?

[mojoee]

@jeiglsperger I will need to check again. I have actually worked with the darts framework in the meantime. This might not be related to above issue, but since you know your framework best, what are the benefits over darts?

[jeiglsperger]

Thanks, if the problem still occurs with different datasets, then I know I have to dig deeper into the code.
The advantage of our framework compared to darts is that we additionally offer hyperparameter optimization, high customizability, the possibility of periodical refitting, and automized feature engineering.
While darts indeed offers an easy-to-use framework for training and interference of time series forecasting models, ForeTiS is developed for proper optimizing and getting the best out of the models for your research.