Question: Clarification on TASAR vs. Gumbeldore Dataset and Hyperparameter Configuration

[starjob42]

Hi,

I’m currently exploring the evaluation process and wanted to test inference using the TASAR approach instead of beam search. However, I’m a bit confused about the distinction between the Gumbeldore dataset and TASAR itself. Could you please clarify how they differ and their respective roles?

Additionally, for evaluation hyperparameters, should I configure them in jssp/config.py or directly modify the variable values in reproduce.py?

Thank you for your assistance!

[johnnycrab]

Hi,
Thanks again for reaching out. Sure, so first, regarding the difference: All in all, the code structure of tasar and the gumbeldore-repo is the same. In both repos, there is this class GumbeldoreDataset (core/gumbeldore_dataset.py), that basically collects problem instances, and distributes them to different workers, which in parallel sample a bunch of solutions for each instance and return the best one. The question is, how these solutions are sampled. And you can see in line 191 of gumbeldore_dataset.py, that there are different options how the solutions are sampled/obtained, for example by beam search, or stochastic beam search, and so on. How this is done is specified in the config-variable gumbeldore_config.search_type. In the gumbeldore-repo you could originally set this to "wor" (sample without replacement using round-wise SBS), or "gumbeldore" for the method in the "Self-Improvement for Neural CO"-paper.

In this repo, you can set the search_type also to tasar (which is the proposed sampling mechanism in the "Take a step and reconsider"-paper), where solutions are generated with the "sample without replacement, then freeze a part of the solution, sample again without replacement"-approach as described in the paper.
Note that when you set it to tasar, then the most important other parameters in gumbeldore_config are:

• beam_width: the beam width for stochastic beam search, as always
• replan_steps: how many actions the best solution should be followed before sampling unseen alternatives and
• min_nucleus_top_p: here, this value is taken for constant Top-p sampling (no growing nucleus as in Gumbeldore). Set to 1 if you don't want top-p sampling.

The bold parts of the README highlight the differences to gumbeldore.

---

Now, regarding your second question. I would suggest configuring them in jssp/config.py. The reproduce.py - file is meant more for a quick way (especially for reviewers) to confirm the reported results of the paper. If you only want to evaluate, but want more control, then I'd suggest setting the parameters in jssp/config.py and run jssp_main.py. These are the most important parameters in the config for evaluation:

• load_checkpoint_from_path: Set it to the location of the model checkpoint to load
• num_epochs: Set this to zero, so the training loop is skipped and we go straight to evaluation
• test_set_path: This should point to some test set, e.g. "./data/jssp/literature/jssp_taillard_15_15.pickle"
• gumbeldore_eval: Set this to False, if you want regular beam search evaluation. In this case, you will also need to set devices_for_eval_workers and beams_with_batch_sizes (as in reproduce.py). If you set it to True, then evaluation will be performed with the settings specified in gumbeldore_config! For example, if you want to use "tasar" as an inference method for JSSP, as in Table 2 in the paper, set gumbeldore_eval to True, and then replan_steps=50, min_nucleus_top_p=0.9, beam_width=64, search_type="tasar".

I hope that this helps so far! Let me know if everything works as expected or you need more clarification / run into problems. 😃

All the best,
Jonathan