specify config outside of defaults list

[dagap]

I have the following config directory structure

- configs
   - dataloaders
      - transforms
           - live.yaml
           - batch.yaml
      basic.yaml

Now in my basic.yaml, I have a list of dataset IDs and each of these datasets need to have specific transforms associated with them.

So, something like:

datasets:
  - 
    id: 10
    transforms: live.yaml
  - 
    id: 12
    transforms: batch.yaml

This does not work as is. I am not sure how to specify such configs outside of the defaults list. Is this paradigm supported by hydra or is there a workaround that can be used?

[Jasha10]

Hydra's defaults list is the primary supported means for "including" other configs (i.e. yaml files) into your composed output config.

A few alternatives to the defaults list:

• If you work at the lower level of OmegaConf, you might be able to define an omegaconf Custom Resolver that includes a yaml file from a given path: transforms: "${my_custom_oc_file_loader:batch.yaml}"
• You could use python code to inspect the value of datasets[0].transforms and then load the given yaml file via OmegaConf.load or pyyaml.

Both of these alternatives sidestep Hydra.
For a Hydra solution using the defaults list, you could try something like this:

defaults:
  - [email protected]: live.yaml
  - [email protected]: batch.yaml
datasets:
  - 
    id: 10
    transforms: live.yaml
  - 
    id: 12
    transforms: batch.yaml

This uses the @-syntax for specifying the package of the given default option.

[dagap]

@Jasha10 Thank you for taking the time to reply during the holidays!

I am getting a Cannot merge DictConfig with ListConfig error with the proposed solution. For completeness here is my full test config

# basic.yaml

defaults:
  - [email protected]: live.yaml
  - [email protected]: live.yaml

datasets:
  - 
    id: 10
    time_column: "timestamp"
    id_column: "record_id"
    transforms: live.yaml
  - 
    id: 12
    time_column: "ts"
    id_column: "instance_id"
    transforms: live.yaml

# live.yaml
_target_: modelzoo.pipeline.make_pipeline

steps_config:

  - TimeSeriesSorter:
      _target_: modelzoo.pipeline.TimeSeriesSorter
      time_column: ${time_column}

The base config is:

defaults:
  - _self_
  - dataloaders: basic

[dagap]

@Jasha10 Just bumping this in case you have some ideas.

[Jasha10]

@dagap Sorry for the slow reply! My suggestion above fell short due to a limitation in Hydra's support for list composition (related to #1939).
My comments here and here outline a workaround. In short, you use the oc.dict.values resolver to convert a dict into a list. This allows you to compose a dict using the defaults list and then convert that dict's values into the desired list lazily at runtime.

Here's the working example:

# app.py
import hydra
from omegaconf import OmegaConf

@hydra.main(version_base=None, config_name="basic.yaml", config_path="configs")
def main(cfg):
    print(OmegaConf.to_yaml(cfg.datasets, resolve=True))

if __name__ == "__main__":
    main()

# configs/dataloader/basic.yaml
defaults:
  - _self_
  - transforms@_datasets_kv.one.transforms: live.yaml
  - transforms@_datasets_kv.two.transforms: live.yaml

datasets: ${oc.dict.values:._datasets_kv}

_datasets_kv:
  one:
    id: 10
    time_column: "timestamp"
    id_column: "record_id"
  two:
    id: 12
    time_column: "ts"
    id_column: "instance_id"

# configs/dataloader/transforms/live.yaml
_target_: modelzoo.pipeline.make_pipeline

steps_config:
  - TimeSeriesSorter:
      _target_: modelzoo.pipeline.TimeSeriesSorter
      time_column: ${.....time_column}

$ python app.py
- id: 10
  time_column: timestamp
  id_column: record_id
  transforms:
    _target_: modelzoo.pipeline.make_pipeline
    steps_config:
    - TimeSeriesSorter:
        _target_: modelzoo.pipeline.TimeSeriesSorter
        time_column: timestamp
- id: 12
  time_column: ts
  id_column: instance_id
  transforms:
    _target_: modelzoo.pipeline.make_pipeline
    steps_config:
    - TimeSeriesSorter:
        _target_: modelzoo.pipeline.TimeSeriesSorter
        time_column: ts

[dagap]

Thank you @Jasha10 ! This seems like a good workaround!

Can I ask what ${.....time_column} syntax does? I never saw this. Is that something to do with relative paths?

[Jasha10]

Is that something to do with relative paths?

Yes, exactly!
While ${foo.bar} is an "absolute path", addressed from the root of the config, ${.foo.bar} is relative to the present node in the config, ${..foo.bar} is relative to the parent node, and so on.

To make an analogy with unix-like filesystems, /foo/bar is an absolute path while ./foo/bar is relative to the pwd and ../foo/bar is relative to the parent.

More info here: https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#variable-interpolation