Interpolation in Hydra's defaults list cause and error

Question

This is the files directory:

|-configs
|----data_conf
|--------csv_images.csv
|--------tf_ds.csv
|----example.yaml 

and example.yaml is:

data: csv_images

defaults:
  - data_conf: "${data}"

and csv_images.yaml:

# @package _group_
a: test_a
b: test_b

My intention is that hydra will replace csv_images with csv_images.yaml. Meaning, the interpretation of example.yaml at run time will be:

data:
    a: test_a
    b: test_b

I defined a ConfigStore this way:

config_name = "example"
cs = ConfigStore.instance()
cs.store(name=config_name, node=Config)
cs.store(group='data_conf', name='csv_images', node=DatasetConfig)

Config and DatasetConfig are defined this way:

@dataclass
class Config:
    data: Any = MISSING

@dataclass
class DatasetConfig:
    a: str = MISSING
    b: str = MISSING

However, when i am running my program, i get the following error:

Interpolation key 'data' not found
    full_key: defaults[0].data_conf
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

When i define data_conf to be csv_images it works fine and run time checking is performed. However, interpolation with ${} doesn't. What is that and what can i do?

Answer 1

It is not possible to reference a key in the main config using interpolation from the defaults list.

From the Hydra docs on Interpolation in the Defaults List:

Interpolation Keys in the Defaults List cannot reference values in the Final Config Object (it does not yet exist).

In particular, it is important to understand that defaults lists are fully constructed before Hydra's output config is constructed. Since the resolved defaults list will be used to guide construction of the output config, it is necessary to process defaults first, before beginning to construct the output config.

Since interpolations in the defaults list must be resolved before construction of the output config, it is not possible to interpolate to values in the output config.

On the other hand, it is possible to use interpolations to refer to other values inside the defaults list. For example, you can give the following a try:

# example.yaml

defaults:
  - data: csv_images
  - data_conf: "${data}"