How to set config properly to launch an experiment on wk103 (of any model)?

[Chord-Chen-30]

I run the command

python train.py wandb.mode=offline experiment=wt103/base pipeline=wt103

from safari/ directory and the program gives:

Error executing job with overrides: ['wandb.mode=offline', 'experiment=wt103/base', 'pipeline=wt103']
Traceback (most recent call last):
  File "/public/home/chenzhuo/cz/safari/train.py", line 679, in main
    config = utils.train.process_config(config)
  File "/public/home/chenzhuo/cz/safari/src/utils/train.py", line 69, in process_config
    config = omegaconf_filter_keys(config, lambda k: not k.startswith('__'))
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in omegaconf_filter_keys
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in <dictcomp>
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in omegaconf_filter_keys
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
omegaconf.errors.InterpolationResolutionError: KeyError raised while resolving interpolation: "Environment variable 'DATA_PATH' not found"
    full_key: dataset.cache_dir
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Then I add HYDRA_FULL_ERROR=1 :

Error executing job with overrides: ['wandb.mode=offline', 'experiment=wt103/base', 'pipeline=wt103']
Traceback (most recent call last):
  File "/public/home/chenzhuo/cz/safari/train.py", line 689, in <module>
    main()
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/public/home/chenzhuo/cz/safari/train.py", line 679, in main
    config = utils.train.process_config(config)
  File "/public/home/chenzhuo/cz/safari/src/utils/train.py", line 69, in process_config
    config = omegaconf_filter_keys(config, lambda k: not k.startswith('__'))
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in omegaconf_filter_keys
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in <dictcomp>
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
  File "/public/home/chenzhuo/cz/safari/src/utils/config.py", line 121, in omegaconf_filter_keys
    {k: omegaconf_filter_keys(v, fn) for k, v in d.items() if fn(k)}
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 562, in items
    return dict(self.items_ex(resolve=True, keys=None)).items()
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 588, in items_ex
    value = self[key]
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 375, in __getitem__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 369, in __getitem__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 451, in _get_impl
    return self._resolve_with_default(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 98, in _resolve_with_default
    resolved_node = self._maybe_resolve_interpolation(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 719, in _maybe_resolve_interpolation
    return self._resolve_interpolation_from_parse_tree(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
    resolved = self.resolve_parse_tree(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 769, in resolve_parse_tree
    raise InterpolationResolutionError(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
    return visitor.visit(parse_tree)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
    return visitor.visitConfigValue(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
    return visitor.visitText(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 301, in visitText
    return self._unescape(list(ctx.getChildren()))
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 389, in _unescape
    text = str(self.visitInterpolation(node))
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
    return visitor.visitInterpolationResolver(self)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
    return self.resolver_interpolation_callback(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
    return self._evaluate_custom_resolver(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/base.py", line 694, in _evaluate_custom_resolver
    return resolver(
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 445, in resolver_wrapper
    ret = resolver(*args, **kwargs)
  File "/public/home/chenzhuo/anaconda3/envs/effseq/lib/python3.9/site-packages/omegaconf/resolvers/oc/__init__.py", line 38, in env
    raise KeyError(f"Environment variable '{key}' not found")
omegaconf.errors.InterpolationResolutionError: KeyError raised while resolving interpolation: "Environment variable 'DATA_PATH' not found"
    full_key: dataset.cache_dir
    object_type=dict

My questions are:

1. Am I using the config correctly? If not, how to set the config?
2. What does the output mean?

[Michael Poli]

You should set a DATA_PATH environment variable: export DATA_PATH=/your/path/.

[Chord-Chen-30]

Thank you for the reply. It seems to fix the problem. However, I am confused about the statement in safari/src/dataloaders/README.md.

In Overview:
By default, data is downloaded to ./data/ by default, ...

In Advanced Usage:
The data path can be configured by setting the environment variable DATA_PATH, which defaults to ./data

Why the default setting is not applied (and we need to explictly set DATA_PATH).