How to integrate evaluation into a fresh version of open compass?

[Peter Greis]

As I am having much more success with a fresh copy of open compass, apparently something is missing. First I have copied over the chat musician configs (at least in my environment):

cp ../ChatMusician/eval/configs/eval_chat_musician_7b.py configs/
cp -r ../ChatMusician/eval/configs/datasets/music_theory_bench/ configs/datasets/
cp -r ../ChatMusician/eval/configs/models/chat_musician/ configs/models/

But a few of the validations fail; notably below
`KeyError: 'opencompass.datasets.MusicTheoryBenchDataset is not in the opencompass::load_dataset registry.

Is there an easy way to fix this?

04/13 15:02:53 - OpenCompass - INFO - Task [Mozart-Transposed/reasoning_few_shot]

Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]
Loading checkpoint shards:  14%|█▍        | 1/7 [00:01<00:11,  1.99s/it]
Loading checkpoint shards:  29%|██▊       | 2/7 [00:03<00:09,  1.97s/it]
Loading checkpoint shards:  43%|████▎     | 3/7 [00:06<00:08,  2.01s/it]
Loading checkpoint shards:  57%|█████▋    | 4/7 [00:08<00:06,  2.04s/it]
Loading checkpoint shards:  71%|███████▏  | 5/7 [00:10<00:04,  2.03s/it]
Loading checkpoint shards:  86%|████████▌ | 6/7 [00:12<00:02,  2.09s/it]
Loading checkpoint shards: 100%|██████████| 7/7 [00:13<00:00,  1.95s/it]
Loading checkpoint shards: 100%|██████████| 7/7 [00:13<00:00,  2.00s/it]
Traceback (most recent call last):
  File "/content/drive/MyDrive/project/opencompass/opencompass/tasks/openicl_infer.py", line 156, in <module>
    inferencer.run()
  File "/content/drive/MyDrive/project/opencompass/opencompass/tasks/openicl_infer.py", line 74, in run
    self.dataset = build_dataset_from_cfg(self.dataset_cfg)
  File "/content/drive/MyDrive/project/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 100, in build_from_cfg
    raise KeyError(
KeyError: 'opencompass.datasets.MusicTheoryBenchDataset is not in the opencompass::load_dataset registry. Please check whether the value of `opencompass.datasets.MusicTheoryBenchDataset` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'
[2024-04-13 15:03:15,430] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 82828) of binary: /root/anaconda3/envs/opencompass/bin/python
Traceback (most recent call last):
  File "/root/anaconda3/envs/opencompass/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.2.2', 'console_scripts', 'torchrun')())
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/anaconda3/envs/opencompass/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/content/drive/MyDrive/project/opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-04-13_15:03:15
  host      : fb7dea0d788a
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 82828)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[hf-lin]

@Magnetic2014 any ideas?

[Tianhao Shen]

Sorry for the late reply. Do you register the dataset in ChatMusician/eval/opencompass/datasets/__init__.py?

Please check out here for your reference:
https://github.com/hf-lin/ChatMusician/blob/440e65ee637156bbde28c63189dcbafbb974f2b6/eval/opencompass/datasets/__init__.py#L79-L80

[Peter Greis]

Ah.... thank you! That explains it...