issue with dataset "openml__one-hundred-plants-texture__9956"
duncanmcelfresh opened this issue · comments
error from log file, for reference:
run_experiment: model_name: LinearModel
run_experiment: dataset_name: openml__one-hundred-plants-texture__9956
run_experiment: env_name: sklearn
run_experiment: instance_name: all-datasets-b-0-59
run_experiment: experiment_name: all-datasets-b
run_experiment: config_file: /home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml
launching instance all-datasets-b-0-59...
Created [https://www.googleapis.com/compute/v1/projects/research-collab-naszilla/zones/us-central1-a/instances/all-datasets-b-0-59].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
all-datasets-b-0-59 us-central1-a n1-highmem-2 10.128.0.17 34.173.62.27 RUNNING
successfully created instance: all-datasets-b-0-59
Warning: Permanently added 'compute.2167066988362256097' (ECDSA) to the list of known hosts.
ENV_NAME: sklearn
MODEL_NAME: LinearModel
DATASET_NAME: openml__one-hundred-plants-texture__9956
EXPERIMENT_NAME: all-datasets-b
CONFIG_FILE: /home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml
no change /opt/conda/condabin/conda
no change /opt/conda/bin/conda
no change /opt/conda/bin/conda-env
no change /opt/conda/bin/activate
no change /opt/conda/bin/deactivate
no change /opt/conda/etc/profile.d/conda.sh
no change /opt/conda/etc/fish/conf.d/conda.fish
no change /opt/conda/shell/condabin/Conda.psm1
no change /opt/conda/shell/condabin/conda-hook.ps1
no change /opt/conda/lib/python3.7/site-packages/xontrib/conda.xsh
no change /opt/conda/etc/profile.d/conda.csh
no change /home/duncan/.bashrc
No action taken.
running experiment with model LinearModel on dataset openml__one-hundred-plants-texture__9956 in env sklearn
ARGS: Namespace(experiment_config='/home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml', dataset_dir='./datasets/openml__one-hundred-plants-texture__9956', model_name='LinearModel')
EXPERIMENT ARGS: Namespace(experiment_config='/home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml', output_dir='./results/', use_gpu=False, gpu_ids=[0], data_parallel=True, n_random_trials=30, hparam_seed=0, n_opt_trials=0, batch_size=128, val_batch_size=256, early_stopping_rounds=20, epochs=500, logging_period=100, experiment_time_limit=36000, trial_time_limit=7200)
evaluating 30 random hyperparameter samples...
A new study created in memory with name: no-name-a2a6db8d-0e26-4178-bb86-23a0c6579aa7
ESC[32m[I 2022-11-03 07:41:37,809]ESC[0m A new study created in memory with name: no-name-a2a6db8d-0e26-4178-bb86-23a0c6579aa7ESC[0m
/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/study.py:393: FutureWarning: `n_jobs` argument has been deprecated in v2.7.0. This feature will be removed in v4.0.0. See https://github.com/optuna/optuna/releases/tag/v2.7.0.
warnings.warn(
Trial 0 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
Trial 1 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 07:49:28,626]ESC[0m Trial 0 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
ESC[32m[I 2022-11-03 07:49:34,782]ESC[0m Trial 1 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 2 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 07:57:15,903]ESC[0m Trial 2 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 3 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 07:57:29,630]ESC[0m Trial 3 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 4 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 08:05:01,552]ESC[0m Trial 4 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 5 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 08:05:23,685]ESC[0m Trial 5 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 6 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 08:12:47,849]ESC[0m Trial 6 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
Trial 7 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.
ESC[32m[I 2022-11-03 08:13:19,891]ESC[0m Trial 7 finished with value: 4.211316808221516 and parameters: {}. Best is trial 0 with value: 4.211316808221516.ESC[0m
packet_write_wait: Connection to 34.173.62.27 port 22: Broken pipe
../utils.sh: line 22: 13604 Killed gcloud compute ssh --ssh-flag="-A" ${instance_name} --zone=${zone} --project=${project} --command=" export ENV_NAME=\"${env_name}\"; export MODEL_NAME=${model_name}; export DATASET_NAME=${dataset_name}; export EXPERIMENT_NAME=${experiment_name}; export CONFIG_FILE=${config_file}; chmod +x ${instance_script}; /bin/bash ${instance_script}"
failed to run experiment during attempt 2... (exit code: 137)
trying again in 30 seconds...
ENV_NAME: sklearn
MODEL_NAME: LinearModel
DATASET_NAME: openml__one-hundred-plants-texture__9956
EXPERIMENT_NAME: all-datasets-b
CONFIG_FILE: /home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml
no change /opt/conda/condabin/conda
no change /opt/conda/bin/conda
no change /opt/conda/bin/conda-env
no change /opt/conda/bin/activate
no change /opt/conda/bin/deactivate
no change /opt/conda/etc/profile.d/conda.sh
no change /opt/conda/etc/fish/conf.d/conda.fish
no change /opt/conda/shell/condabin/Conda.psm1
no change /opt/conda/shell/condabin/conda-hook.ps1
no change /opt/conda/lib/python3.7/site-packages/xontrib/conda.xsh
no change /opt/conda/etc/profile.d/conda.csh
no change /home/duncan/.bashrc
No action taken.
running experiment with model LinearModel on dataset openml__one-hundred-plants-texture__9956 in env sklearn
ARGS: Namespace(experiment_config='/home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml', dataset_dir='./datasets/openml__one-hundred-plants-texture__9956', model_name='LinearModel')
EXPERIMENT ARGS: Namespace(experiment_config='/home/shared/tabzilla/TabSurvey/tabzilla_experiment_config.yml', output_dir='./results/', use_gpu=False, gpu_ids=[0], data_parallel=True, n_random_trials=30, hparam_seed=0, n_opt_trials=0, batch_size=128, val_batch_size=256, early_stopping_rounds=20, epochs=500, logging_period=100, experiment_time_limit=36000, trial_time_limit=7200)
evaluating 30 random hyperparameter samples...
A new study created in memory with name: no-name-d9dfc8d4-c011-485d-bc2f-2c4f3f82949b
ESC[32m[I 2022-11-03 10:41:57,367]ESC[0m A new study created in memory with name: no-name-d9dfc8d4-c011-485d-bc2f-2c4f3f82949bESC[0m
/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/study.py:393: FutureWarning: `n_jobs` argument has been deprecated in v2.7.0. This feature will be removed in v4.0.0. See https://github.com/optuna/optuna/releases/tag/v2.7.0.
warnings.warn(
Trial 0 failed because of the following error: AssertionError('file already exists: /home/shared/tabzilla/TabSurvey/results/default_trial0_results.json')
Traceback (most recent call last):
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
value_or_values = func(trial)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 163, in __call__
result.write(result_file_base, compress=False)
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 136, in write
write_dict_to_json(
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 300, in write_dict_to_json
assert not filepath.is_file(), f"file already exists: {filepath}"
AssertionError: file already exists: /home/shared/tabzilla/TabSurvey/results/default_trial0_results.json
ESC[33m[W 2022-11-03 10:49:37,791]ESC[0m Trial 0 failed because of the following error: AssertionError('file already exists: /home/shared/tabzilla/TabSurvey/results/default_trial0_results.json')ESC[0m
Traceback (most recent call last):
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
value_or_values = func(trial)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 163, in __call__
result.write(result_file_base, compress=False)
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 136, in write
write_dict_to_json(
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 300, in write_dict_to_json
assert not filepath.is_file(), f"file already exists: {filepath}"
AssertionError: file already exists: /home/shared/tabzilla/TabSurvey/results/default_trial0_results.json
Trial 1 failed because of the following error: AssertionError('file already exists: /home/shared/tabzilla/TabSurvey/results/random_1_s0_trial1_results.json')
Traceback (most recent call last):
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
value_or_values = func(trial)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 163, in __call__
result.write(result_file_base, compress=False)
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 136, in write
write_dict_to_json(
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 300, in write_dict_to_json
assert not filepath.is_file(), f"file already exists: {filepath}"
AssertionError: file already exists: /home/shared/tabzilla/TabSurvey/results/random_1_s0_trial1_results.json
ESC[33m[W 2022-11-03 10:49:40,030]ESC[0m Trial 1 failed because of the following error: AssertionError('file already exists: /home/shared/tabzilla/TabSurvey/results/random_1_s0_trial1_results.json')ESC[0m
Traceback (most recent call last):
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
value_or_values = func(trial)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 163, in __call__
result.write(result_file_base, compress=False)
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 136, in write
write_dict_to_json(
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 300, in write_dict_to_json
assert not filepath.is_file(), f"file already exists: {filepath}"
AssertionError: file already exists: /home/shared/tabzilla/TabSurvey/results/random_1_s0_trial1_results.json
Traceback (most recent call last):
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 284, in <module>
main(experiment_args, args.model_name, args.dataset_dir)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 203, in main
study.optimize(
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/study.py", line 400, in optimize
_optimize(
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 106, in _optimize
f.result()
File "/opt/conda/envs/sklearn/lib/python3.10/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/opt/conda/envs/sklearn/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/opt/conda/envs/sklearn/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 163, in _optimize_sequential
trial = _run_trial(study, func, catch)
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 264, in _run_trial
raise func_err
File "/opt/conda/envs/sklearn/lib/python3.10/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
value_or_values = func(trial)
File "/home/shared/tabzilla/TabSurvey/tabzilla_experiment.py", line 163, in __call__
result.write(result_file_base, compress=False)
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 136, in write
write_dict_to_json(
File "/home/shared/tabzilla/TabSurvey/tabzilla_utils.py", line 300, in write_dict_to_json
assert not filepath.is_file(), f"file already exists: {filepath}"
AssertionError: file already exists: /home/shared/tabzilla/TabSurvey/results/default_trial0_results.json
failed to run experiment during attempt 3... (exit code: 1)
too many SSH attempts. giving up and deleting instance.
The following instances will be deleted. Any attached disks configured to be
auto-deleted will be deleted unless they are attached to any other instances or
the `--keep-disks` flag is given and specifies them for keeping. Deleting a disk
is irreversible and any data on the disk will be lost.
- [all-datasets-b-0-59] in [us-central1-a]
Do you want to continue (Y/n)?
Deleted [https://www.googleapis.com/compute/v1/projects/research-collab-naszilla/zones/us-central1-a/instances/all-datasets-b-0-59].
this error is similar to #62
no longer relevant