Cannot do pseudo-labelling due to some error
alicewith opened this issue · comments
Hi, folks. I'm struggling to train the distil-whisper by myself, as I've encountered some error when I do pseudo-labelling by following the training manual.
$ accelerate launch run_pseudo_labelling.py \
--model_name_or_path "openai/whisper-large-v2" \
--dataset_name "mozilla-foundation/common_voice_13_0" \
--dataset_config_name "hi" \
--dataset_split_name "train+validation+test" \
--text_column_name "sentence" \
--id_column_name "path" \
--output_dir "./common_voice_13_0_hi_pseudo_labelled" \
--wandb_project "distil-whisper-labelling" \
--per_device_eval_batch_size 64 \
--dtype "bfloat16" \
--dataloader_num_workers 16 \
--preprocessing_num_workers 16 \
--logging_steps 500 \
--max_label_length 128 \
--report_to "wandb" \
--language "hi" \
--task "transcribe" \
--return_timestamps \
--attn_type "flash_attn" \
--streaming False \
--generation_num_beams 1 \
--decode_token_ids False \
--push_to_hub
[redacted]
Traceback (most recent call last):
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\training\run_pseudo_labelling.py", line 849, in <module>
main()
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\training\run_pseudo_labelling.py", line 532, in main
model = model.to_bettertransformer()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Lib\site-packages\transformers\modeling_utils.py", line 4479, in to_bettertransformer
return BetterTransformer.transform(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\scoop\apps\python311\current\Lib\contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Lib\site-packages\optimum\bettertransformer\transformation.py", line 211, in transform
raise ValueError(
ValueError: Transformers now supports natively BetterTransformer optimizations (torch.nn.functional.scaled_dot_product_attention) for the model type whisper. Please upgrade to transformers>=4.36 and torch>=2.1.1 to use it. Details: https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-and-memory-efficient-attention-through-pytorchs-scaleddotproductattention
wandb: You can sync this run to the cloud by running:
wandb: wandb sync C:\Users\user\Desktop\distil-whisper\distil-whisper\training\wandb\offline-run-20240213_104952-ta9ws4ck
wandb: Find logs at: .\wandb\offline-run-20240213_104952-ta9ws4ck\logs
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
args.func(args)
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Lib\site-packages\accelerate\commands\launch.py", line 1023, in launch_command
simple_launcher(args)
File "C:\Users\user\Desktop\distil-whisper\distil-whisper\venv\Lib\site-packages\accelerate\commands\launch.py", line 643, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\user\\Desktop\\distil-whisper\\distil-whisper\\venv\\Scripts\\python.exe', 'run_pseudo_labelling.py', '--model_name_or_path', 'openai/whisper-large-v2', '--dataset_name', 'mozilla-foundation/common_voice_13_0', '--dataset_config_name', 'hi', '--dataset_split_name', 'train+validation+test', '--text_column_name', 'sentence', '--id_column_name', 'path', '--output_dir', './common_voice_13_0_hi_pseudo_labelled', '--wandb_project', 'distil-whisper-labelling', '--per_device_eval_batch_size', '64', '--dtype', 'bfloat16', '--dataloader_num_workers', '16', '--preprocessing_num_workers', '16', '--logging_steps', '500', '--max_label_length', '128', '--report_to', 'wandb', '--language', 'hi', '--task', 'transcribe', '--return_timestamps', '--attn_type', 'flash_attn', '--streaming', 'False', '--generation_num_beams', '1', '--decode_token_ids', 'False', '--push_to_hub']' returned non-zero exit status 1.How could I solve this? Thanks.
Try the code change in #76 ?