cannot install fastbpe
guangyuli-uoe opened this issue · comments
hi,
i noticed that "FastBPE, fast C++ implementation of byte-pair encoding (installed automatically)"
but it seems that there is no fastbpe,
and then i tried to install it manually, but it said that :
g++ -std=c++11 -pthread -O3 fastBPE/main.cc -IfastBPE -o fast
clang: error: no such file or directory: 'fastBPE/main.cc'
clang: error: no input files
could anyone know how to solve this problem ?
thanks !
specifically, the original problem occurred when i tried to run the bucc task:
./bucc.sh
Processing BUCC data in .
Traceback (most recent call last):
File "home/LASER/source/embed.py", line 36, in
from lib.text_processing import Token, BPEfastApply, SPMApply
File "home/LASER/source/lib/text_processing.py", line 21, in
import fastBPE
ModuleNotFoundError: No module named 'fastBPE'
Hi @guangyuli-uoe. Can you try pulling and then re-running your above command? This dependency has been removed.
the problem of fastbpe has been solved, thanks a lot,
but when i tried to re-run the command: ./bucc.sh, new problem occurred (it seems i did not have the 'models' directory and realted files, ):
it says that:
Processing BUCC data in .
2022-07-13 15:50:19,653 | INFO | fairseq.tasks.text_to_speech | Please install tensorboardX: pip install tensorboardX
2022-07-13 15:50:19,913 | INFO | embed | loading encoder: home/LASER/models/bilstm.93langs.2018-12-26.pt
Traceback (most recent call last):
File "home/LASER/source/embed.py", line 630, in
sort_kind=args.sort_kind,
File "home/LASER/source/embed.py", line 503, in embed_sentences
cpu=cpu,
File "home/LASER/source/embed.py", line 368, in load_model
encoder, spm_vocab=spm_vocab, verbose=verbose, **encoder_kwargs
File "home/LASER/source/embed.py", line 86, in init
state_dict = torch.load(model_path)
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/torch/serialization.py", line 594, in load
with _open_file_like(f, 'rb') as opened_file:
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/torch/serialization.py", line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/torch/serialization.py", line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'home/LASER/models/bilstm.93langs.2018-12-26.pt'
@guangyuli-uoe can you try running the model download script located here?
i found a new problem when i tried to run bucc task: Symbol not found: _mkl_blas_caxpy
TT
Processing BUCC data in .
2022-07-14 14:50:25,193 | INFO | embed | loading encoder: home/LASER/models/bilstm.93langs.2018-12-26.pt
2022-07-14 14:50:25,672 | INFO | preprocess | tokenizing in language fr
2022-07-14 14:50:36,032 | INFO | preprocess | fastBPE: processing tok
2022-07-14 14:50:38,551 | INFO | embed | encoder: bucc2018.fr-en.train.enc.fr exists already
2022-07-14 14:50:40,517 | INFO | embed | loading encoder: home/LASER/models/bilstm.93langs.2018-12-26.pt
2022-07-14 14:50:40,986 | INFO | preprocess | tokenizing in language en
2022-07-14 14:50:54,122 | INFO | preprocess | fastBPE: processing tok
2022-07-14 14:50:57,329 | INFO | embed | encoder: bucc2018.fr-en.train.enc.en exists already
Traceback (most recent call last):
File "home/LASER/source/mine_bitexts.py", line 20, in
import faiss
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/faiss/init.py", line 18, in
from .loader import *
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/faiss/loader.py", line 65, in
from .swigfaiss import *
File "home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/faiss/swigfaiss.py", line 13, in
from . import _swigfaiss
ImportError: dlopen(home/opt/anaconda3/envs/laser/lib/python3.6/site-packages/faiss/_swigfaiss.so, 2): Symbol not found: _mkl_blas_caxpy
Referenced from: home/opt/anaconda3/envs/laser/lib/libmkl_intel_lp64.dylib
Expected in: flat namespace
in home/opt/anaconda3/envs/laser/lib/libmkl_intel_lp64.dylib
Hi @guangyuli-uoe, this looks to be related to your faiss library install. Perhaps this thread on their website might help resolve the issue!
Closing as main issue(s) have been solved and remaining comment is related to a faiss library install.