why JFastText allowed only model trained with JFastext?

Question

why JFastText allowed only model trained with JFastext?

ali3assi opened this issue 7 years ago · comments

Hello,

How can read a pretrained model? I try to load the preexisting files .vec and .bin, but the load model raises an excpetion. Its looks like the format incompatible and JFastText allowed only model trained with JFastext.

Dalei Li · Answer 1 · Fri Jan 05 2018 16:16:35 GMT+0800 (China Standard Time)

You can upgrade the fastText within the cpp folder to the released version. Then run mvn clean install. The compiled jar package with dependency will be compatible with newer pre-trained models.

xikunlun001 · Answer 2 · Sat Feb 03 2018 02:18:39 GMT+0800 (China Standard Time)

@lidalei I got errors in upgrade fastText as below. Could you check this in your convenience time? Thanks.

In file included from /Users/xichen/Desktop/NLP project/JFastText/target/classes/com/github/jfasttext/jniFastTextWrapper.cpp:102:
In file included from /Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper_javacpp.h:13:
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper.cc:83:18: warning:
'getVector' is deprecated: getVector is being deprecated and replaced by
getWordVector. [-Wdeprecated-declarations]
fastText.getVector(vec, word);
^
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fastText/src/fasttext.h:63:3: note:
'getVector' has been explicitly marked deprecated here
FASTTEXT_DEPRECATED(
^
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fastText/src/utils.h:15:50: note:
expanded from macro 'FASTTEXT_DEPRECATED'

define FASTTEXT_DEPRECATED(msg) attribute((deprecated(msg)))

In file included from /Users/xichen/Desktop/NLP project/JFastText/target/classes/com/github/jfasttext/jniFastTextWrapper.cpp:102:
In file included from /Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper_javacpp.h:13:
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper.cc:84:38: error:
'data_' is a protected member of 'fasttext::Vector'
return std::vector(vec.data_, vec.data_ + vec.m_);
^
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fastText/src/vector.h:26:23: note:
declared protected here
std::vector data_;
^
In file included from /Users/xichen/Desktop/NLP project/JFastText/target/classes/com/github/jfasttext/jniFastTextWrapper.cpp:102:
In file included from /Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper_javacpp.h:13:
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper.cc:84:49: error:
'data_' is a protected member of 'fasttext::Vector'
return std::vector(vec.data_, vec.data_ + vec.m_);
^
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fastText/src/vector.h:26:23: note:
declared protected here
std::vector data_;
^
In file included from /Users/xichen/Desktop/NLP project/JFastText/target/classes/com/github/jfasttext/jniFastTextWrapper.cpp:102:
In file included from /Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper_javacpp.h:13:
/Users/xichen/Desktop/NLP project/JFastText/src/main/java/../cpp/fasttext_wrapper.cc:84:61: error:
no member named 'm_' in 'fasttext::Vector'
return std::vector(vec.data_, vec.data_ + vec.m_);

Dalei Li · Answer 3 · Sat Feb 03 2018 05:12:08 GMT+0800 (China Standard Time)

'getVector' is deprecated: getVector is being deprecated and replaced by
getWordVector. Besides, class Vector was rewritten. You cannot access data_ or m_ member of a vector. Instead, you have to use vector.data() and vector.size(). I'd suggest have a look at my fork https://github.com/lidalei/JFastText

xikunlun001 · Answer 4 · Sat Feb 03 2018 08:58:05 GMT+0800 (China Standard Time)

@lidalei
Thanks.
I just used the code in your fork but got the following error in loadModel as following, could you have a look:

Exception in thread "main" java.lang.UnsatisfiedLinkError: com.github.jfasttext.FastTextWrapper$FastTextApi.checkModel(Ljava/lang/String;)Z
at com.github.jfasttext.FastTextWrapper$FastTextApi.checkModel(Native Method)
at com.github.jfasttext.JFastText.loadModel(JFastText.java:29)
at com.github.jfasttext.JFastText.main(JFastText.java:203)

Dalei Li · Answer 5 · Sat Feb 03 2018 18:36:01 GMT+0800 (China Standard Time)

Could you release you code?

Tam · Answer 6 · Fri Apr 13 2018 23:34:36 GMT+0800 (China Standard Time)

Hello Sir @lidalei I just install your code : https://github.com/lidalei/JFastText

I take the generated two jar JFastText/target/ and added them to buildinf path in eclipse.

In my testDriver method i declared:

import com.github.jfasttext.JFastText;

public class TestDriver {

	public static void main(String[]args){
		JFastText jft = new JFastText();
		
		
	}
}

So, runing the code i get the follwing exception:

Exception in thread "main" java.lang.UnsatisfiedLinkError: no jniFastTextWrapper in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1191)
	at org.bytedeco.javacpp.Loader.load(Loader.java:953)
	at org.bytedeco.javacpp.Loader.load(Loader.java:854)
	at com.github.jfasttext.FastTextWrapper.<clinit>(FastTextWrapper.java:11)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.bytedeco.javacpp.Loader.load(Loader.java:913)
	at org.bytedeco.javacpp.Loader.load(Loader.java:854)
	at com.github.jfasttext.FastTextWrapper$FastTextApi.<clinit>(FastTextWrapper.java:442)
	at com.github.jfasttext.JFastText.<init>(JFastText.java:23)
	at TestDriver.main(TestDriver.java:6)

Any idea how to solve this issue please?

Dalei Li · Answer 7 · Sat Apr 14 2018 01:35:02 GMT+0800 (China Standard Time)

You should merely use jfasttext-0.1.0-jar-with-dependencies.jar, which can be generated by running mvn clean install.

Dalei Li · Answer 8 · Sat Apr 14 2018 01:36:49 GMT+0800 (China Standard Time)

Btw, you should clone the subfolder 'src/main/cpp/fastText' to compile a native library. @TamouzeAssi @xikunlun001 https://github.com/lidalei/fastText

Tam · Answer 9 · Sat Apr 14 2018 05:14:12 GMT+0800 (China Standard Time)

Unfortunately, it is not working under windows

Tam · Answer 10 · Sat Apr 14 2018 07:43:27 GMT+0800 (China Standard Time)

The problem still existing. We try to load pre-trained model, When we read this model by jft.loadModel(path/to/pretarined_model)
we get the following exception

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Model file's format is not compatible with this JFastText version!

Note that we get the the new fork of JFastText then we delete the file in src/cpp/ fasttext and clone agian this fasttext then run mvn clean install.

Any idea on solving this problem

Dalei Li · Answer 11 · Sun Apr 15 2018 19:24:55 GMT+0800 (China Standard Time)

@TamouzeAssi Model file's format is not compatible with this JFastText version means you should train your model with the corresponding fastText or JFastText. Don't use pip to install fasttext which is not official. Follow this to install Python binding, https://github.com/facebookresearch/fastText/tree/master/python.

Tam · Answer 12 · Sun Apr 15 2018 22:57:55 GMT+0800 (China Standard Time)

@lidalei Thank you first for your cooperation. I will try to clone the fastext in your mentionned link to the subfolder cpp in JFastext please correct me if im wrong.

I want to use the pretrained model bi the library fasttext like wiki.en. So this model trained by fastext which is different from JFasttext. I dont want to train again due to several reason.
Thank you

Dalei Li · Answer 13 · Mon Apr 16 2018 05:22:24 GMT+0800 (China Standard Time)

You can download word embeddings from https://fasttext.cc/docs/en/pretrained-vectors.html. I haven't tried but believe they work. JFastText relies on fastText. If JFastText complains, it means the model was trained with a non-compatible version fastTex with the fastText JFastText is using.

Tam · Answer 14 · Wed Apr 18 2018 15:26:11 GMT+0800 (China Standard Time)

@lidalei Sorry but still not working. The same exception is raised when i try to load word embeddings from l.

I clone your fork for JFastext then delete the folder cpp/fastText and clone again this file from where you said and then mvn clean install.
and the exception still existing.

Can you please descrive the step or try to load a pretained model using JFastText?

Dalei Li · Answer 15 · Thu Apr 19 2018 00:45:37 GMT+0800 (China Standard Time)

@TamouzeAssi I guess you were trying to load a word embedding. It cannot! Try to load a model from https://fasttext.cc/docs/en/language-identification.html.

Dalei Li · Answer 16 · Thu Apr 19 2018 03:16:42 GMT+0800 (China Standard Time)

@TamouzeAssi If it did not work, try to use python interface of fastText to load your model.

Tam · Answer 17 · Thu Apr 19 2018 22:11:08 GMT+0800 (China Standard Time)

@lidalei:
the model lid.176.bin from https://fasttext.cc/docs/en/language-identification.html can be loaded in JFastext without any error.

But withe wiki.en from https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md the model generate the incompatible file format.

By the way do you have any good reference to a model learned on wikipedia. Im looking to use the vector embedding to can cover the OOV. please

Dalei Li · Answer 18 · Thu Apr 19 2018 23:20:42 GMT+0800 (China Standard Time)

@TamouzeAssi There is no problem with this. pretrained vectors are just a word embedding that represents a word as a vector. It does not do any classification task. A classifier is built on the word embedding. For example, you can represent a sentence as mean vector of its words's vectors and train a classifier to classify an unknown sentence.

I don't have a model for you. It really depends on your task. What do you want to achieve?

Tam · Answer 19 · Thu Apr 19 2018 23:23:22 GMT+0800 (China Standard Time)

@lidalei for the moment i want just to compute the similarity between two sentences where some noise exists (miss typo). So, i used word2vec but i get bad result due to OOV so i go to use fastText to can use the subword information.

Dalei Li · Answer 20 · Thu Apr 19 2018 23:26:11 GMT+0800 (China Standard Time)

@TamouzeAssi Have a look at https://radimrehurek.com/gensim/models/word2vec.html#module-gensim.models.word2vec

Tam · Answer 21 · Thu Apr 19 2018 23:28:38 GMT+0800 (China Standard Time)

@lidalei i was used gensim to get the word2vec model but i developped my algo in java, and gensim can be used with java even we use jython language.

Dalei Li · Answer 22 · Thu Apr 19 2018 23:35:55 GMT+0800 (China Standard Time)

@TamouzeAssi I will add the function to my JfastText repo and tell you as soon as I complete.

Dalei Li · Answer 23 · Thu Apr 19 2018 23:43:38 GMT+0800 (China Standard Time)

@TamouzeAssi It won't help you soon. I'd suggest you check

void FastText::loadVectors(std::string filename) {
  std::ifstream in(filename);
  std::vector<std::string> words;
  std::shared_ptr<Matrix> mat; // temp. matrix for pretrained vectors
  int64_t n, dim;
  if (!in.is_open()) {
    throw std::invalid_argument(filename + " cannot be opened for loading!");
  }
  in >> n >> dim;
  if (dim != args_->dim) {
    throw std::invalid_argument(
        "Dimension of pretrained vectors (" + std::to_string(dim) +
        ") does not match dimension (" + std::to_string(args_->dim) + ")!");
  }
  mat = std::make_shared<Matrix>(n, dim);
  for (size_t i = 0; i < n; i++) {
    std::string word;
    in >> word;
    words.push_back(word);
    dict_->add(word);
    for (size_t j = 0; j < dim; j++) {
      in >> mat->at(i, j);
    }
  }
  in.close();

  dict_->threshold(1, 0);
  input_ = std::make_shared<Matrix>(dict_->nwords()+args_->bucket, args_->dim);
  input_->uniform(1.0 / args_->dim);

  for (size_t i = 0; i < n; i++) {
    int32_t idx = dict_->getId(words[i]);
    if (idx < 0 || idx >= dict_->nwords()) continue;
    for (size_t j = 0; j < dim; j++) {
      input_->at(idx, j) = mat->at(i, j);
    }
  }
}

and write some Java code to read pretrained vectors.

Tam · Answer 24 · Thu Apr 19 2018 23:56:15 GMT+0800 (China Standard Time)

@lidalei Thank you i will try to write similar code. By the way let me know when you add the function to your JFastText repo please

Li RAO · Answer 25 · Tue Nov 20 2018 17:58:25 GMT+0800 (China Standard Time)

val fasttext = new JFastText()
fasttext.loadModel("/home/work/XX/model/model.bin")

java.lang.IllegalArgumentException: Model file doesn't exist!