Integrate model into existing NLP system

[Stefan Schweter]

Hi @timoschick,

first, thanks for releasing the code of the form-context model ❤️

I'm currently reading the AAAI and the NAACL papers, and I'm really excited to try model on downstream tasks (like e.g. for NER).

I would like to ask, if you could provide a kind of example notebook, that shows a) how to load a trained model and b) how to get embeddings for rare words.

I don't have any experience of how attentive mimicking works on tasks like NER or PoS tagging, but I would really like to integrate such an approach into the flair library.

Cheers,

Stefan

[Timo Schick]

Hi @stefan-it ,

thanks for your interest in our work :)
Unfortunately, I do not have much time right now, so at least for the next few weeks, I will not be able to provide an example notebook. However, I have just today uploaded a new version of the form-context model that also comes with an updated Readme file which contains more detailed instructions on how to (i) train a model and (ii) get embeddings for rare words using a trained model.

Best regards

Timo

[Stefan Schweter]

Hi Timo,

thanks for your reply! Over the holidays I'll definitely check out your implementation (great, that preprocessing is done in Python now) :)

Do you know how long the training will run on standard hardware (like an RTX card)? I would like to train a model from scratch.

Thanks,

Stefan

[Timo Schick]

Hi Stefan,

Do you know how long the training will run on standard hardware (like an RTX card)?

the version used in the paper was trained using a single Nvidia GeForce GTX 1080 Ti, with which training on Wikipedia for 5 epochs took approximately 8 hours, if I remember correctly. However, some of the model's operations (in particular, the n-gram embedding lookup) are executed on CPU.

Best regards

Timo

[Stefan Schweter]

Thanks :)

I was able to train a model from scratch and I had no problems 👍

I'll close that issue now and will open a new one as long as I have further questions :)