Using udpipe without creating file on disk
EmilHvitfeldt opened this issue · comments
I'm trying to use {udpipe} but my use-case doesn't allow me to depend on having a file on disk to read from.
It is possible to use {udpipe} and have the .udpipe model saved in the R environment instead of to a file?
I have tried to read the .udpipe into a character vector and write it to a temporary file when needed but udpipe() doesn't like that.
library(udpipe)
udmodel <- udpipe_download_model(language = "dutch")
#> Downloading udpipe model from https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.4/master/inst/udpipe-ud-2.4-190531/dutch-alpino-ud-2.4-190531.udpipe to /private/var/folders/m0/zmxymdmd7ps0q_tbhx0d_26w0000gn/T/RtmpEd9tDj/reprexe2d274ccfb69/dutch-alpino-ud-2.4-190531.udpipe
#> Visit https://github.com/jwijffels/udpipe.models.ud.2.4 for model license details
xxx <- readr::read_lines(udmodel$file_model)
temp_file <- tempfile()
writeLines(xxx, temp_file)
udmodel$file_model <- temp_file
x <- udpipe(x = "Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.",
object = udmodel)
#> Error in udp_tokenise_tag_parse(object$model, x, doc_id, tokenizer, tagger, : external pointer is not validCreated on 2020-09-14 by the reprex package (v0.3.0)
udpipe models are binary objects. e.g. udpipe_train creates a std::ofstream::binary as in https://github.com/bnosac/udpipe/blob/master/src/rcpp_udpipe.cpp#L154
udpipe_load_model takes the binary object and loads it
There is a method which loads it from file:
- https://github.com/ufal/udpipe/blob/9aacb3f1dc06651d3ad0b46c093dabb09fb754f7/src/model/model.cpp#L23
and a method which loads it from an istream - https://github.com/ufal/udpipe/blob/9aacb3f1dc06651d3ad0b46c093dabb09fb754f7/src/model/model.cpp#L29
This last one could in principle be used to pass binary data from R to the load functionality. This would require changes here: https://github.com/bnosac/udpipe/blob/master/src/rcpp_udpipe.cpp#L14
Oh! it is a binary file! thank you. I got it working now
library(udpipe)
udmodel <- udpipe_download_model(language = "dutch")
#> Downloading udpipe model from https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.4/master/inst/udpipe-ud-2.4-190531/dutch-alpino-ud-2.4-190531.udpipe to /private/var/folders/m0/zmxymdmd7ps0q_tbhx0d_26w0000gn/T/RtmpB19qoj/reprexe6a740205f6/dutch-alpino-ud-2.4-190531.udpipe
#> Visit https://github.com/jwijffels/udpipe.models.ud.2.4 for model license details
xxx <- readBin(file(udmodel$file_model, "rb"), "raw", file.info(udmodel$file_model)$size)
temp_file <- tempfile()
writeBin(xxx, temp_file)
udmodel$file_model <- temp_file
udpipe(x = "Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.",
object = udmodel)
#> doc_id paragraph_id sentence_id
#> 1 doc1 1 1
#> 2 doc1 1 1
#> 3 doc1 1 1
#> 4 doc1 1 1
#> 5 doc1 1 1
#> 6 doc1 1 1
#> 7 doc1 1 1
#> 8 doc1 1 1
#> 9 doc1 1 1
#> 10 doc1 1 1
#> 11 doc1 1 1
#> 12 doc1 1 1
#> 13 doc1 1 1
#> 14 doc1 1 1
#> 15 doc1 1 1
#> 16 doc1 1 1
#> 17 doc1 1 1
#> 18 doc1 1 1
#> sentence
#> 1 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 2 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 3 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 4 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 5 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 6 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 7 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 8 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 9 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 10 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 11 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 12 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 13 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 14 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 15 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 16 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 17 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> 18 Ik ging op reis en ik nam mee: mijn laptop, mijn zonnebril en goed humeur.
#> start end term_id token_id token lemma upos
#> 1 1 2 1 1 Ik ik PRON
#> 2 4 7 2 2 ging gaan VERB
#> 3 9 10 3 3 op op ADP
#> 4 12 15 4 4 reis reis NOUN
#> 5 17 18 5 5 en en CCONJ
#> 6 20 21 6 6 ik ik PRON
#> 7 23 25 7 7 nam nemen VERB
#> 8 27 29 8 8 mee mee ADP
#> 9 30 30 9 9 : : PUNCT
#> 10 32 35 10 10 mijn mijn PRON
#> 11 37 42 11 11 laptop laptop NOUN
#> 12 43 43 12 12 , , PUNCT
#> 13 45 48 13 13 mijn mijn PRON
#> 14 50 58 14 14 zonnebril zonnebril NOUN
#> 15 60 61 15 15 en en CCONJ
#> 16 63 66 16 16 goed goed ADJ
#> 17 68 73 17 17 humeur humeur NOUN
#> 18 74 74 18 18 . . PUNCT
#> xpos
#> 1 VNW|pers|pron|nomin|vol|1|ev
#> 2 WW|pv|verl|ev
#> 3 VZ|init
#> 4 N|soort|ev|basis|zijd|stan
#> 5 VG|neven
#> 6 VNW|pers|pron|nomin|vol|1|ev
#> 7 WW|pv|verl|ev
#> 8 VZ|fin
#> 9 LET
#> 10 VNW|bez|det|stan|vol|1|ev|prenom|zonder|agr
#> 11 N|soort|ev|basis|zijd|stan
#> 12 LET
#> 13 VNW|bez|det|stan|vol|1|ev|prenom|zonder|agr
#> 14 N|soort|ev|basis|zijd|stan
#> 15 VG|neven
#> 16 ADJ|prenom|basis|zonder
#> 17 N|soort|ev|basis|onz|stan
#> 18 LET
#> feats head_token_id dep_rel deps
#> 1 Case=Nom|Person=1|PronType=Prs 2 nsubj <NA>
#> 2 Number=Sing|Tense=Past|VerbForm=Fin 0 root <NA>
#> 3 <NA> 4 case <NA>
#> 4 Gender=Com|Number=Sing 2 obl <NA>
#> 5 <NA> 7 cc <NA>
#> 6 Case=Nom|Person=1|PronType=Prs 7 nsubj <NA>
#> 7 Number=Sing|Tense=Past|VerbForm=Fin 2 conj <NA>
#> 8 <NA> 7 compound:prt <NA>
#> 9 <NA> 7 punct <NA>
#> 10 Person=1|PronType=Prs 11 nmod:poss <NA>
#> 11 Gender=Com|Number=Sing 7 obj <NA>
#> 12 <NA> 14 punct <NA>
#> 13 Person=1|PronType=Prs 14 nmod:poss <NA>
#> 14 Gender=Com|Number=Sing 11 conj <NA>
#> 15 <NA> 17 cc <NA>
#> 16 Degree=Pos 17 amod <NA>
#> 17 Gender=Neut|Number=Sing 11 conj <NA>
#> 18 <NA> 2 punct <NA>
#> misc
#> 1 <NA>
#> 2 <NA>
#> 3 <NA>
#> 4 <NA>
#> 5 <NA>
#> 6 <NA>
#> 7 <NA>
#> 8 SpaceAfter=No
#> 9 <NA>
#> 10 <NA>
#> 11 SpaceAfter=No
#> 12 <NA>
#> 13 <NA>
#> 14 <NA>
#> 15 <NA>
#> 16 <NA>
#> 17 SpaceAfter=No
#> 18 SpacesAfter=\\nCreated on 2020-09-14 by the reprex package (v0.3.0)
If you just want the model to be stored in a file with your name in the tempdir, why don't you just do
f <- udpipe_download_model("english", model_dir = tempdir(), overwrite = FALSE)
temp_file <- tempfile()
file.copy(from = f$file_model, to = temp_file)
model <- udpipe_load_model(temp_file)
x <- setNames(c("Just showing an example", "And another one"), c("a", "b"))
udpipe(x = x, object = model)
I see you are basically interested in textrecipes in the tokenizer part as you recently added tokenizers.bpe.
Just a note: R package sentencepiece is a good candidate for tokenisation in textrecipes as well
Next - R package word2vec is great for wordvectors instead of using Glove.
And even crfsuite for named entity recognition.
But all these require of course building a model - which is not what textrecipes is about - seems to me more like tooling to prepare your data for modelling.
And another note when looking at your code at https://github.com/tidymodels/textrecipes/blob/9fd38e2f68c4cd5fe448613841cdd409cc111e56/R/tokenizer-udpipe.R
I don't know when the function generator in textrecipes is used but know that if you use udpipe::udpipe by default it looks up an already loaded model in .loaded_models https://github.com/bnosac/udpipe/blob/master/R/pkg.R#L11 such that it does not need to be loaded again. You code seems to be have to be reloading each time the model when the function generator will be called.
I need the file to be stored in the R object itself. Then I simply write it to a temp file right before udpipe tries to load it. Works great now.
The way I make it work was by having the user explicitly pass the model to step_tokenize()
library(textrecipes)
library(recipes)
library(modeldata)
data(okc_text)
library(udpipe)
udmodel <- udpipe_download_model(language = "english")
#> Downloading udpipe model from https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.4/master/inst/udpipe-ud-2.4-190531/english-ewt-ud-2.4-190531.udpipe to /private/var/folders/m0/zmxymdmd7ps0q_tbhx0d_26w0000gn/T/RtmpiJQ34o/reprexf49131fb9589/english-ewt-ud-2.4-190531.udpipe
#> Visit https://github.com/jwijffels/udpipe.models.ud.2.4 for model license details
aa <- recipe(~ essay0, data = okc_text) %>%
step_tokenize(all_predictors(),
engine = "udpipe",
training_options = list(model = udmodel)) %>%
prep() %>%
bake(new_data = NULL)
aa$essay0
#> <textrecipes_tokenlist[750]>
#> [1] [227 tokens] [41 tokens] [458 tokens] [87 tokens] [90 tokens]
#> ...
#> [746] [146 tokens] [72 tokens] [346 tokens] [119 tokens] [56 tokens]
#> # Unique Tokens: 9965Created on 2020-09-15 by the reprex package (v0.3.0)
Thank you, both {sentencepiece} and {word2vec} is on the shortlist for next release.
👍