Is it possible to load my own quantized model from local

[PRADEEP T]

Here is the code I tried for the coreference resolution

from allennlp.predictors.predictor import Predictor
model_url = 'https://storage.googleapis.com/pandora-intelligence/models/crosslingual-coreference/minilm/model.tar.gz'
predictor = Predictor.from_path(model_url)  
text = "Eva and Martha didn't want their friend Jenny \
    to feel lonely so they invited her to the party."
prediction = predictor.predict(document=text)  
print(prediction['clusters'])  
print(predictor.coref_resolved(text))  

And it worked well I got the output with solved coreference. like below

Eva and Martha didn't want Eva and Martha's friend Jenny     to feel lonely so Eva and Martha invited their friend Jenny to the party.

Now I have quantized the model used here (https://storage.googleapis.com/pandora-intelligence/models/crosslingual-coreference/minilm/model.tar.gz) and the new quantized model is stored in a specific path in my local machine.

Shall I use that customized(quantized) model from my local path in model_url value and use this prediction command like below?

model_url = <Path to the quantized model in my local machine>
predictor = Predictor.from_path(model_url)  

[Akshita Bhagia]

@pradeepdev-1995 In general, yes, a local path should work. The model archive needs to contain an allennlp config.json and a weights.th file. Additionally, if your model code is changed, it should be imported with the correct registered name, which is referenced in the config file.

I'll be happy to assist further if you run into errors with this.

[PRADEEP T]

Thank you @AkshitaB
But one doubt where can I see the weights.th and config.json file of allennlp for this

[Akshita Bhagia]

If you download the model archive tar file at model_url and unzip it, you will see the relevant files.

[PRADEEP T]

@AkshitaB
Okay. So the actual model config.json contains the model_name like this in 3 places

And you are suggesting that after quantization there should be these 2 files( config.json and weights.th) in the quantized model folder. Along with that config.json should contain the model_name value as a new quantized model path

Is it so?

[Akshita Bhagia]

@pradeepdev-1995 Yes. Essentially, your final model needs to be in a format that's readable by allennlp. For further details regarding config files and serialization, you can read the section on configuration files here: https://guide.allennlp.org/training-and-prediction#2 .

[PRADEEP T]

@AkshitaB
As you mentioned I have quantized the nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large model using dynamic quantization and kept new weigths.th file in the folder named quantized as you see here

FYI: I removed the new config.json file getting after the quantization because it conflicts with the allennlp format config.json file. Only weights were only taken after quantization. (Will it cause any issues?)

Also changed the model_name to the quantized model folder local path at 3 places in the allennlp config.json file as below

Then execute the below code

from allennlp.predictors.predictor import Predictor
model_url = '/home/Downloads/quantized'
predictor = Predictor.from_path(model_url)  
text = "Eva and Martha didn't want their friend Jenny \
    to feel lonely so they invited her to the party."
prediction = predictor.predict(document=text)  
print(prediction['clusters'])  
print(predictor.coref_resolved(text))  

But it shows the following error
FileNotFoundError: file /tmp/tmpj8o6jszw/config.json not found

[Akshita Bhagia]

@pradeepdev-1995 Your new model needs to be a registered name; an allennlp Model class object. That is what the Predictor will require.

So, something like:

from allennlp.model import Model
@Model.register("new-model-name")
class NewModelClass(Model):
    <create/load quantized version of model>

Then, in your config, you can specify the model name as "new-model-name".

Also see this guide chapter for more details on the config file.

[nemeer]

Hi,
With my quantized model, I have tried the following:

model_url = r"custom_model/model"
predictor = Predictor.from_path(model_url)

model_url points to directory including Vocabulary, config, meta files(3 of them before quantizing) and new quantized weights.

Unfortunately, I ran into a RuntimeError as follows:

raise RuntimeError(
RuntimeError: Error loading state dict for CoreferenceResolver
        Missing keys: []
        Unexpected keys: ['embeddings.position_ids', 'embeddings.word_embeddings.weight', ...

Could you please help me resolve this issue with quantized model?

Do I need to change any config parameters in config.json to support my quantized model(Please note that my model name remains the same after quantization, hence the quantized model is not registered)

[nemeer]

I had tried registering my quantized model as below:

@Model.register("quantized_model")
class QuantizedModel(Model):
    def __init__(self):
        super().__init__(serialization_dir="custom_model/model")

Also changed the model key in original config.json

"model": {
     "type": "quantized_model"
 },

Now, when I try running

model_url = r"custom_model/model"
predictor = Predictor.from_path(model_url)

got TypeError: __init__() missing 1 required positional argument: 'vocab'

Is the model registration correct? Are there any additional params required for registration and in config.json. Please mention