Running Inference

[Krishna Deepak]

Hi, I am using your multi-label classification notebook as reference for my learning. I was wondering what would be the efficient way to run inference on these trained models. Any referece to the code if any would be appreciated.

Thanks

[Abhishek Kumar Mishra]

Hi @nkdeepak , i hope your fining the notebooks useful in your learning.

As far as the inference is concerned, i have not gotten a chance to work on them as of yet. Planning to roll them in by end of next month. Meanwhile, you can save your trained models, so that you can use them in the future when we have inference scripts ready.

[Dezoito]

@abhimishra91,

First of all thank you very much for making your notebooks available... I am learning a lot from them.

I am currently working on running inference in the multi-label classification notebook (and having some issues with the shape of the inputs).

Would it be OK for me to post the relevant part of the code here, for discussion and troubleshooting?

[Dezoito]

@nkdeepak , If you are still interested, here's my take on running predictions.
Just add this in a cell at the end of the notebook

import regex
from sklearn.preprocessing import MultiLabelBinarizer
model.eval()

# Defines list of possible labels in results in the same order they were indexed
labels = ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']
mlb = MultiLabelBinarizer(sparse_output=False, classes=labels)

def pre_process(text):
    """ Not really needed here """
    # Removes extra line breaks
    text = regex.sub('\n', ' ', text)

    # removes extra white-spaces
    text = regex.sub(' +', ' ', text)
    return text


def get_prediction(text_input):
    # tokenizer has been defined above as 
    # BertTokenizer.from_pretrained(BERT_MODEL) 
    inputs = tokenizer.encode_plus(
        text_input,
        None,
        add_special_tokens=True,
        max_length=MAX_LEN,
        truncation=True,
        pad_to_max_length=True,
        return_token_type_ids=True
    )

    ids = inputs['input_ids']
    mask = inputs['attention_mask']
    token_type_ids = inputs["token_type_ids"]

    # We have to add another dimension to the tensors
    # hence the [ids], [mask], etc...
    ids = torch.tensor([ids], dtype=torch.long)    
    mask = torch.tensor([mask], dtype=torch.long)    
    token_type_ids = torch.tensor([token_type_ids], dtype=torch.long)

    ids = ids.to(device, dtype = torch.long)
    mask = mask.to(device, dtype = torch.long)
    token_type_ids = token_type_ids.to(device, dtype = torch.long)
    outputs = model(ids, mask, token_type_ids)
    outputs = outputs >= 0.5    

    # fit possible labels to y
    mlb.fit(outputs)

    # Decode prediction array (returns tuple with categories)
    prediction = mlb.inverse_transform(outputs.cpu())[0]
    return prediction

def predict(text):
    text_input = pre_process(text)
    prediction = get_prediction(text_input)
    return prediction

print(predict("I hate you and I hope you die"))  # ('toxic', 'threat')
print(predict("Bunnies are cute!"))  # ()
print(predict("Have a nice day, a*****e!"))  # ('toxic', 'obscene', 'insult')
print(predict("Green people are horrible"))  # ()
print(predict("Your mommy is ugly"))  # ('toxic', 'insult')

I admit this could be cleaner, but it's just a first attempt.
Any help in making this more efficient is welcome.

[Abhishek Kumar Mishra]

@dezoito thank you for chipping in and contributing!

I am now working on a template for inference scripts for the NLP tasks, hope to add some notebooks for that in next few days. Meanwhile feel free to use this repo for discussion, issues and questions.

Keeping thew issue open until i have uploaded the inference notebooks.

[Abhishek Kumar Mishra]

HI @nkdeepak and @dezoito , an update on this issue. In terms of loading model and inference you can check how i have used the models in my other project. Project Insight.

Have a look at any of the backend services code in the src_fastapi directory. I am leveraging the models trained with these notebooks in that project.

[Dezoito]

HI @nkdeepak and @dezoito , an update on this issue. In terms of loading model and inference you can check how i have used the models in my other project. Project Insight.

Have a look at any of the backend services code in the src_fastapi directory. I am leveraging the models trained with these notebooks in that project.

Certainly looks much cleaner than my implementation! ( I will rewrite after studying your code).

Finished a NER API last week using spaCy and FastAPI as well... both tools are a pleasure to work with.