[BUG] Prediction code called from abstractive_summarization_bertsumabs_cnndm notebook is broken

[Arjun Singh]

Description

Tried using the abstractive_summarization_bertsumabs_cnndm notebook after installing the env based on the instructions, but the part where the predictions on the test data is performed returns an error
RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float'

How do we replicate the bug?

Followed the instructions here to create the environment on an Azure DS VM GPU
https://github.com/microsoft/nlp-recipes/blob/master/SETUP.md#dependencies-setup

Followed the instructions here to open JupyterHub and access this notebook from the repo.
https://github.com/microsoft/nlp-recipes/blob/master/SETUP.md#register-conda-environment-in-dsvm-jupyterhub

Get the following stack trace when executing the block for running predictions on the test data:

Generating summary:   0%|          | 0/32 [00:00<?, ?it/s]
dataset length is 32

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-9a677eb8bcb6> in <module>
     12 reference_summaries = [" ".join(t).rstrip("\n") for t in shortened_dataset.get_target()]
     13 generated_summaries = summarizer.predict(
---> 14     shortened_dataset, batch_size=BATCH_SIZE, num_gpus=NUM_GPUS
     15 )
     16 assert len(generated_summaries) == len(reference_summaries)

~/notebooks/Summarization/nlp-recipes/utils_nlp/models/transformers/abstractive_summarization_bertsum.py in predict(self, test_dataset, num_gpus, gpu_ids, local_rank, batch_size, alpha, beam_size, min_length, max_length, fp16, verbose)
    785         ):
    786             input = self.processor.get_inputs(batch, device, "bert", train_mode=False)
--> 787             translations, scores = predictor(**input)
    788 
    789             translations_text = generate_summary_from_tokenid(translations, scores)

/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    163 
    164         if len(self.device_ids) == 1:
--> 165             return self.module(*inputs[0], **kwargs[0])
    166         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    167         outputs = self.parallel_apply(replicas, inputs, kwargs)

/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~/notebooks/Summarization/nlp-recipes/utils_nlp/models/transformers/bertsum/predictor.py in forward(self, src, segs, mask_src)
    155         with torch.no_grad():
    156             predictions, scores = self._fast_translate_batch(
--> 157                 src, segs, mask_src, self.max_length, min_length=self.min_length
    158             )
    159             return predictions, scores

~/notebooks/Summarization/nlp-recipes/utils_nlp/models/transformers/bertsum/predictor.py in _fast_translate_batch(self, src, segs, mask_src, max_length, min_length)
    268             # Append last prediction.
    269             alive_seq = torch.cat(
--> 270                 [alive_seq.index_select(0, select_indices), topk_ids.view(-1, 1)], -1
    271             )
    272 

RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float'

Expected behavior (i.e. solution)

The notebook should run smoothly

Other Comments

[Saif Khan]

I am also getting the same error while trying to predict the summaries. Have you found any solution yet?

[Shane-Yuhe-Xin]

same , i have the same issue.

[AnliLiu]

try torch 1.5.0

[beomseok]

Adding a rounding_mode='trunc' parameter to get topk_beam_index would work.

topk_beam_index = topk_ids.div(vocab_size, rounding_mode='trunc')

[Bilal Ghanem]

try torch 1.5.0

I was using 1.9 and when I downgraded it to 1.5 this error disappear. Thanks!

[Dominic124]

is cuda11.1 compatible with torch1.5.0?
because i am using rtx3090 which is only compatible with 11.1 or higher.

[smile-yc]

在尝试预测摘要时，我也遇到了同样的错误。您找到任何解决方案了吗？

Have you solved this problem now？

[smile-yc]

same , i have the same issue.

have you solve it now?