is_return_output_log_probs doesn't return logits for T5 model
swairshah opened this issue · comments
swairshah commented
Branch/Tag/Commit
main
Docker Image Version
763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-ec2
GPU name
A10
CUDA Driver
450.119.03
Reproduced Steps
For any FasterTransformer t5 model hosted on Sagemaker using the DJL images https://sagemaker.readthedocs.io/en/stable/frameworks/djl/sagemaker.djl_inference.html#fastertransformermodel (the model i am using is Flan-t5-xxl) pass the config:
= {
"max_length" : 64,
"min_length" : 0,
"temperature" : 0,
"is_return_output_log_probs" : True,
}
It only returns generated text and not the logits.