torch_dtype only for torch.float16？

Question

yumianhuli1 opened this issue 4 months ago · comments

Does inference currently only support torch_dtype=torch.float16？int8_float16、int8 will be supported？