两个不同类型的tensor相加，不同类型的case GPU行为不一致

[Jianbang Yang]

bug描述 Describe the Bug

GPU case1：

import paddle

a = paddle.randn((3, 5)).astype("float64")
b = paddle.randn((3, 5)).astype("float32")
print("a:", a)
print("b:", b)
a.add_(b)
print("a:", a)

GPU case1结果：

a: Tensor(shape=[3, 5], dtype=float64, place=Place(gpu:0), stop_gradient=True,
       [[-1.04869020, -0.80085981, -0.72781432, -1.06803286, -0.40499219],
        [ 0.47871953, -1.56587374,  0.33300665,  0.52691936,  1.70032883],
        [ 1.29407358, -0.35217470,  1.04975128, -0.96133667, -0.75396192]])
b: Tensor(shape=[3, 5], dtype=float32, place=Place(gpu:0), stop_gradient=True,
       [[ 0.86691004,  1.47818065, -2.18001342, -0.49277630,  1.47790968],
        [-0.08248178, -1.38634968, -0.89387017, -0.62922710,  0.12351245],
        [ 0.65899324,  0.73825586, -0.36215657,  0.22549616, -1.83211803]])
a: Tensor(shape=[3, 5], dtype=float32, place=Place(gpu:0), stop_gradient=True,
       [[36893488147419103232., -0.40290558          , -4.18001366          ,
         -2.31799126          ,  3.47790956          ],
        [-1.88943529          , -1.38634968          , -2.77737427          ,
         36893488147419103232., -1.57898355          ],
        [ 2.65899324          ,  2.47761559          , -0.36215657          ,
         -1.72023797          , -1.83211803          ]])

GPU case2：

import paddle

a = paddle.randn((3, 5)).astype("float")
b = paddle.randn((3, 5)).astype("float16")
print("a:", a)
print("b:", b)
a.add_(b)
print("a:", a)

GPU case2结果：

a: Tensor(shape=[3, 5], dtype=float32, place=Place(gpu:0), stop_gradient=True,
       [[ 1.28292370,  0.60509706, -0.07252523,  1.22371161,  0.36248189],
        [-2.08635950, -0.70022446,  0.05288927, -0.39726844,  0.83958137],
        [ 0.61249077, -0.84752667,  0.58558685,  0.04670615,  0.99223375]])
b: Tensor(shape=[3, 5], dtype=float16, place=Place(gpu:0), stop_gradient=True,
       [[-0.03347778, -0.20471191, -1.27929688,  1.64453125, -1.55664062],
        [ 0.24682617,  0.54443359, -1.91796875, -0.78662109,  0.99267578],
        [-0.60351562, -1.16406250,  0.27539062,  0.68164062, -0.15222168]])
a: Tensor(shape=[3, 5], dtype=float32, place=Place(gpu:0), stop_gradient=True,
       [[ 1.24944592,  0.40038514, -1.35182214,  2.86824274, -1.19415879],
        [-1.83953333, -0.15579087, -1.86507952, -1.18388951,  1.83225715],
        [ 0.00897515, -2.01158905,  0.86097747,  0.72834677,  0.84001207]])

可以看到，tensor_f64.add_(tensor_f32)，tensor_fp64的类型会被改为float32，而tensor_f32.add_(tensor_fp16)，tensor_f32的类型不会被改变

请问上述行为是否合理？设计的行为是什么？

参考：

• PyTorch的Type Promotion

其他补充信息 Additional Supplementary Information

No response

[Jianbang Yang]

Paddle目前似乎不支持Type Promotion，GPU fp32 + fp16/bf16的case是为了master_grad支持的。

参考：

• #54415
• #57309