sugarme / gotch

package main

import(
    "fmt"
    "github.com/sugarme/gotch/nn"
)

func main() {
	mat_3x5 := nn.NewRandnInit(0.0, 1.0).InitTensor([]int64{3, 5}, gotch.CPU)
	fmt.Printf("%v", mat_3x5)
}

output

   1.000     1.000     1.000     1.000     1.000
   1.000     1.000     1.000     1.000     1.000
   1.000     1.000     1.000     1.000     1.000

According to this line of code, when mean=0, all elements will be initialized with stdev

gotch/nn/init.go

Line 81 in 96b0967

data[i] = float32(rand.NormFloat64()*r.mean + r.stdev)

This is different from PyTorch's randn function

""" Returns a tensor filled with random numbers from a normal distribution
with mean `0` and variance `1`"""

torch.randn(3, 5)

############### output ###########################
tensor([[ 0.1769, -2.0933, -0.8882,  0.0051,  0.9833],
        [-0.6342,  0.4093,  0.6266,  0.3935,  0.2045],
        [ 0.3055, -0.4522, -1.7044,  1.8426,  0.4553]])

I know MustRandn can solve this problem, but NewRandnInit as the default initialization method of Embedding will directly affect the initialization weights.

Is this a bug or is it designed this way for some reason?

@FelixHo ,

Hopefully it's fixed. Thanks for pointing out.

NewRandnInit didn't works as expected when mean=0.0