Broker hang if multiple goroutines make calls
casskir opened this issue · comments
Describe the bug
Sometimes when we make calls from multiple goroutines broker is hanged. Also, a timeout error is not fired.
To Reproduce
Make calls from multiple goroutines. Often reproduced on a slow machine.
broker := broker.New(config)
for t := 0; t < 10000; t++ {
wg := sync.WaitGroup{}
wg.Add(3)
go func() {
defer wg.Done()
<-broker.Call(action, params)
log.Info("first done")
}()
go func() {
defer wg.Done()
<-broker.Call(action, params)
log.Info("second done")
}()
go func() {
defer wg.Done()
<-broker.Call(action, params)
log.Info("third done")
}()
wg.Wait()
}
Expected behavior
All requests successfully finished or execution timeout error returned.
Additional context
My investigation showed that multiple async calls generate requests with the same ID:
And the main problem in:
moleculer/context/contextFactory.go
Line 51 in b092546
This function using random generator that unsafe for multiple goroutines:
// NewSource returns a new pseudo-random Source seeded with the given value.
// Unlike the default Source used by top-level functions, this source is not
// safe for concurrent use by multiple goroutines.
Another way how reproduce:
t := make([]string, 1000)
wg := sync.WaitGroup{}
wg.Add(1000)
for i := 0; i < 1000; i++ {
go func(idx int) {
defer wg.Done()
t[idx] = util.RandomString(12)
}(i)
}
wg.Wait()
v := map[string]interface{}{}
for i := 0; i < len(t); i++ {
v[t[i]] = nil
}
fmt.Printf("Uniq: %d", len(v))
generates less than 1000 unique strings.