minio / md5-simd

Accelerate aggregated MD5 hashing performance up to 8x for AVX512 and 4x for AVX2. Useful for server applications that need to compute many MD5 sums in parallel.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

As more hashers are created, the speed slows down

youngk1019 opened this issue · comments

Hello, I encountered the same problem as this issue #33 (comment) in the production environment. I will create a large number of Hasher instances in the production environment, but there are not many existing Hasher instances at the same time. There are a large number of scenes of creation and destruction, and it is guaranteed that they will be successfully closed eventually, but my speed will continue to decrease, from 3GiB/s to 100MiB/s
I used this test case in this issue #33 (comment),

func TestSomething(t *testing.T) {
	var total int64
	var finish int64

	md5Server := md5simd.NewServer()
	n := 2000
	go func() {
		t := time.NewTicker(1 * time.Second)
		lastTime := time.Now()
		for range t.C {
			elapsed := time.Since(lastTime)
			processed := atomic.SwapInt64(&total, 0)
			finished := atomic.SwapInt64(&finish, 0)
			lastTime = time.Now()
			fmt.Printf("%0.2fGB/s finish:%d\n", float64(processed)/elapsed.Seconds()/float64(1<<30), finished)
		}
	}()

	var wg sync.WaitGroup
	wg.Add(n)
	for i := 0; i < n; i++ {
		go func() {
			defer wg.Done()
			buf := make([]byte, 1048569)
			for i := 0; i < len(buf); i++ {
				buf[i] = byte(i * 53)
			}
			for i := 0; i < 1000; i++ {
				time.Sleep(time.Duration(1500*rand.Float32()) * time.Millisecond)
				c := md5Server.NewHash()
				for j := 0; j < 15; j++ {
					c.Write(buf)
					atomic.AddInt64(&total, int64(len(buf)))
				}
				c.Sum(nil)
				c.Close()
				atomic.AddInt64(&finish, 1)
			}
		}()
	}
	wg.Wait()
}

and the final results are as follows:


=== RUN   TestSomething
2.62GB/s finish:0
3.05GB/s finish:0
3.49GB/s finish:0
3.48GB/s finish:0
2.88GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0
0.00GB/s finish:0

I suspect that it is due to competition, because when I try to replace Close when I can use Reset, the decline speed will be much slower in the production environment, but it will still decline to a very low speed in the end