error.callers holds on to more memory than needed

[Sunil Thaha]

We were analysing an OOM case and found that errors.callers is holding on to more memory than needed.
E.g.

Given this convoluted testcase

package main

import (

	// pkgerrs "github.com/pkg/errors"
	pkgerrs "emperror.dev/errors"
	"github.com/pkg/profile"
)

var all = []error{}

// go:noinline
func foobarbaz() {
	foobar()

}

// go:noinline
func foobar() {
	foo()
}

// go:noinline
func foo() {
	moo()
}

// go:noinline
func moo() {
	mootoo()
}

// go:noinline
func mootoo() {
	all = append(all, pkgerrs.New("foo"))
}

func main() {
	defer profile.Start(
		profile.MemProfile,
		profile.ProfilePath("."),
	).Stop()

	count := 10_000_000
	for i := 0; i < count; i++ {
		foobarbaz()
	}
}

❯ go run main.go && go tool pprof mem.pprof
2022/02/23 10:20:33 profile: memory profiling enabled (rate 4096), mem.pprof
(pprof) top 3
Showing nodes accounting for 1172.78MB, 95.29% of 1230.71MB total; Dropped 17 nodes (cum <= 6.15MB)
Showing top 3 nodes out of 11
      flat  flat%   sum%        cum   cum%
 1020.92MB 82.95% 82.95%  1020.92MB 82.95%  emperror.dev/errors.callers
   87.18MB  7.08% 90.04%  1108.10MB 90.04%  emperror.dev/errors.WithStackDepth (inline)
   64.68MB  5.26% 95.29%  1230.67MB   100%  main.mootoo

(pprof) list emperror.dev/errors.callers
Total: 1.20GB
ROUTINE ======================== emperror.dev/errors.callers in memleaks/vendor/emperror.dev/errors/stack.go
 1020.92MB  1020.92MB (flat, cum) 82.95% of Total
         .          .     56:func callers(depth int) *stack {
         .          .     57:   const maxDepth = 32
         .          .     58:
  933.54MB   933.54MB     59:   var pcs [maxDepth]uintptr
         .          .     60:
         .          .     61:   n := runtime.Callers(2+depth, pcs[:])
         .          .     62:
   87.39MB    87.39MB     63:   var st stack = pcs[0:n]
         .          .     64:
         .          .     65:   return &st
         .          .     66:}

Here you can see that pcs escapes to heap which is confirmed by

❯ go build -gcflags='-m -m' vendor/emperror.dev/errors/stack.go 2>&1 | grep pcs
vendor/emperror.dev/errors/stack.go:59:6: pcs escapes to heap:
vendor/emperror.dev/errors/stack.go:59:6:   flow: st = &pcs:
vendor/emperror.dev/errors/stack.go:59:6:     from pcs (address-of) at vendor/emperror.dev/errors/stack.go:63:20
vendor/emperror.dev/errors/stack.go:59:6:     from pcs[0:n] (slice) at vendor/emperror.dev/errors/stack.go:63:20
vendor/emperror.dev/errors/stack.go:59:6:     from st = pcs[0:n] (assign) at vendor/emperror.dev/errors/stack.go:63:6
vendor/emperror.dev/errors/stack.go:59:6: moved to heap: pcs

This leads to an error holding onto maxDepth = 32 uintptr than n . The fix is to ensure that pcs doesn't escape to heap which can be achieved by the following.

func callers(depth int) *stack {
	const maxDepth = 32

	var pcs [maxDepth]uintptr

	n := runtime.Callers(2+depth, pcs[:])
	st := make(stack, n)
	copy(st, pcs[:n])

	return &st
}

With this change the pprof for the testcase above shows a deduction of 75%.

(pprof) top 3
Showing nodes accounting for 599.08MB, 90.21% of 664.12MB total
Dropped 16 nodes (cum <= 3.32MB)
Showing top 3 nodes out of 11
      flat  flat%   sum%        cum   cum%
  355.93MB 53.59% 53.59%   355.93MB 53.59%  emperror.dev/errors.callers
  145.53MB 21.91% 75.51%   664.08MB   100%  main.mootoo
   97.62MB 14.70% 90.21%   453.55MB 68.29%  emperror.dev/errors.WithStackDepth (inline)
(pprof) list emperror.dev/errors.callers
Total: 664.12MB
ROUTINE ======================== emperror.dev/errors.callers in memleaks/vendor/emperror.dev/errors/stack.go
  355.93MB   355.93MB (flat, cum) 53.59% of Total
         .          .     57:   const maxDepth = 32
         .          .     58:
         .          .     59:   var pcs [maxDepth]uintptr
         .          .     60:
         .          .     61:   n := runtime.Callers(2+depth, pcs[:])
  355.93MB   355.93MB     62:   st := make(stack, n)
         .          .     63:   copy(st, pcs[:n])
         .          .     64:
         .          .     65:   return &st
         .          .     66:}

[Márk Sági-Kazár]

Nice catch @sthaha. Thanks for providing a fix as well!

[Márk Sági-Kazár]

Fixed in #28

[Márk Sági-Kazár]

Released in 0.8.1

[Sunil Thaha]

Thank you for accepting the patch 🙇