Runtime panic when use logrus in AWS lambda environment.

[David Liu]

We used logrus in our AWS lambda code and found occasionally runtime panic in our log system.

Below are the runtime panic info:

runtime error: invalid memory address or nil pointer dereference: errorString
[{"path":"github.com/aws/aws-lambda-go@v1.17.0/lambda/function.go","line":36,"label":"(*Function).Invoke.func1"},{"path":"runtime/panic.go","line":965,"label":"gopanic"},{"path":"encoding/json/encode.go","line":328,"label":"(*encodeState).marshal.func1"},{"path":"runtime/panic.go","line":965,"label":"gopanic"},{"path":"runtime/panic.go","line":212,"label":"panicmem"},{"path":"runtime/signal_unix.go","line":734,"label":"sigpanic"},{"path":"encoding/json/encode.go","line":1033,"label":"(*encodeState).string"},{"path":"encoding/json/encode.go","line":647,"label":"stringEncoder"},{"path":"encoding/json/encode.go","line":360,"label":"(*encodeState).reflectValue"},{"path":"encoding/json/encode.go","line":716,"label":"interfaceEncoder"},{"path":"encoding/json/encode.go","line":813,"label":"mapEncoder.encode"},{"path":"encoding/json/encode.go","line":360,"label":"(*encodeState).reflectValue"},{"path":"encoding/json/encode.go","line":332,"label":"(*encodeState).marshal"},{"path":"encoding/json/stream.go","line":206,"label":"(*Encoder).Encode"},{"path":"github.com/sirupsen/logrus@v1.8.1/json_formatter.go","line":123,"label":"(*JSONFormatter).Format"},{"path":"github.com/sirupsen/logrus@v1.8.1/entry.go","line":279,"label":"(*Entry).write"},{"path":"github.com/sirupsen/logrus@v1.8.1/entry.go","line":251,"label":"(*Entry).log"},{"path":"github.com/sirupsen/logrus@v1.8.1/entry.go","line":293,"label":"(*Entry).Log"},{"path":"github.com/sirupsen/logrus@v1.8.1/entry.go","line":310,"label":"(*Entry).Info"},{"path":"api_management/openapi/util/util.go","line":295,"label":"(*SummaryLog).Summarize"},{"path":"openapi/openapi.go","line":196,"label":"openapi"},{"path":"reflect/value.go","line":476,"label":"Value.call"},{"path":"reflect/value.go","line":337,"label":"Value.Call"},{"path":"github.com/aws/aws-lambda-go@v1.17.0/lambda/handler.go","line":124,"label":"NewHandler.func1"},{"path":"github.com/aws/aws-lambda-go@v1.17.0/lambda/handler.go","line":24,"label":"lambdaHandler.Invoke"},{"path":"github.com/aws/aws-lambda-go@v1.17.0/lambda/function.go","line":69,"label":"(*Function).Invoke"},{"path":"reflect/value.go","line":476,"label":"Value.call"},{"path":"reflect/value.go","line":337,"label":"Value.Call"},{"path":"net/rpc/server.go","line":377,"label":"(*service).call"},{"path":"runtime/asm_amd64.s","line":1371,"label":"goexit"}]

I searched the issue list, is this a dup with #1284 or #1046

We used logrus 1.8.1 version and golang 1.16.7.

[David Bariod]

Can we have a minimal code sample which exhibits the issue ?
We can't do much with the information you've provided.

[David Liu]

Hello @dgsb

Thank you for your reply, yes, pls see the minimal code sample below:

package main

import (
	"context"
	"github.com/aws/aws-lambda-go/events"
	"github.com/pkg/errors"
	logs "github.com/sirupsen/logrus"
	"sync"
)

var (
	once            sync.Once
	mtx             sync.Mutex
)

func init(){
	once.Do(func() {
		logs.SetFormatter(&logs.JSONFormatter{})
	})
}

func entry(ctx context.Context, req events.S3Event) (*events.S3Event, error) {
	sl := &SummaryLog{}
	defer sl.Summarize()
	sl.RequestId = "request-id-xxx"
	sl.Error = errors.New("some error")
	return nil,nil
}

type SummaryLog struct {
	RequestId string `json:"requestid,omitempty"`
	Error     error  `json:"error,omitempty"`
}

func (summar *SummaryLog) Summarize() error {
	mtx.Lock()
	defer mtx.Unlock()
	fls := logs.Fields{}
	fls["requestId"] = summar.RequestId
	if summar.Error != nil {
		fls["Error"] = summar.Error.Error()
	}
	logs.WithFields(fls).Info()
	return nil
}

As the lambda code could be running in concurrency, we used mutex lock to protect it, but we still could see the errors mentioned above, hope it helps.

[David Bariod]

I think your locking pattern is incorrect as it involves a mutex copy as it isn't declared as a pointer.
you should replace:

var mtx sync.Mutex

by something like

var mtx = &sync.Mutex{}

[David Liu]

@dgsb , I don't think I copied the mutex, the mtx declared as a global var, I just used it in the summarize() method, could you please explain a little bit?

[David Liu]

FYI, Below are the mutex copy stack trace:
atal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_SemacquireMutex(0xc0001220ac, 0x0, 0x1)
/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc0001220a8)
/usr/local/go/src/sync/mutex.go:138 +0x105
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:81
......