Log filename and line number
gust1n opened this issue · comments
Both stdlib package log and many other logging frameworks are supporting this (e.g. https://github.com/inconshreveable/log15). Am I missing how to enable it or is it not supported?
There's currently no way of doing it. Would definitely be interested in a contribution adding this.
I was part of the discussion leading to this solution: inconshreveable/log15@dbf739e
Does it look good to you? Then I could start working on a PR in a couple of days. I guess the question os how to enable it, since we probably should let users decide whether to use it or not...
Hm.. it'd be nice if formatters and hooks could lazily load this if they need it, but I'm not sure if we can do that nicely. I don't want to call those functions and pass it down with the entry every time we log.
I'm interested in this as well. @sirupsen from your perspective how do you feel about adding some sort of a middleware layer that allows people to gather their own context and automatically append it to an entry's Fields?
That mechanism would allow us to add this sort of thing in quite easily. I am not sure about what other use cases this might also open up.
I wouldn't be opposed to adding it by default and then allowing the formatters/hooks to use it appropriately. This is also useful for exception tracking etc., however, it might have a performance impact. We'll need to measure that
Yes please, this feature is very much needed.
I like the idea of a middleware layer but agree with @sirupsen that this is not something I would want to occur in a production environment. If we're logging errors the existing ability to log detailed and helpful values should be sufficient; however, in a development environment quick access to filepath and line number is very valuable.
Thanks everyone for this discussion.
Yes, unless someone can prove a negligible performance impact I think this should be a hook.
I solve this in a performant manner by doing the following:
var log *logrus.Logger
func init() {
log = mylog.NewLogger()
}
func NewLogger() *logrus.Logger {
_, file, _, ok := runtime.Caller(1)
if !ok {
file = "unknown"
}
ret := logrus.New()
inst := &logInstance{
innerLogger: ret,
file: filepath.Base(file),
}
ret.Hooks.Add(inst)
return ret
}logInstance is a hook that adds "file" to the log entry. This saves me calling runtime.Caller each time I log a message while still having code that does
log.WithField()....Info()The disadvantage is that I only have one "log" per package that contains the filename of where it was created. This usually works enough for me since I use many packages. If not, I would give each file it's own log() variable.
+1 it's a MUST-HAVE, and saying "completely API compatible with the standard library logger" in the README is wrong as we have that feature in the standard Go logger (SetFlags) http://golang.org/pkg/log/#SetFlags
+1
Perhaps someone take a closer look at SetFlags implementation and port it to logrus? Currently all the implementations I've seen rely on hardcoding number of stack frames which is not very portable and doesn't work in some cases.
@sirupsen, I can definitely have a look :)
@chreble awesome, let me know what you find!
@sirupsen unfortunately, the SetFlags implementation in the standard library uses a (sort of) hardcoded number of stack frames: http://golang.org/src/log/log.go#L130. Each exported output function, such as Println, passes callDepth like this: http://golang.org/src/log/log.go#L169.
Is this such a big deal to implement in logrus?
@dmlyons no it's not, I've just not needed this myself and haven't seen an implementation that hasn't failed on different platforms.
Has anyone resolved this in a reasonable way? I just want the filename and line number. I'd also prefer to be able to configure it to only print on specific levels (e.g. err, panic, warn)
We have wrapped the stuff we most use from logrus with the file information, i don't know if you will find it useful or not but here you are https://gist.github.com/gonzaloserrano/d360c1b195f7313b6d19
+1
+1
I will try to take a swing at it, as @dmlyons said it is pretty straightforward to implement.
+1
+1
My solution was to use https://github.com/inconshreveable/log15
+1
Thanks @davisford. That's what I did as well.
I'm kind of confused why you couldn't just make your formatter, or wrap a formatter you already use, accomplish this. I do this for our production systems. The only downside is if logrus' internals change, it could possibly change the callstack. But, you can catch this early if you make a testcase for your formatter.
I would rather logrus not implement this by default because it has performance implications. Unless you provide a way for people to disable it if they don't need it, which just adds another API.
[edit]
Ah I see. The callstack for a hook is different than the callstack just for writing to Logger.Out. So unless you care about ouputting to all your hooks and Logger.Out, this method should work.
[edit2]
Just noticed the callstack for using WithFields and not is different. It's looking more and more that the best solution would be logrus to just support this feature.
The cleanest solution I came up with was just to mimic the export.go with my own logging logic to add caller line info. The interface is exactly the same as Sirupsen/logrus/logger.go so applications can easily switch back to this logger.
Hi, this might help
https://github.com/facebookgo/stack
+1
I proposed a merge that can log file name and line numbers. It is located in the master branch of the clone https://github.com/omidnikta/logrus
One can use ShowCaller(bool) to enable/disable logging file-line.
we need this ASAP, anyone working on this?
+1
@omidnikta where is that PR?
+1
Sent from my iPhone
On 09 Mar 2016, at 00:17, Boyan Dimitrov notifications@github.com wrote:
+1
—
Reply to this email directly or view it on GitHub.
+1
@apriendeau The pull request is
#312
But when I checked correctly the source of logrus, it is not thread safe at all, but the pull request is OK anyway. That is why I started https://github.com/omidnikta/nlog which is a minimal thread safe logger.
+1
+1
+1
+1
To avoid any more +1 comments (which BTW GitHub implemented reactions to avoid this) @apriendeau has this work done in a PR that is being reviewed right now.
-_-...... 2016.7.21 this feature remains fail..
I am using this as workaround while developing
Edit: Just noticed it doesn't work with WithField because of the skip parameter of runtime.Caller.
type ContextHook struct {
}
func (hook ContextHook) Levels() []log.Level {
return log.AllLevels
}
func (hook ContextHook) Fire(entry *log.Entry) error {
if pc, file, line, ok := runtime.Caller(8); ok {
funcName := runtime.FuncForPC(pc).Name()
entry.Data["file"] = path.Base(file)
entry.Data["func"] = path.Base(funcName)
entry.Data["line"] = line
}
return nil
}
func main() {
log.AddHook(ContextHook{})
…
}@renke seems can work
Modified @renke's workaround a bit so it works for WithField.
type ContextHook struct{}
func (hook ContextHook) Levels() []logrus.Level {
return logrus.AllLevels
}
func (hook ContextHook) Fire(entry *logrus.Entry) error {
pc := make([]uintptr, 3, 3)
cnt := runtime.Callers(6, pc)
for i := 0; i < cnt; i++ {
fu := runtime.FuncForPC(pc[i] - 1)
name := fu.Name()
if !strings.Contains(name, "github.com/Sirupsen/logrus") {
file, line := fu.FileLine(pc[i] - 1)
entry.Data["file"] = path.Base(file)
entry.Data["func"] = path.Base(name)
entry.Data["line"] = line
break
}
}
return nil
}
func main() {
logrus.AddHook(ContextHook{})
....
}It's more expensive however since it allocates memory every time.
My solution is to call a function on each logging instance.
That way you can take it out for entries that are performance sensitive.
func Locate(fields logrus.Fields) logrus.Fields {
_, path, line, ok := runtime.Caller(1)
if ok {
_, file := filepath.Split(path)
fields["file"] = file
fields["line"] = line
}
return fields
}
...
log.WithFields(Locate(logrus.Fields{
"url": url,
})).Info("Bound management service")
Is it still not printing the stacktrace? Any way to get output like in std error stacktrace?
+1
my slotion is add a hook
https://github.com/zhl11b/logrus-hook-caller
it can check entry.data make sure get a right caller
In case no one has seen the Go 1.7 release notes yet, a new function was added to the runtime package which has improvements over FuncForPC:
The new function CallersFrames translates a PC slice obtained from Callers into a sequence of frames corresponding to the call stack. This new API should be preferred instead of direct use of FuncForPC, because the frame sequence can more accurately describe call stacks with inlined function calls.
Go 1.7 release notes
Here's the godoc: https://golang.org/pkg/runtime/#CallersFrames
With cues from here and facebookgo/stack's structs combining file, line and func name, I've written a hook that is working well for me: https://github.com/Gurpartap/logrus-stack 🎯. Provides the last "caller" as well as the "stack" trace.
A race condition occurs when writing to entry.Data inside the hook.
Is it possible to prevent the race condition from occurring?
Race condition point:
// Inside the hook.
entry.Data["line"] = line
// https://github.com/Sirupsen/logrus/blob/a437dfd2463eaedbec3dfe443e477d3b0a810b3f/text_formatter.go#L61
var keys []string = make([]string, 0, len(entry.Data))
+1
+1
I was able to make my own functionality with what they have currently and borrowing code from others that have solved their own issues with this. I didn't like having to deorate each caller with a wrapped function, so I made my own Decorate(logger *logrus.Entry) func() *logrus.Entry that works for me.
package logger
import (
"fmt"
"net/http"
"runtime"
"strings"
"github.com/Sirupsen/logrus"
)
/* truncated */
// Decorate appends line, file and function context to the logger and returns a function to call before
// each log
func Decorate(logger *logrus.Entry) func() *logrus.Entry {
return func() *logrus.Entry {
if pc, f, line, ok := runtime.Caller(1); ok {
fnName := runtime.FuncForPC(pc).Name()
file := strings.Split(f, "mobilebid")[1]
caller := fmt.Sprintf("%s:%v %s", file, line, fnName)
return logrus.WithField("caller", caller)
}
return logger
}
}
// Log appends line, file and function context to the logger
func Log() *logrus.Entry {
if pc, f, line, ok := runtime.Caller(1); ok {
fnName := runtime.FuncForPC(pc).Name()
file := strings.Split(f, "mobilebid")[1]
caller := fmt.Sprintf("%s:%v %s", file, line, fnName)
return logrus.WithField("caller", caller)
}
return &logrus.Entry{}
}In use:
// Using context
logctx := logger.Decorate(WithField("remote_address", conn.RemoteAddr()))
logctx().WithField("test", "works").Info("it works fine")
// Standard logging
logger.Log().WithError(err).Error("error upgrading websocket")
// With multiple fields
logger.Log().WithFields(logrus.Fields{
"field": "something",
"field2": "something else",
}).WithError(err).Error("error upgrading websocket")
will this function to be added
after three years, still doesn't have this functionality?
+1
Since this is probably never going to be added or merged, it would be great if the authors could give their blessing to a plugin or a workaround to achieve this.
ps. Thank you for the awesome library!
Any news on this ? 😃
It's very important to log filename and line number. This can make it easier for engineers to debug.
Could you please let us know whether this feature will be implemented?
Thanks a lot.
I don't know if that's helpful but the prometheus team did it like this
+1
is there a solution merged to this ?
Actually I found a workaround not full but something for this, using package:
github.com/x-cray/logrus-prefixed-formatter
And in code it looks like:
var log = logrus.WithFields(logrus.Fields{"prefix": "main"})
func init() {
logrus.SetFormatter(&prefixed.TextFormatter{})
logrus.SetLevel(logrus.DebugLevel)
}After that you just need to add in each package somethink like:
var log = logrus.WithFields(logrus.Fields{"prefix": "package"})In logs it looks like:
[0000] INFO cmds: Sending ping to server: 127.0.0.1
[0000] DEBUG client: Establishing grpc connection with: 127.0.0.1Hopefully it helps :)
@damekr How does that solve the issue of logging the file name and line number?
@mihaitodor Instead of {"prefix": "main"} you can use filename. It is not a full solution as I said, but can help a bit to solve the problem.
Ah, I see. Thanks!
@damekr I'm using filename. When you say that is not a full solution, what do you mean ?
filenameHook := filename.NewHook()
filenameHook.Field = filenameField
log.AddHook(filenameHook)
{"level":"info","message":"test","origem":"mobileServer/server.go:65","time":"2017-12-07T09:54:28.449479945-02:00"}
It works fine for me. Should I be aware of something ?
@RenathoAzevedo No, it's just another way to do that :)
@damekr Thank you, I'm still wating for 'native' solution from logrus.
Dear logrus maintainers, it's self-evident that the community would like a simple and critical feature such as filename and line number logging to be native in the logrus library. It's much needed and we'd all greatly appreciate this feature. Thanks!
+1
package main
import (
"encoding/json"
"runtime"
log "github.com/sirupsen/logrus"
)
type MyJSONFormatter struct {
Level string `json:"level"`
Info string `json:"info"`
Msg string `json:"msg"`
Time string `json:"time"`
File string `json:"file"`
Line int `json:"line"`
}
func (f *MyJSONFormatter) Format(entry *log.Entry) ([]byte, error) {
logrusJF := &(log.JSONFormatter{})
bytes, _ := logrusJF.Format(entry)
myF := MyJSONFormatter{}
json.Unmarshal(bytes, &myF)
_, file, no, _ := runtime.Caller(7)
myF.File = file
myF.Line = no
return json.Marshal(myF)
}
func main() {
log.SetFormatter(&MyJSONFormatter{})
log.Infoln("fire")
}
So any updated on this feature? Is this available now? I walk through comments, it seems nothing is official yet?
@RenathoAzevedo in the example where you mention filename, where is that package? If possible could you post a full example of your solution?
Hello @andytsnowden, I'm using filename hook from https://github.com/onrik/logrus, It's quite simple to use.
filenameHook := filename.NewHook()
filenameHook.Field = "source"
logrus.AddHook(filenameHook)
Jesus Christ, it still doesn't supports a common feature after four years.
Please keep it civil. I think the maintainers voiced valid concerns.
I have reviewed one or two PRs that propose adding this feature and most seemed to add a cost whether your using it or not. If there is a PR with this functionality, that doesn't create a performance cost, point me to and I'll have a look.
I think #450 was looking promising, although it may need the additional file and line number.
I am using https://github.com/onrik/logrus as well for printing file names and lines:
package main
import (
"github.com/onrik/logrus/filename"
log "github.com/sirupsen/logrus"
)
func main(){
log.AddHook(filename.NewHook())
...
log.Println(something)
logs look like this:
INFO[0000] 404 Not Found source="scraper/main.go:46"
INFO[0000] 404 Not Found source="scraper/main.go:46"
#450 shouldn't add a cost unless the flag is set to enable the extra data. There's some (very small) cost in gathering the data, but we've been working from my fork in production for over a year now with no issues seen. I am looking forward to getting back on mainline Logrus someday, though :)
If there's more data desired than what I've added there, we can of course continue to evolve...
@stevvooe I've resolved the merge conflict in that PR, FYI.
I would also appreciate this feature (as everybody else does) there are so many proposals and pull requests. Whats actually blocking this issue?
Are you just afraid of cross platform bugs?
I will have a look at this pr soon.
4 year open-issues ~~~
4 year open-issues ~~~
+1, any update here? I check the solutions in the comments. I want to get the official answer about the hook, is there some other concern? Like performance issues?
@Jing-Li I've been using the solution posted here: #63 (comment) in production for about 6 months now and haven't really had any issues at all. In my use-case we emit about 10m lines of logging daily and I really haven't seen logging be a "slow" point.
up
Dear all, this feature has been waited for a long time. I hope you'll all enjoy the recent merge of #850 which should provide the implementation of this feature. I'll let this issue open still a few weeks in order to gather feedback from users.
Right now the implementation is available on the tip of master and the new interface to enable the feature can be considered as stable.
I will probably release a v1.2.0 verison at the beginning of november.
Thanks for your patience.
Thanks to @dclendenan which has provided most of the code which has made this feature available.
Yay, but please do provide instructions here (or in docs) on how #850 gives us file and line number. It's certainly not apparent from a quick view.
Here's how to have logrus log the filename and line number with the latest release. Basically just add log.SetReportCaller(true) when initializing.
Example:
package main
import (
log "github.com/sirupsen/logrus"
)
func main() {
// Add this line for logging filename and line number!
log.SetReportCaller(true)
log.Println("hello world")
}Output:
INFO[0000]/Users/bob/go/src/github.com/bob/test/main.go:17 main.main() hello world@rfay the pull request contains a change in the readme which describes how to enable this feature.
This feature is contained in v1.2.0
Here's how to have logrus log the filename and line number with the latest release. Basically just add
log.SetReportCaller(true)when initializing.Example:
package main import ( log "github.com/sirupsen/logrus" ) func main() { // Add this line for logging filename and line number! log.SetReportCaller(true) log.Println("hello world") }Output:
INFO[0000]/Users/bob/go/src/github.com/bob/test/main.go:17 main.main() hello world
Hi all, is there a way to cut the longest file path? like just to display the
INFO[0000]/test/main.go:17 main.main() hello worldThanks...QQ
@Jing-Li I've been using the solution posted here: #63 (comment) in production for about 6 months now and haven't really had any issues at all. In my use-case we emit about 10m lines of logging daily and I really haven't seen logging be a "slow" point.
@andytsnowden , this helps a lot, thanks.
Here's how to have logrus log the filename and line number with the latest release. Basically just add
log.SetReportCaller(true)when initializing.
Example:package main import ( log "github.com/sirupsen/logrus" ) func main() { // Add this line for logging filename and line number! log.SetReportCaller(true) log.Println("hello world") }Output:
INFO[0000]/Users/bob/go/src/github.com/bob/test/main.go:17 main.main() hello worldHi all, is there a way to cut the longest file path? like just to display the
INFO[0000]/test/main.go:17 main.main() hello worldThanks...QQ
any idea on how to cut file name ?
@Wamani please don't comment on closed issue, because it's kind of hard to find it back.
The basic filter on the github interface doesn't show closed issues whatever how recently they have been updated. The next time open a new one, it will be easier for maintainers to provide feedback and guidance.
That said if you want to customiza the output of the caller field you can give a specific formatting function in the CallerPrettyfier field of the JSONFormatter (https://godoc.org/github.com/sirupsen/logrus#JSONFormatter) or of the TextFormatter (https://godoc.org/github.com/sirupsen/logrus#TextFormatter).
You also have an example in the logrus repository on how to use that with the JSONFormatter here
https://github.com/sirupsen/logrus/blob/master/example_custom_caller_test.go