Server runtime error when execute Watch command

Question

Server runtime error when execute Watch command

marouane-kech opened this issue 11 years ago · comments

I try to test the wath command with ( curl -L http://127.0.0.1:4001/v2/keys/message?wait=true ) and I do the same test with python-client. I got the same result:
the etcd instance crash with exeption :

panic serving 127.0.0.1:32907: runtime error: invalid memory address or nil pointer dereference
goroutine 758 [running]:
net/http.func·007()
/usr/local/go/src/pkg/net/http/server.go:1022 +0x9f
sync/atomic.AddUint64()
/usr/local/go/src/pkg/sync/atomic/asm_386.s:69 +0xc
github.com/coreos/etcd/store.(_watcherHub).watch(0x1868b4c0, 0x20186080, 0x8, 0x0, 0x23c, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/store/watcher_hub.go:71 +0x165
github.com/coreos/etcd/store.(_store).Watch(0x1868e480, 0x20186080, 0x8, 0x82a0400, 0x0, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/store/store.go:295 +0x151
github.com/coreos/etcd/server/v2.GetHandler(0x1e53be20, 0x1d2a3c80, 0x1d254310, 0x1ec206f0, 0x18691540, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/v2/get_handler.go:49 +0x455
github.com/coreos/etcd/server.func·003(0x1e53be20, 0x1d2a3c80, 0x1d254310, 0x0, 0x84be0e0, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/server.go:141 +0x66
github.com/coreos/etcd/server.func·004(0x1e53be20, 0x1d2a3c80, 0x1d254310)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/server.go:162 +0x23b
net/http.HandlerFunc.ServeHTTP(0x1868bda0, 0x1e53be20, 0x1d2a3c80, 0x1d254310)
/usr/local/go/src/pkg/net/http/server.go:1149 +0x3c
github.com/gorilla/mux.(_Router).ServeHTTP(0x18666540, 0x1e53be20, 0x1d2a3c80, 0x1d254310)
/home/tprojadmin/etcd/src/github.com/gorilla/mux/mux.go:90 +0x193
net/http.serverHandler.ServeHTTP(0x18691540, 0x1e53be20, 0x1d2a3c80, 0x1d254310)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x121
net/http.(_conn).serve(0x1f3f5190)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x653
created by net/http.(_Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x242
^Ctprojadmin@tproj-devt-04:~/etcd$ ./etcd -data-dir node0 -name node0
[etcd] Dec 2 15:41:49.280 INFO | etcd server [name node0, listen on 127.0.0.1:4001, advertised url http://127.0.0.1:4001]
[etcd] Dec 2 15:42:26.081 INFO | URLs: / node0 (http://127.0.0.1:7001,http://127.0.0.1:7002,http://127.0.0.1:7003)
[etcd] Dec 2 15:42:26.082 WARNING | the entire cluster is down! this peer will restart the cluster.
[etcd] Dec 2 15:42:26.082 INFO | raft server [name node0, listen on 127.0.0.1:7001, advertised url http://127.0.0.1:7001]
2013/12/02 15:43:32 http: panic serving 127.0.0.1:55893: runtime error: invalid memory address or nil pointer dereference
goroutine 781 [running]:
net/http.func·007()
/usr/local/go/src/pkg/net/http/server.go:1022 +0x9f
sync/atomic.AddUint64()
/usr/local/go/src/pkg/sync/atomic/asm_386.s:69 +0xc
github.com/coreos/etcd/store.(_watcherHub).watch(0x1868b4c0, 0x20eb8030, 0x8, 0x0, 0x23c, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/store/watcher_hub.go:71 +0x165
github.com/coreos/etcd/store.(_store).Watch(0x1868e480, 0x20eb8030, 0x8, 0x82a0400, 0x0, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/store/store.go:295 +0x151
github.com/coreos/etcd/server/v2.GetHandler(0x1dc65640, 0x1b1138c0, 0x1aa0dd90, 0x1e48fe10, 0x18691540, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/v2/get_handler.go:49 +0x455
github.com/coreos/etcd/server.func·003(0x1dc65640, 0x1b1138c0, 0x1aa0dd90, 0x0, 0x84be0e0, ...)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/server.go:141 +0x66
github.com/coreos/etcd/server.func·004(0x1dc65640, 0x1b1138c0, 0x1aa0dd90)
/home/tprojadmin/etcd/src/github.com/coreos/etcd/server/server.go:162 +0x23b
net/http.HandlerFunc.ServeHTTP(0x1868bda0, 0x1dc65640, 0x1b1138c0, 0x1aa0dd90)
/usr/local/go/src/pkg/net/http/server.go:1149 +0x3c
github.com/gorilla/mux.(_Router).ServeHTTP(0x18666540, 0x1dc65640, 0x1b1138c0, 0x1aa0dd90)
/home/tprojadmin/etcd/src/github.com/gorilla/mux/mux.go:90 +0x193
net/http.serverHandler.ServeHTTP(0x18691540, 0x1dc65640, 0x1b1138c0, 0x1aa0dd90)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x121
net/http.(_conn).serve(0x1f3d91e0)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x653
created by net/http.(_Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x242

Some one have a solution please ?

Björn Stein commented 10 years ago

+1

Xiang Li · Answer 1 · Tue Dec 03 2013 00:09:08 GMT+0800 (China Standard Time)

@marouane-kech Hmm... Are you on a 32bit machine? If it is, then seems like a "bug" in golang. I cannot reproduce your problem on my current machine. But I will find a 32bit machine to give a try.

We have a struct like

type watcherHub struct {
    watchers     map[string]*list.List
    count        int64 // current number of watchers.
    EventHistory *EventHistory
}

When you access count via 64bit operation, it panics.
I guess the problem is that the struct is not 64bit aligned on your machine.
Can you try to move count to the first like

type watcherHub struct {
    count        int64 // current number of watchers.
    watchers     map[string]*list.List
    EventHistory *EventHistory
}

Xiang Li · Answer 2 · Tue Dec 03 2013 00:13:50 GMT+0800 (China Standard Time)

@marouane-kech Oh. I found a similar issue on golang forum. https://code.google.com/p/go/issues/detail?id=6404. I am going to fix it now.

Brandon Philips · Answer 3 · Tue Dec 03 2013 01:19:01 GMT+0800 (China Standard Time)

@xiangli-cmu Is this already fixed in go1.2 maybe?

On Mon, Dec 2, 2013 at 8:13 AM, Xiang Li notifications@github.com wrote:

@marouane-kech https://github.com/marouane-kech Oh. I found a similar
issue on golang forum. https://code.google.com/p/go/issues/detail?id=6404.
I am going to fix it now.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/358#issuecomment-29630859
.

marouane-kech · Answer 4 · Wed Dec 04 2013 17:35:58 GMT+0800 (China Standard Time)

@xiangli-cmu Yes I have a 32bit machine, I do the change like you said in watcherHub struct (file watcher_hub.go) but I still have the same problem.
do you find a solution to this yet ? How can I fix it ?

Thank you.

Xiang Li · Answer 5 · Thu Dec 05 2013 01:01:11 GMT+0800 (China Standard Time)

@marouane-kech I will check it on a 32bit machine.

marouane-kech · Answer 6 · Thu Jan 02 2014 17:31:17 GMT+0800 (China Standard Time)

Good morning @xiangli-cmu Have you some news about watcher function on a 32bit machine.
Thank you.

Vladimir Pouzanov · Answer 7 · Sun Mar 16 2014 18:44:38 GMT+0800 (China Standard Time)

I'm hitting this one as well. That's one of the most amazing bugs in golang I've seem so far, and I see lots of those recently.

It's possible to reorder some fields accessed by atomic where that's required. A possible fix could be seen at farcaller@1257e1c. If this is an acceptable solution for etcd, I can clean that up and send a PR.

Brandon Philips · Answer 8 · Mon Mar 17 2014 10:26:12 GMT+0800 (China Standard Time)

@farcaller Is there an upstream bug for this alignment problem?

Max Neunhöffer · Answer 9 · Thu Apr 24 2014 17:52:10 GMT+0800 (China Standard Time)

This is to remind you of this issue. We are hit by this as well. Upstream says
"On both ARM and x86-32, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically." and has the issue status set to "Unfortunate".
I would consider this as a bug in etcd on 32bit ARM and x86-32.
To fix it one would have to make sure that all 64bit values that must be changed atomically are 8byte aligned. farcaller's patch might achieve this.

Jan · Answer 10 · Thu Apr 24 2014 17:56:09 GMT+0800 (China Standard Time)

Also hitting this bug on i386. As @farcaller's patch is not yet included in current master, is there any hope this is going to be fixed / worked around in etcd soon?

Brandon Philips · Answer 11 · Thu Apr 24 2014 23:52:49 GMT+0800 (China Standard Time)

@jsteemann @neunhoef @farcaller Since you have the hardware can one of you figure out the alignment of the structs that makes this work for you and send a patch? I am fine taking a patch as long as it doesn't have any functional changes. I will also have to rely on you all to test releases so we don't regress.

Thanks.

Vladimir Pouzanov · Answer 12 · Fri Apr 25 2014 00:09:13 GMT+0800 (China Standard Time)

Based on my discussion at go-nuts, there are no plans to make atomic safer. So far, the safest way is:

make sure that any integer accessible by atomic is first in the struct;
if more than one such integer is required, they must be ordered from longest (in number of bits) to shortest; alternatively, always use int64 for atomics.

Yicheng Qin · Answer 13 · Fri Apr 25 2014 13:43:04 GMT+0800 (China Standard Time)

@farcaller I like your patch!
I think this is the best way to fix it for now.
For third party lib, we may need to send separate PR to upstream one.
But we could merge the watch hub one if it works. :)

Jan · Answer 14 · Fri May 09 2014 20:30:51 GMT+0800 (China Standard Time)

@unihorn @farcaller: I have tried farcaller's patch (farcaller@1257e1c) with ETCD 0.3 on a 32 bit Ubuntu and it worked fine.
With patch, the crashes when using watches do not seem to occur anymore.
I have also tried the patch on 64 bit Linuxes, and it did not do any harm there either.

I'd love to see farcaller's merged being merged into ETCD.

Yicheng Qin · Answer 15 · Sat May 10 2014 01:27:35 GMT+0800 (China Standard Time)

The patch covers three repos, and I think it could only be a workaround for watch command.
I have grep int64 in the repo, and found out tens of lines. If I want to use this solution fully, I have to make changes on about five structs, which break their original sequences. And other changes should be applied to third_party repo.
Is this acceptable? @xiangli-cmu @philips @bmizerany

Blake Mizerany · Answer 16 · Sat May 10 2014 13:47:16 GMT+0800 (China Standard Time)

This is a big change since it is going to touch a ton in third_party, which we don't own and don't have a good way to know what that is going to affect, I think.

Jan · Answer 17 · Mon May 12 2014 18:08:02 GMT+0800 (China Standard Time)

If this is issue is not going to be fixed, I think there should at least be some statement in ETCD's README.md like ETCD works on 64 bit operating systems only. Support for 32 bit operating systems may be added in the future.
Just to save others from trouble.

What do you think?

Kelsey Hightower · Answer 18 · Tue Oct 21 2014 01:36:52 GMT+0800 (China Standard Time)

etcd does not support 32bit systems. We have updated our README to state the fact. Thanks.