fatal error: concurrent map read and map write
rdwatters opened this issue · comments
Left this on the forums as well. Getting this at a rate of almost every other save/livereload at this point. Unfortunately, the repo is private, but I can invite if necessary for a more comprehensive look at the code. Here's a link to the gist:
https://gist.github.com/rdwatters/ab7d3226fff97c7b432e5264fab57466
Probably same as #2224. If you could invite me so I could have a look that would be great.
Done and done @bep. Thanks much.
Thanks, @rdwatters -- I can reproduce with this one, which means I can figure it out.
@rdwatters in your case it is the nested partial invocations that makes the race happen:
taxonomy.html -> partial -> partial
That should be enough to create a test case.
@bep. Just to make sure I follow. Is this an example?
That's too bad if it's the nested partials that are causing the issue since the partials/components/title comes in awfully handy.
That said, I haven't had a single issue at build time (I'm using Netlify for CD), but the local deve definitely seems to be getting worse.
Removing the nested partial makes the rebuilds stable -- but that is just a symptom ... or something. We'll fix this. Regular builds seems to be stable with your site.
Is this an example?
Yes.
Great. Thanks much for looking into this. Should I post an issue elsewhere (perhaps add to the previous golang thread? I ask because I use that heading partial all over the place. At this point, I'm not sure it's worth extracting just to prevent the panic during dev.
No, I will try go boil it down to a simpler reproduction and post it at Go stdlib. If it is an issue we can probably work around it on Hugo side by adding a mutex.
Here is a stack of the crash with the race detector enabled:
WARNING: DATA RACE
Read at 0x00c424139110 by goroutine 6:
runtime.mapaccess1_faststr()
/usr/local/go/src/runtime/hashmap_fast.go:192 +0x0
text/template.(*state).walkTemplate()
/usr/local/go/src/text/template/exec.go:369 +0x12b
text/template.(*state).walk()
/usr/local/go/src/text/template/exec.go:239 +0x3a4
text/template.(*state).walk()
/usr/local/go/src/text/template/exec.go:234 +0x1f7
text/template.(*Template).execute()
/usr/local/go/src/text/template/exec.go:189 +0x4a2
text/template.(*Template).Execute()
/usr/local/go/src/text/template/exec.go:175 +0x64
html/template.(*Template).Execute()
/usr/local/go/src/html/template/template.go:104 +0xd0
github.com/spf13/hugo/hugolib.(*Site).renderThing()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2436 +0xcf
github.com/spf13/hugo/hugolib.(*Site).renderForLayouts()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x2b8
github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x138
github.com/spf13/hugo/hugolib.pageRenderer()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x23e
Previous write at 0x00c424139110 by goroutine 7:
runtime.mapassign1()
/usr/local/go/src/runtime/hashmap.go:442 +0x0
text/template.(*Template).associate()
/usr/local/go/src/text/template/template.go:216 +0x166
text/template.(*Template).AddParseTree()
/usr/local/go/src/text/template/template.go:128 +0x26c
html/template.(*escaper).commit()
/usr/local/go/src/html/template/escape.go:746 +0x30c
html/template.escapeTemplate()
/usr/local/go/src/html/template/escape.go:39 +0x4fd
html/template.(*Template).escape()
/usr/local/go/src/html/template/template.go:85 +0x2fb
html/template.(*Template).Execute()
/usr/local/go/src/html/template/template.go:101 +0x3c
github.com/spf13/hugo/hugolib.(*Site).renderThing()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2436 +0xcf
github.com/spf13/hugo/hugolib.(*Site).renderForLayouts()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x2b8
github.com/spf13/hugo/hugolib.(*Site).renderAndWritePage()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x138
github.com/spf13/hugo/hugolib.pageRenderer()
/Users/bep/go/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x23e
The Go issue:
Thanks for the due diligence on this @bep. I offered the repo on the Go issue thread as well.
OK -- now I have this issue on my site ass well ... pain in the ass. I think the cause of it was adding the second block to my base template.
The good news: I have a small reproduceable (I will try to make it even smaller) for the Go team -- the bad news is I don't see a workaround, yet ...
Saw that workaround offered by cespare on the other thread. Glad it's been added to the milestones. I'm cool with pressing up/enter in the terminal each time it panics until v18. Cheers.
Or you could build from source ...
Or you could point out the obvious, which I should already know. Haha. Thanks!
i'm obtaining the same error in a random way, as you can see, three hugo compilation runs fine , but another compilation raise the error.
$ hugo
Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
108 pages created
0 non-page files copied
12 paginator pages created
0 tags created
0 categories created
total in 448 ms
$ hugo
Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
108 pages created
0 non-page files copied
12 paginator pages created
0 tags created
0 categories created
total in 431 ms
$ hugo
Started building sites ...
Built site for language en:
0 draft content
0 future content
0 expired content
108 pages created
0 non-page files copied
12 paginator pages created
0 categories created
0 tags created
total in 426 ms
$ hugo
Started building sites ...
fatal error: concurrent map read and map write
goroutine 211 [running]:
runtime.throw(0xaebb56, 0x21)
/usr/local/Cellar/go/1.7.1/libexec/src/runtime/panic.go:566 +0x95 fp=0xc420aa1778 sp=0xc420aa1758
runtime.mapaccess1_faststr(0xa313a0, 0xc4203861e0, 0xc420095ed3, 0x4, 0xdf05c0)
/usr/local/Cellar/go/1.7.1/libexec/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc420aa17d8 sp=0xc420aa1778
text/template.(_state).walkTemplate(0xc420aa1a48, 0xace2e0, 0xc42074af00, 0x16, 0xc42038ec80)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:369 +0x9a fp=0xc420aa18a8 sp=0xc420aa17d8
text/template.(_state).walk(0xc420aa1a48, 0xace2e0, 0xc42074af00, 0x16, 0xdf0800, 0xc42038ec80)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:239 +0x24c fp=0xc420aa1930 sp=0xc420aa18a8
text/template.(_state).walk(0xc420aa1a48, 0xace2e0, 0xc42074af00, 0x16, 0xdf05c0, 0xc42038dec0)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:234 +0x13d fp=0xc420aa19b8 sp=0xc420aa1930
text/template.(_Template).execute(0xc42037ccc0, 0xde6c60, 0xc42016f1f0, 0xace2e0, 0xc42074af00, 0x0, 0x0)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:189 +0x204 fp=0xc420aa1aa0 sp=0xc420aa19b8
text/template.(_Template).Execute(0xc42037ccc0, 0xde6c60, 0xc42016f1f0, 0xace2e0, 0xc42074af00, 0xc4203088b0, 0xc4205ade01)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:175 +0x53 fp=0xc420aa1ae8 sp=0xc420aa1aa0
html/template.(_Template).Execute(0xc4203871d0, 0xde6c60, 0xc42016f1f0, 0xace2e0, 0xc42074af00, 0x1a, 0xc4203871d0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:104 +0x78 fp=0xc420aa1b30 sp=0xc420aa1ae8
github.com/spf13/hugo/hugolib.(_Site).renderThing(0xc4202ae000, 0xace2e0, 0xc42074af00, 0xc4205adee0, 0x1a, 0xde6c60, 0xc42016f1f0, 0xa058e0, 0x5bdf93)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2436 +0x96 fp=0xc420aa1b98 sp=0xc420aa1b30
github.com/spf13/hugo/hugolib.(_Site).renderForLayouts(0xc4202ae000, 0xc420687080, 0x76, 0xace2e0, 0xc42074af00, 0xde6c60, 0xc42016f1f0, 0xc420118300, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x22a fp=0xc420aa1c78 sp=0xc420aa1b98
github.com/spf13/hugo/hugolib.(_Site).renderAndWritePage(0xc4202ae000, 0xc420687080, 0x76, 0xc420686f80, 0x73, 0xace2e0, 0xc42074af00, 0xc420118300, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x128 fp=0xc420aa1ea8 sp=0xc420aa1c78
github.com/spf13/hugo/hugolib.pageRenderer(0xc4202ae000, 0xc42043ba40, 0xc42043b9e0, 0xc4206cb560)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x22e fp=0xc420aa1f70 sp=0xc420aa1ea8
runtime.goexit()
/usr/local/Cellar/go/1.7.1/libexec/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420aa1f78 sp=0xc420aa1f70
created by github.com/spf13/hugo/hugolib.(_Site).renderPages
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1635 +0x330
goroutine 1 [chan send]:
github.com/spf13/hugo/hugolib.(_Site).renderPages(0xc4202ae000, 0xae3e85, 0x16)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1639 +0x3a7
github.com/spf13/hugo/hugolib.(_Site).render(0xc4202ae000, 0x100, 0x0)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:820 +0x16f
github.com/spf13/hugo/hugolib.(_HugoSites).Build(0xc420264120, 0x100, 0x0, 0x1b, 0x0)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/hugo_sites.go:232 +0x257
github.com/spf13/hugo/commands.buildSites(0x0, 0x0, 0x0)
/Users/spf13/gopath/src/github.com/spf13/hugo/commands/hugo.go:666 +0xf0
github.com/spf13/hugo/commands.build(0x0, 0x0, 0x0, 0x0, 0x0)
/Users/spf13/gopath/src/github.com/spf13/hugo/commands/hugo.go:467 +0x99
github.com/spf13/hugo/commands.glob..func8(0xe32be0, 0xe58218, 0x0, 0x0, 0x0, 0x0)
/Users/spf13/gopath/src/github.com/spf13/hugo/commands/hugo.go:124 +0x7d
github.com/spf13/hugo/vendor/github.com/spf13/cobra.(_Command).execute(0xe32be0, 0xc42000a330, 0x0, 0x0, 0xe32be0, 0xc42000a330)
/Users/spf13/gopath/src/github.com/spf13/hugo/vendor/github.com/spf13/cobra/command.go:599 +0x234
github.com/spf13/hugo/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xe32be0, 0xb4e270, 0xe3bcf0, 0x1)
/Users/spf13/gopath/src/github.com/spf13/hugo/vendor/github.com/spf13/cobra/command.go:689 +0x367
github.com/spf13/hugo/commands.Execute()
/Users/spf13/gopath/src/github.com/spf13/hugo/commands/hugo.go:174 +0x6d
main.main()
/Users/spf13/gopath/src/github.com/spf13/hugo/main.go:26 +0x32
goroutine 214 [semacquire]:
sync.runtime_Semacquire(0xc42036fe24)
/usr/local/Cellar/go/1.7.1/libexec/src/runtime/sema.go:47 +0x30
sync.(_Mutex).Lock(0xc42036fe20)
/usr/local/Cellar/go/1.7.1/libexec/src/sync/mutex.go:85 +0xd0
html/template.(_Template).escape(0xc4203871d0, 0x0, 0x0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:79 +0x53
html/template.(_Template).Execute(0xc4203871d0, 0xde6c60, 0xc42022f650, 0xace2e0, 0xc420789900, 0x1a, 0xc4203871d0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:101 +0x2f
github.com/spf13/hugo/hugolib.(_Site).renderThing(0xc4202ae000, 0xace2e0, 0xc420789900, 0xc420584020, 0x1a, 0xde6c60, 0xc42022f650, 0xa058e0, 0x5bdf93)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2436 +0x96
github.com/spf13/hugo/hugolib.(_Site).renderForLayouts(0xc4202ae000, 0xc420151f50, 0x2e, 0xace2e0, 0xc420789900, 0xde6c60, 0xc42022f650, 0xc420367a00, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x22a
github.com/spf13/hugo/hugolib.(_Site).renderAndWritePage(0xc4202ae000, 0xc420151f50, 0x2e, 0xc420151f20, 0x2b, 0xace2e0, 0xc420789900, 0xc420367a00, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x128
github.com/spf13/hugo/hugolib.pageRenderer(0xc4202ae000, 0xc42043ba40, 0xc42043b9e0, 0xc4206cb560)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x22e
created by github.com/spf13/hugo/hugolib.(*Site).renderPages
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1635 +0x330
goroutine 210 [chan receive]:
github.com/spf13/hugo/hugolib.errorCollator(0xc42043b9e0, 0xc42043baa0)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1667 +0x8d
created by github.com/spf13/hugo/hugolib.(*Site).renderPages
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1607 +0xce
goroutine 213 [runnable]:
reflect.(_rtype).Name(0xa6b6e0, 0xc420aa4e90, 0x0)
/usr/local/Cellar/go/1.7.1/libexec/src/reflect/type.go:893
reflect.(_structType).FieldByNameFunc(0xacab20, 0xc420aa5090, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/reflect/type.go:1322 +0xb7b
reflect.(_structType).FieldByName(0xacab20, 0xc42031a921, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/reflect/type.go:1392 +0x19f
reflect.(_rtype).FieldByName(0xacab20, 0xc42031a921, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/reflect/type.go:966 +0x9c
text/template.(_state).evalField(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xc42031a921, 0x4, 0xdf04a0, 0xc42038df50, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:555 +0x8b6
text/template.(_state).evalFieldChain(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xace2e0, 0xc42077e000, 0x16, 0xdf04a0, 0xc42038df50, 0xc4203904c0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:517 +0x12b
text/template.(_state).evalFieldNode(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xc42038df50, 0xc42038a6e0, 0x1, 0x1, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:484 +0x114
text/template.(_state).evalCommand(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xc42038df20, 0x0, 0x0, 0x0, 0x0, 0x6000103, ...)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:422 +0x866
text/template.(_state).evalPipeline(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xc420383630, 0x0, 0x0, 0x40f09e)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:400 +0xee
text/template.(_state).walk(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xdf0260, 0xc42038df80)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:226 +0x423
text/template.(_state).walk(0xc420aa5a48, 0xace2e0, 0xc42077e000, 0x16, 0xdf05c0, 0xc42038dec0)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:234 +0x13d
text/template.(_Template).execute(0xc42037ccc0, 0xde6c60, 0xc42016f180, 0xace2e0, 0xc42077e000, 0x0, 0x0)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:189 +0x204
text/template.(_Template).Execute(0xc42037ccc0, 0xde6c60, 0xc42016f180, 0xace2e0, 0xc42077e000, 0xc4203088b0, 0xc4205adf01)
/usr/local/Cellar/go/1.7.1/libexec/src/text/template/exec.go:175 +0x53
html/template.(_Template).Execute(0xc4203871d0, 0xde6c60, 0xc42016f180, 0xace2e0, 0xc42077e000, 0x1a, 0xc4203871d0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:104 +0x78
github.com/spf13/hugo/hugolib.(_Site).renderThing(0xc4202ae000, 0xace2e0, 0xc42077e000, 0xc4205adf80, 0x1a, 0xde6c60, 0xc42016f180, 0xa058e0, 0x5bdf93)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2436 +0x96
github.com/spf13/hugo/hugolib.(_Site).renderForLayouts(0xc4202ae000, 0xc4208da7e0, 0x62, 0xace2e0, 0xc42077e000, 0xde6c60, 0xc42016f180, 0xc4204d9500, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x22a
github.com/spf13/hugo/hugolib.(_Site).renderAndWritePage(0xc4202ae000, 0xc4208da7e0, 0x62, 0xc4206b72c0, 0x3c, 0xace2e0, 0xc42077e000, 0xc4204d9500, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x128
github.com/spf13/hugo/hugolib.pageRenderer(0xc4202ae000, 0xc42043ba40, 0xc42043b9e0, 0xc4206cb560)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x22e
created by github.com/spf13/hugo/hugolib.(_Site).renderPages
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1635 +0x330
goroutine 212 [runnable]:
sync.runtime_Semacquire(0xc42036fe24)
/usr/local/Cellar/go/1.7.1/libexec/src/runtime/sema.go:47 +0x30
sync.(_Mutex).Lock(0xc42036fe20)
/usr/local/Cellar/go/1.7.1/libexec/src/sync/mutex.go:85 +0xd0
html/template.(_Template).escape(0xc4203871d0, 0x0, 0x0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:79 +0x53
html/template.(_Template).Execute(0xc4203871d0, 0xde6c60, 0xc42089dce0, 0xace2e0, 0xc42071f900, 0x1a, 0xc4203871d0)
/usr/local/Cellar/go/1.7.1/libexec/src/html/template/template.go:101 +0x2f
github.com/spf13/hugo/hugolib.(_Site).renderThing(0xc4202ae000, 0xace2e0, 0xc42071f900, 0xc4205ade60, 0x1a, 0xde6c60, 0xc42089dce0, 0xa058e0, 0x5bdf93)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2436 +0x96
github.com/spf13/hugo/hugolib.(_Site).renderForLayouts(0xc4202ae000, 0xc420831380, 0x76, 0xace2e0, 0xc42071f900, 0xde6c60, 0xc42089dce0, 0xc42088c400, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2408 +0x22a
github.com/spf13/hugo/hugolib.(_Site).renderAndWritePage(0xc4202ae000, 0xc420831380, 0x76, 0xc420831280, 0x73, 0xace2e0, 0xc42071f900, 0xc42088c400, 0xa, 0x10, ...)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:2316 +0x128
github.com/spf13/hugo/hugolib.pageRenderer(0xc4202ae000, 0xc42043ba40, 0xc42043b9e0, 0xc4206cb560)
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1658 +0x22e
created by github.com/spf13/hugo/hugolib.(*Site).renderPages
/Users/spf13/gopath/src/github.com/spf13/hugo/hugolib/site.go:1635 +0x330
@VladimirPlokhov this issue is closed as in -- fixed. Please don't comment on fixed issues if you have nothing new to add.
I'm getting this same error when attempting to build using v0.75.x on AWS Codebuild (but strangely not when using v0.75.1 locally on macOS). Installing Hugo using the following commands:
wget https://github.com/gohugoio/hugo/releases/download/v0.75.1/hugo_extended_0.75.1_Linux-64bit.deb
dpkg -i hugo_extended_0.75.1_Linux-64bit.deb```
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.