shard crashes with "slice bounds out of range"

[Tobias Gruetzmacher]

I get this crash, probably related to 8029ba6:

2018/05/30 09:18:55 query: substr:"importscenario"
2018/05/30 09:18:55 crashed shard: shard(/home/zoekt/.zoekt/repo_v15.00000.zoekt): runtime error: slice bounds out of range, goroutine 9 [running]:
runtime/debug.Stack(0xc420132440, 0xc420ec8000, 0x61)
        /usr/lib/go-1.10/src/runtime/debug/stack.go:24 +0xa7
github.com/google/zoekt/shards.searchOneShard.func1(0x80c260, 0xc420132440, 0xc4200a0300)
        /home/zoekt/go/src/github.com/google/zoekt/shards/shards.go:228 +0x9c
panic(0x75a060, 0x9ba720)
        /usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
github.com/google/zoekt.(*candidateMatch).matchContent(0xc420094690, 0x7ff65d287230, 0x2aa7, 0x1dc0dd0, 0x1dc0dd0)
        /home/zoekt/go/src/github.com/google/zoekt/matchiter.go:57 +0x13d
github.com/google/zoekt.(*substrMatchTree).matches(0xc4200643c0, 0xc4200ec3c0, 0x2, 0xc420087800, 0x9c0000)
        /home/zoekt/go/src/github.com/google/zoekt/matchtree.go:485 +0x13e
github.com/google/zoekt.evalMatchTree(0xc4200ec3c0, 0x2, 0xc420087800, 0x80c0a0, 0xc4200643c0, 0xc420080100)
        /home/zoekt/go/src/github.com/google/zoekt/matchtree.go:454 +0xa6
github.com/google/zoekt.(*orMatchTree).matches(0xc420132580, 0xc4200ec3c0, 0x2, 0xc420087800, 0x800000)
        /home/zoekt/go/src/github.com/google/zoekt/matchtree.go:414 +0xa6
github.com/google/zoekt.(*indexData).Search(0xc42010ca80, 0x80baa0, 0xc420064340, 0x808740, 0xc4201322e0, 0xc4204b7c40, 0x0, 0x0, 0x0)
        /home/zoekt/go/src/github.com/google/zoekt/eval.go:169 +0x508
github.com/google/zoekt/shards.searchOneShard(0x80baa0, 0xc420064340, 0x80c260, 0xc420132440, 0x808740, 0xc4201322e0, 0xc4204b7c40, 0xc4200a0300)
        /home/zoekt/go/src/github.com/google/zoekt/shards/shards.go:236 +0xbd
github.com/google/zoekt/shards.(*shardedSearcher).Search.func2(0xc4200a0360, 0xc420082be0, 0x808740, 0xc4201322e0, 0xc4204b7c40, 0xc4200a0300)
        /home/zoekt/go/src/github.com/google/zoekt/shards/shards.go:170 +0x8f
created by github.com/google/zoekt/shards.(*shardedSearcher).Search
        /home/zoekt/go/src/github.com/google/zoekt/shards/shards.go:168 +0x5ce
2018/05/30 09:18:55 stats: zoekt.Stats{ContentBytesLoaded:0, IndexBytesLoaded:0, Crashes:1, Duration:318999, FileCount:0, ShardFilesConsidered:0, FilesConsidered:0, FilesLoaded:0, FilesSkipped:0, ShardsSkipped:0, MatchCount:0, NgramMatches:0, Wait:1116}

It looks like I can fix it like this, but I'm not sure why and if this just masks another problem:

--- a/indexbuilder.go
+++ b/indexbuilder.go
@@ -90,7 +90,7 @@ func (s *postingsBuilder) newSearchableString(data []byte, byteSections []Docume
                if c == 0 {
                        return nil, nil, &skipError{fmt.Sprintf("binary content at byte offset %d", byteCount)}
                }
-               if c == utf8.RuneError {
+               if c == 1 && c == utf8.RuneError {
                        return nil, nil, &skipError{fmt.Sprintf("invalid UTF-8 at byte offset %d", byteCount)}
                }
                data = data[sz:]

[Han-Wen Nienhuys]

this has been fixed already, but I forgot to push to GH.

[Han-Wen Nienhuys]

fixed by 2a6b814