quickStart run into panic: SIGILL: illegal instruction

Question

quickStart run into panic: SIGILL: illegal instruction

jerryzhen01 opened this issue 3 years ago · comments

General Issues

Hi,

I followed the quickStart tutorial to try to bring up the M3 stack and run the example, but the service crashed every time when I used the below curl command to write metrics. (https://m3db.io/docs/quickstart/docker/#writing-metrics)

#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
  "tags": 
    {
      "__name__": "third_avenue",
      "city": "new_york",
      "checkout": "1"
    },
    "timestamp": '\"$(date "+%s")\"',
    "value": 3347.26
}'

Detail panic logs are attached below, please help take a look, thanks!

Please provide the following information along with a description of the issue that you're experiencing:

What service is experiencing the issue? (M3Coordinator, M3DB, M3Aggregator, etc)
M3Coordinator and M3DB
What is the configuration of the service? Please include any YAML files, as well as namespace / placement configuration.
I followed the quickStart docs here: https://m3db.io/docs/quickstart/docker/
How are you using the service? For example, are you performing read/writes to the service via Prometheus, or are you using a custom script?
I followed the quickStart tutorial and used HTTP API (https://m3db.io/docs/quickstart/docker/#writing-metrics) to write metrics to M3DB.
m3crash.log
Is there a reliable way to reproduce the behavior? If so, please provide detailed instructions.
I can reproduce this on my local dev env. Just follow the quickStart's steps.
m3crash.log

part of the panic logs:

2021/10/08 00:04:57 Go Runtime version: go1.13.15
2021/10/08 00:04:57 Build Version:      v1.1.0
2021/10/08 00:04:57 Build Revision:     468ab1c4b
2021/10/08 00:04:57 Build Branch:       HEAD
2021/10/08 00:04:57 Build Date:         2021-02-04-20:24:02
2021/10/08 00:04:57 Build TimeUnix:     1612470242
{"level":"info","ts":1633651497.66925,"msg":"setting MutexProfileFraction: 0"}
{"level":"info","ts":1633651497.6692986,"msg":"setting BlockProfileRate: 0"}
{"level":"info","ts":1633651497.669362,"msg":"raised rlimit no file fds limit","required":true,"sysNROpenValue":3000000,"noFileMaxValue":3000000,"noFileCurrValue":3000000}
{"level":"info","ts":1633651497.6693943,"msg":"setting MutexProfileFraction: 0"}
{"level":"info","ts":1633651497.6694212,"msg":"setting BlockProfileRate: 0"}
{"level":"info","ts":1633651497.6697774,"msg":"tracing disabled for m3query; set `tracing.backend` to enable"}
{"level":"info","ts":1633651497.6866865,"msg":"tracing disabled; set `tracing.backend` to enable"}
{"level":"info","ts":1633651497.6867647,"msg":"resolving seed node configuration","hostID":"m3db_local","seedNodeHostIDs":["m3db_local"]}
{"level":"info","ts":1633651497.686773,"msg":"seed node, starting etcd server"}
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-10-08 00:04:57.686946 I | embed: name = m3db_local
2021-10-08 00:04:57.686958 I | embed: data dir = /var/lib/m3db/etcd
2021-10-08 00:04:57.686963 I | embed: member dir = /var/lib/m3db/etcd/member
2021-10-08 00:04:57.686967 I | embed: heartbeat = 100ms
2021-10-08 00:04:57.686972 I | embed: election = 1000ms
2021-10-08 00:04:57.686976 I | embed: snapshot count = 100000
2021-10-08 00:04:57.686996 I | embed: advertise client URLs = http://127.0.0.1:2379
2021-10-08 00:04:57.743110 I | etcdserver: starting member b71f75320dc06a6c in cluster 1c45a069f3a1d796
raft2021/10/08 00:04:57 INFO: b71f75320dc06a6c switched to configuration voters=()
raft2021/10/08 00:04:57 INFO: b71f75320dc06a6c became follower at term 0
raft2021/10/08 00:04:57 INFO: newRaft b71f75320dc06a6c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/10/08 00:04:57 INFO: b71f75320dc06a6c became follower at term 1
raft2021/10/08 00:04:57 INFO: b71f75320dc06a6c switched to configuration voters=(13195394291058371180)
2021-10-08 00:04:57.761882 W | auth: simple token is not cryptographically signed
{"level":"info","ts":1633651497.7706604,"msg":"successfully created new cache dir","path":"/var/lib/m3kv","mode":0}
{"level":"warn","ts":1633651497.7707934,"msg":"could not load cache from file","file":"/var/lib/m3kv/_kv_default_env_m3db_embedded.json","error":"error opening cache file /var/lib/m3kv/_kv_default_env_m3db_embedded.json: open /var/lib/m3kv/_kv_default_env_m3db_embedded.json: no such file or directory"}
2021-10-08 00:04:57.781424 I | etcdserver: starting server... [version: 3.4.13, cluster version: to_be_decided]
2021-10-08 00:04:57.790754 I | etcdserver: b71f75320dc06a6c as single-node; fast-forwarding 9 ticks (election ticks 10)
2021-10-08 00:04:57.791539 I | embed: listening for peers on [::]:2380
raft2021/10/08 00:04:57 INFO: b71f75320dc06a6c switched to configuration voters=(13195394291058371180)
2021-10-08 00:04:57.792539 I | etcdserver/membership: added member b71f75320dc06a6c [http://127.0.0.1:2380] to cluster 1c45a069f3a1d796
{"level":"info","ts":1633651497.8051488,"msg":"creating namespaces watcher","clusters":1}
{"level":"info","ts":1633651497.805422,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.node.namespaces","revision":0}
{"level":"warn","ts":1633651498.1583128,"msg":"max index query IDs concurrency was not set, falling back to default value"}
{"level":"info","ts":1633651498.1835651,"msg":"set thrift bytes pool alloc size","size":2048}
{"level":"info","ts":1633651498.1836274,"msg":"bytes pool configured","capacity":16,"size":524288,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1836393,"msg":"bytes pool configured","capacity":32,"size":262144,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1836445,"msg":"bytes pool configured","capacity":64,"size":131072,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1836493,"msg":"bytes pool configured","capacity":128,"size":65536,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1836605,"msg":"bytes pool configured","capacity":256,"size":65536,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1836693,"msg":"bytes pool configured","capacity":1440,"size":16384,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.183674,"msg":"bytes pool configured","capacity":4096,"size":8192,"refillLowWaterMark":0.3,"refillHighWaterMark":0.6}
{"level":"info","ts":1633651498.1837325,"msg":"bytes pool init start"}
raft2021/10/08 00:04:58 INFO: b71f75320dc06a6c is starting a new election at term 1
raft2021/10/08 00:04:58 INFO: b71f75320dc06a6c became candidate at term 2
raft2021/10/08 00:04:58 INFO: b71f75320dc06a6c received MsgVoteResp from b71f75320dc06a6c at term 2
raft2021/10/08 00:04:58 INFO: b71f75320dc06a6c became leader at term 2
raft2021/10/08 00:04:58 INFO: raft.node: b71f75320dc06a6c elected leader b71f75320dc06a6c at term 2
2021-10-08 00:04:58.344304 I | etcdserver: setting up the initial cluster version to 3.4
2021-10-08 00:04:58.344505 I | etcdserver: published {Name:m3db_local ClientURLs:[http://127.0.0.1:2379]} to cluster 1c45a069f3a1d796
2021-10-08 00:04:58.344544 I | embed: ready to serve client requests
2021-10-08 00:04:58.345236 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
2021-10-08 00:04:58.376117 N | etcdserver/membership: set the initial cluster version to 3.4
2021-10-08 00:04:58.376214 I | etcdserver/api: enabled capabilities for version 3.4
{"level":"info","ts":1633651498.3772557,"msg":"no initial namespaces found. namespaces will be added via watch updates"}
{"level":"info","ts":1633651498.3775146,"msg":"configuring downsampler to use with aggregated cluster namespaces","numAggregatedClusterNamespaces":0}
{"level":"info","ts":1633651498.3917642,"msg":"no m3msg server configured"}
{"level":"info","ts":1633651498.3917906,"msg":"using registered interrupt handler"}
{"level":"info","ts":1633651498.3918061,"msg":"starting API server","address":"[::]:7201"}
{"level":"info","ts":1633651498.8174858,"msg":"bytes pool init end"}
{"level":"info","ts":1633651499.1914165,"msg":"creating dynamic config service client with m3cluster"}
{"level":"info","ts":1633651499.192006,"msg":"successfully created new cache dir","path":"/var/lib/m3kv","mode":493}
{"level":"warn","ts":1633651499.192089,"msg":"could not load cache from file","file":"/var/lib/m3kv/_kv_default_env_m3db_embedded.json","error":"error opening cache file /var/lib/m3kv/_kv_default_env_m3db_embedded.json: open /var/lib/m3kv/_kv_default_env_m3db_embedded.json: no such file or directory"}
{"level":"info","ts":1633651499.22125,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env//namespaces","revision":0}
{"level":"info","ts":1633651499.5463812,"msg":"node tchannelthrift: listening","address":"0.0.0.0:9000"}
{"level":"info","ts":1633651499.5467942,"msg":"node httpjson: listening","address":"0.0.0.0:9002"}
{"level":"info","ts":1633651499.546844,"msg":"waiting for dynamic topology initialization, if this takes a long time, make sure that a topology/placement is configured"}
{"level":"info","ts":1633651499.5468519,"msg":"adding a watch","service":"m3db","env":"default_env","zone":"embedded","includeUnhealthy":true}
{"level":"info","ts":1633651499.5469384,"msg":"successfully created new cache dir","path":"/var/lib/m3kv","mode":493}
{"level":"warn","ts":1633651499.546989,"msg":"could not load cache from file","file":"/var/lib/m3kv/m3db_embedded.json","error":"error opening cache file /var/lib/m3kv/m3db_embedded.json: open /var/lib/m3kv/m3db_embedded.json: no such file or directory"}
{"level":"info","ts":1633651499.5470655,"msg":"creating etcd watch at revision","watch_key":"_sd.placement/default_env/m3db","revision":0}
{"level":"info","ts":1633651507.5983667,"msg":"initial topology / placement value received"}
{"level":"info","ts":1633651507.6437201,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.client.bootstrap-consistency-level","revision":0}
{"level":"info","ts":1633651507.6441371,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.client.read-consistency-level","revision":0}
{"level":"info","ts":1633651507.644534,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.client.write-consistency-level","revision":0}
{"level":"info","ts":1633651507.645073,"msg":"dynamic namespace registry updated to version","version":1}
{"level":"info","ts":1633651507.6451054,"msg":"received update from kv namespace watch"}
{"level":"info","ts":1633651507.6451392,"msg":"refreshed cluster namespaces","added":["default"],"updated":[],"removed":[]}
{"level":"info","ts":1633651507.6451535,"msg":"namespace has non-ready staging state status","namespace":"default","status":"unknown"}
{"level":"info","ts":1633651507.7559125,"msg":"successfully updated topology","numHosts":1}
{"level":"info","ts":1633651508.3112855,"msg":"cluster tchannelthrift: listening","address":"0.0.0.0:9001"}
{"level":"info","ts":1633651508.3114874,"msg":"cluster httpjson: listening","address":"0.0.0.0:9003"}
{"level":"info","ts":1633651508.31159,"msg":"cluster database initializing topology"}
{"level":"info","ts":1633651508.3116407,"msg":"cluster database resolving topology"}
{"level":"info","ts":1633651508.311649,"msg":"cluster database resolved topology"}
{"level":"info","ts":1633651508.3668034,"msg":"creating namespaces watch"}
{"level":"info","ts":1633651508.366873,"msg":"waiting for dynamic namespace registry initialization, if this takes a long time, make sure that a namespace is configured"}
{"level":"info","ts":1633651508.3669152,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.node.namespaces","revision":0}
{"level":"info","ts":1633651508.3682854,"msg":"initial namespace value received"}
{"level":"info","ts":1633651508.3684433,"msg":"resolving namespaces with namespace watch"}
{"level":"info","ts":1633651508.368591,"msg":"updating database namespaces","adds":"[default]","updates":"[]","removals":"[]"}
{"level":"info","ts":1633651508.3706417,"msg":"set namespace runtime index options","namespace":"default","namespace":"default","writeIndexingPerCPUConcurrency":null,"flushIndexingPerCPUConcurrency":null}
{"level":"info","ts":1633651508.370677,"msg":"namespace forward indexing not enabled","namespace":"default","namespace":"default","enabled":false,"threshold":0,"probability":0}
{"level":"info","ts":1633651508.381217,"msg":"using registered interrupt handler"}
{"level":"info","ts":1633651508.3814738,"msg":"bootstrap prepare"}
{"level":"info","ts":1633651508.382379,"msg":"bootstrap started","numShards":64}
{"level":"info","ts":1633651508.382693,"msg":"bootstrap range starting","bootstrapper":"base","namespace":"default","numShards":64,"dataFrom":1633608000,"dataTo":1633649400,"dataRange":41400,"indexFrom":1633608000,"indexTo":1633649400,"indexRange":41400}
{"level":"info","ts":1633651508.3827689,"msg":"bootstrap from source requested","bootstrapper":"filesystem","namespace":"default","numShards":64,"dataRange":41400,"dataFrom":1633608000,"dataTo":1633649400,"indexRange":41400,"indexFrom":1633608000,"indexTo":1633649400}
{"level":"info","ts":1633651508.3831997,"msg":"bootstrap from source ready after availability query","bootstrapper":"filesystem","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.3832126,"msg":"bootstrap from source hook begin started","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.3832307,"msg":"bootstrap from source started","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.390904,"msg":"bootstrapping time series data start","bootstrapper":"filesystem","cachePolicy":"lru"}
{"level":"info","ts":1633651508.3909373,"msg":"bootstrapping time series data success","bootstrapper":"filesystem","cachePolicy":"lru","took":0.000034162}
{"level":"info","ts":1633651508.3909435,"msg":"bootstrapping index metadata start","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.3909552,"msg":"bootstrapping index metadata success","bootstrapper":"filesystem","took":0.000011485}
{"level":"info","ts":1633651508.3909612,"msg":"bootstrap from source hook end started","bootstrapper":"filesystem","took":0.007747865}
{"level":"info","ts":1633651508.3913794,"msg":"bootstrap from source completed","bootstrapper":"filesystem","took":0.007747865}
{"level":"info","ts":1633651508.3917336,"msg":"bootstrapping from source completed successfully","bootstrapper":"filesystem","took":0.007747865,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3917754,"msg":"bootstrap from source requested","bootstrapper":"commitlog","namespace":"default","numShards":64,"dataRange":41400,"dataFrom":1633608000,"dataTo":1633649400,"indexRange":41400,"indexFrom":1633608000,"indexTo":1633649400}
{"level":"info","ts":1633651508.391846,"msg":"bootstrap from source ready after availability query","bootstrapper":"commitlog","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.391859,"msg":"bootstrap from source hook begin started","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.3918707,"msg":"bootstrap from source started","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.391879,"msg":"read snapshots start","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.3934803,"msg":"read snapshots done","bootstrapper":"commitlog","took":0.001600628}
{"level":"info","ts":1633651508.3935864,"msg":"read commit logs start","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.3954582,"msg":"returning not-unfulfilled: ReturnUnfulfilledForCorruptCommitLogFiles is false","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.3954842,"msg":"read commit logs done","bootstrapper":"commitlog","took":0.001893386,"datapointsRead":0,"datapointsSkippedNotBootstrappingNamespace":0,"datapointsSkippedNotBootstrappingShard":0,"datapointsSkippedShardNoLongerOwned":0}
{"level":"info","ts":1633651508.3954973,"msg":"bootstrap from source hook end started","bootstrapper":"commitlog","took":0.003637973}
{"level":"info","ts":1633651508.3959804,"msg":"bootstrap from source completed","bootstrapper":"commitlog","took":0.003637973}
{"level":"info","ts":1633651508.3962154,"msg":"bootstrapping from source completed successfully","bootstrapper":"commitlog","took":0.003637973,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3962555,"msg":"bootstrap from source requested","bootstrapper":"peers","namespace":"default","numShards":64,"dataRange":41400,"dataFrom":1633608000,"dataTo":1633649400,"indexRange":41400,"indexFrom":1633608000,"indexTo":1633649400}
{"level":"info","ts":1633651508.396404,"msg":"bootstrap from source ready after availability query","bootstrapper":"peers","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.396415,"msg":"bootstrap from source hook begin started","bootstrapper":"peers"}
{"level":"info","ts":1633651508.3964279,"msg":"bootstrap from source started","bootstrapper":"peers"}
{"level":"info","ts":1633651508.3964386,"msg":"bootstrap from source hook end started","bootstrapper":"peers","took":0.000023413}
{"level":"info","ts":1633651508.3968408,"msg":"bootstrap from source completed","bootstrapper":"peers","took":0.000023413}
{"level":"info","ts":1633651508.3970723,"msg":"bootstrapping from source completed successfully","bootstrapper":"peers","took":0.000023413,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3971062,"msg":"bootstrap from source requested","bootstrapper":"uninitialized_topology","namespace":"default","numShards":64,"dataRange":41400,"dataFrom":1633608000,"dataTo":1633649400,"indexRange":41400,"indexFrom":1633608000,"indexTo":1633649400}
{"level":"info","ts":1633651508.397223,"msg":"bootstrap from source ready after availability query","bootstrapper":"uninitialized_topology","namespace":"default","numShards":64,"dataRange":41400,"dataFrom":1633608000,"dataTo":1633649400,"indexRange":41400,"indexFrom":1633608000,"indexTo":1633649400}
{"level":"info","ts":1633651508.3972738,"msg":"bootstrap from source hook begin started","bootstrapper":"uninitialized_topology"}
{"level":"info","ts":1633651508.397284,"msg":"bootstrap from source started","bootstrapper":"uninitialized_topology"}
{"level":"info","ts":1633651508.397374,"msg":"bootstrap from source hook end started","bootstrapper":"uninitialized_topology","took":0.000099783}
{"level":"info","ts":1633651508.3980863,"msg":"bootstrap from source completed","bootstrapper":"uninitialized_topology","took":0.000099783}
{"level":"info","ts":1633651508.3984084,"msg":"bootstrapping from source completed successfully","bootstrapper":"uninitialized_topology","took":0.000099783,"namespace":"default","numShards":64,"dataRangeRequested":41400,"dataRangeFulfilled":41400,"indexRangeRequested":41400,"indexRangeFulfilled":41400,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3984387,"msg":"bootstrap range completed","bootstrapper":"base","namespace":"default","numShards":64,"dataFrom":1633608000,"dataTo":1633649400,"dataRange":41400,"indexFrom":1633608000,"indexTo":1633649400,"indexRange":41400,"took":0.015719616,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3984551,"msg":"bootstrap range starting","bootstrapper":"base","namespace":"default","numShards":64,"dataFrom":1633649400,"dataTo":1633653000,"dataRange":3600,"indexFrom":1633649400,"indexTo":1633653000,"indexRange":3600}
{"level":"info","ts":1633651508.3984976,"msg":"bootstrap from source requested","bootstrapper":"filesystem","namespace":"default","numShards":64,"dataRange":3600,"dataFrom":1633649400,"dataTo":1633653000,"indexRange":3600,"indexFrom":1633649400,"indexTo":1633653000}
{"level":"info","ts":1633651508.398601,"msg":"bootstrap from source ready after availability query","bootstrapper":"filesystem","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.3986123,"msg":"bootstrap from source hook begin started","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.3986208,"msg":"bootstrap from source started","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.3986554,"msg":"bootstrapping time series data start","bootstrapper":"filesystem","cachePolicy":"lru"}
{"level":"info","ts":1633651508.3986654,"msg":"bootstrapping time series data success","bootstrapper":"filesystem","cachePolicy":"lru","took":0.000010227}
{"level":"info","ts":1633651508.3986712,"msg":"bootstrapping index metadata start","bootstrapper":"filesystem"}
{"level":"info","ts":1633651508.3986816,"msg":"bootstrapping index metadata success","bootstrapper":"filesystem","took":0.000010326}
{"level":"info","ts":1633651508.3986871,"msg":"bootstrap from source hook end started","bootstrapper":"filesystem","took":0.000074592}
{"level":"info","ts":1633651508.3993716,"msg":"bootstrap from source completed","bootstrapper":"filesystem","took":0.000074592}
{"level":"info","ts":1633651508.3996902,"msg":"bootstrapping from source completed successfully","bootstrapper":"filesystem","took":0.000074592,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.3997657,"msg":"bootstrap from source requested","bootstrapper":"commitlog","namespace":"default","numShards":64,"dataRange":3600,"dataFrom":1633649400,"dataTo":1633653000,"indexRange":3600,"indexFrom":1633649400,"indexTo":1633653000}
{"level":"info","ts":1633651508.3998375,"msg":"bootstrap from source ready after availability query","bootstrapper":"commitlog","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.3998911,"msg":"bootstrap from source hook begin started","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.399901,"msg":"bootstrap from source started","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.39991,"msg":"read snapshots start","bootstrapper":"commitlog"}
{"level":"info","ts":1633651508.4005165,"msg":"read snapshots done","bootstrapper":"commitlog","took":0.000606648}
{"level":"info","ts":1633651508.4005349,"msg":"bootstrap from source hook end started","bootstrapper":"commitlog","took":0.000643302}
{"level":"info","ts":1633651508.4012837,"msg":"bootstrap from source completed","bootstrapper":"commitlog","took":0.000643302}
{"level":"info","ts":1633651508.4015105,"msg":"bootstrapping from source completed successfully","bootstrapper":"commitlog","took":0.000643302,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.401549,"msg":"bootstrap from source requested","bootstrapper":"peers","namespace":"default","numShards":64,"dataRange":3600,"dataFrom":1633649400,"dataTo":1633653000,"indexRange":3600,"indexFrom":1633649400,"indexTo":1633653000}
{"level":"info","ts":1633651508.401734,"msg":"bootstrap from source ready after availability query","bootstrapper":"peers","namespace":"default","numShards":64,"dataRange":0,"indexRange":0}
{"level":"info","ts":1633651508.4017484,"msg":"bootstrap from source hook begin started","bootstrapper":"peers"}
{"level":"info","ts":1633651508.4017565,"msg":"bootstrap from source started","bootstrapper":"peers"}
{"level":"info","ts":1633651508.4017649,"msg":"bootstrap from source hook end started","bootstrapper":"peers","took":0.000016313}
{"level":"info","ts":1633651508.4024665,"msg":"bootstrap from source completed","bootstrapper":"peers","took":0.000016313}
{"level":"info","ts":1633651508.4027529,"msg":"bootstrapping from source completed successfully","bootstrapper":"peers","took":0.000016313,"namespace":"default","numShards":64,"dataRangeRequested":0,"dataRangeFulfilled":0,"indexRangeRequested":0,"indexRangeFulfilled":0,"numIndexBlocks":0}
{"level":"info","ts":1633651508.4027982,"msg":"bootstrap from source requested","bootstrapper":"uninitialized_topology","namespace":"default","numShards":64,"dataRange":3600,"dataFrom":1633649400,"dataTo":1633653000,"indexRange":3600,"indexFrom":1633649400,"indexTo":1633653000}
{"level":"info","ts":1633651508.4029179,"msg":"bootstrap from source ready after availability query","bootstrapper":"uninitialized_topology","namespace":"default","numShards":64,"dataRange":3600,"dataFrom":1633649400,"dataTo":1633653000,"indexRange":3600,"indexFrom":1633649400,"indexTo":1633653000}
{"level":"info","ts":1633651508.402931,"msg":"bootstrap from source hook begin started","bootstrapper":"uninitialized_topology"}
{"level":"info","ts":1633651508.4029386,"msg":"bootstrap from source started","bootstrapper":"uninitialized_topology"}
{"level":"info","ts":1633651508.4030912,"msg":"bootstrap from source hook end started","bootstrapper":"uninitialized_topology","took":0.000159839}
{"level":"info","ts":1633651508.4038315,"msg":"bootstrap from source completed","bootstrapper":"uninitialized_topology","took":0.000159839}
{"level":"info","ts":1633651508.4043508,"msg":"bootstrapping from source completed successfully","bootstrapper":"uninitialized_topology","took":0.000159839,"namespace":"default","numShards":64,"dataRangeRequested":3600,"dataRangeFulfilled":3600,"indexRangeRequested":3600,"indexRangeFulfilled":3600,"numIndexBlocks":0}
{"level":"info","ts":1633651508.4043796,"msg":"bootstrap range completed","bootstrapper":"base","namespace":"default","numShards":64,"dataFrom":1633649400,"dataTo":1633653000,"dataRange":3600,"indexFrom":1633649400,"indexTo":1633653000,"indexRange":3600,"took":0.005909421,"numIndexBlocks":0}
{"level":"info","ts":1633651508.4043941,"msg":"bootstrap succeeded, marking namespaces complete","numShards":64,"bootstrapDuration":0.022917193}
{"level":"info","ts":1633651508.404424,"msg":"bootstrap marking all shards as bootstrapped","namespace":"default","namespace":"default","numShards":64}
{"level":"info","ts":1633651508.4187925,"msg":"bootstrap index with bootstrapped index segments","namespace":"default","numIndexBlocks":0}
{"level":"info","ts":1633651508.4188282,"msg":"bootstrap success","numShards":64,"bootstrapDuration":0.022917193}
{"level":"info","ts":1633651508.4188418,"msg":"bootstrapped"}
{"level":"info","ts":1633651508.419644,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.node.cluster-new-series-insert-limit","revision":0}
{"level":"info","ts":1633651508.4201925,"msg":"creating etcd watch at revision","watch_key":"_kv/default_env/m3db.node.encoders-per-block-limit","revision":0}
{"level":"error","ts":1633651509.2215254,"msg":"error initializing namespaces values, retrying in the background","key":"/namespaces","error":"initializing value error (key='/namespaces'): init watch timeout"}
{"level":"info","ts":1633651509.3703597,"msg":"received kv update","version":1,"key":"/placement"}
{"level":"info","ts":1633651509.370804,"msg":"election manager opened successfully"}
{"level":"info","ts":1633651509.370978,"msg":"election state changed from follower to leader"}
{"level":"info","ts":1633651514.383619,"msg":"cluster db successfully marked shards as available","shards":[36,40,41,1,38,57,3,25,23,45,50,53,61,9,14,15,17,20,31,33,48,4,58,43,19,12,28,34,55,62,5,16,27,60,8,63,22,26,32,35,39,54,0,21,24,42,46,47,49,52,13,59,11,7,10,18,29,30,37,44,6,51,56,2]}
{"level":"info","ts":1633651514.383726,"msg":"successfully parsed placement","version":2}
{"level":"info","ts":1633651514.383958,"msg":"received update from kv topology watch"}
{"level":"info","ts":1633651514.3839757,"msg":"received update for topology"}
{"level":"info","ts":1633651514.3841026,"msg":"successfully updated topology","numHosts":1}


{"level":"info","ts":1633651554.6802778,"msg":"dynamic namespace registry updated to version","version":2}
{"level":"info","ts":1633651554.6803148,"msg":"received update from kv namespace watch"}
{"level":"info","ts":1633651554.6803524,"msg":"refreshed cluster namespaces","added":[],"updated":["default"],"removed":[]}
{"level":"info","ts":1633651554.6803668,"msg":"dynamic namespace registry updated to version","version":2}
{"level":"info","ts":1633651554.680389,"msg":"received update from kv namespace watch"}
{"level":"info","ts":1633651554.680566,"msg":"updating database namespaces","adds":"[]","updates":"[default]","removals":"[]"}
{"level":"warn","ts":1633651554.6839418,"msg":"skipping namespace removals and updates (except schema updates and runtime options), restart the process if you want changes to take effect"}


SIGILL: illegal instruction
PC=0xd01bf0 m=10 sigcode=2

goroutine 17529 [running]:
github.com/m3db/m3/src/x/sync.getCore(0xc013284870, 0x3, 0x3, 0x1, 0x230eba0, 0xc00568f500, 0x22bc140, 0xc00568f500, 0x22bc120, 0xc00568f500, ...)
        /go/src/github.com/m3db/m3/src/x/sync/cpu_linux_amd64.s:9 fp=0xc013250218 sp=0xc013250210 pc=0xd01bf0
github.com/m3db/m3/src/x/sync.CPUCore(...)
        /go/src/github.com/m3db/m3/src/x/sync/index_cpu.go:72
github.com/m3db/m3/src/dbnode/storage.(*dbShardInsertQueue).Insert(0xc066657970, 0xc01324cc60, 0x0, 0x0, 0xed8f18290, 0x32b7fc0, 0x40aa26851eb851ec, 0x2, 0x0, 0x0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/shard_insert_queue.go:291 +0x26b fp=0xc013250288 sp=0xc013250218 pc=0x11a21fb
github.com/m3db/m3/src/dbnode/storage.(*dbShard).insertSeriesAsyncBatched(0xc00568f500, 0x231a940, 0xc093403140, 0x7f7420cbe270, 0xc0685220c0, 0x0, 0x0, 0xed8f18290, 0x32b7fc0, 0x40aa26851eb851ec, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/shard.go:1365 +0x209 fp=0xc013250580 sp=0xc013250288 pc=0x1193919
github.com/m3db/m3/src/dbnode/storage.(*dbShard).writeAndIndex(0xc00568f500, 0x2330140, 0xc082933a40, 0x231a940, 0xc093403140, 0x7f7420cbe270, 0xc0685220c0, 0x0, 0xed8f18290, 0x32b7fc0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/shard.go:1010 +0x24e fp=0xc013250b40 sp=0xc013250580 pc=0x119106e
github.com/m3db/m3/src/dbnode/storage.(*dbShard).WriteTagged(0xc00568f500, 0x2330140, 0xc082933a40, 0x231a940, 0xc093403140, 0x7f7420cbe270, 0xc0685220c0, 0x0, 0xed8f18290, 0x32b7fc0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/shard.go:897 +0x15e fp=0xc013250de0 sp=0xc013250b40 pc=0x1190b7e
github.com/m3db/m3/src/dbnode/storage.(*dbNamespace).WriteTagged(0xc000386700, 0x2330140, 0xc082933a40, 0x231a940, 0xc093403140, 0x7f7420cbe270, 0xc0685220c0, 0x0, 0xed8f18290, 0x32b7fc0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/namespace.go:730 +0x35c fp=0xc0132510f0 sp=0xc013250de0 pc=0x116fe6c
github.com/m3db/m3/src/dbnode/storage.(*db).writeBatch(0xc0000ac000, 0x2330140, 0xc082933a40, 0x231a940, 0xc0934030e0, 0x7f7420cbe238, 0xc09010c5a0, 0x22bbf00, 0xc0933f2e00, 0xed8f18201, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/database.go:782 +0x34f fp=0xc013251808 sp=0xc0132510f0 pc=0x113ecbf
github.com/m3db/m3/src/dbnode/storage.(*db).WriteTaggedBatch(0xc0000ac000, 0x2330140, 0xc082933a40, 0x231a940, 0xc0934030e0, 0x7f7420cbe238, 0xc09010c5a0, 0x22bbf00, 0xc0933f2e00, 0x0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/storage/database.go:740 +0x99 fp=0xc013251878 sp=0xc013251808 pc=0x113e939
github.com/m3db/m3/src/dbnode/storage/cluster.(*clusterDB).WriteTaggedBatch(0xc004e247e0, 0x2330140, 0xc082933a40, 0x231a940, 0xc0934030e0, 0x7f7420cbe238, 0xc09010c5a0, 0x22bbf00, 0xc0933f2e00, 0x0, ...)
        <autogenerated>:1 +0xa9 fp=0xc0132518e0 sp=0xc013251878 pc=0x183d2f9
github.com/m3db/m3/src/dbnode/network/server/tchannelthrift/node.(*service).WriteTaggedBatchRaw(0xc092bd3200, 0x7f7425640580, 0xc0131bfae0, 0xc0132ac120, 0x0, 0x0)
        /go/src/github.com/m3db/m3/src/dbnode/network/server/tchannelthrift/node/service.go:1711 +0xa82 fp=0xc013251ad8 sp=0xc0132518e0 pc=0x18254c2
github.com/m3db/m3/src/dbnode/generated/thrift/rpc.(*tchanNodeServer).handleWriteTaggedBatchRaw(0xc09f14ad60, 0x7f7425640580, 0xc0131bfae0, 0x234adc0, 0xc09f2326c0, 0x1ce67e0, 0xc0056fbba8, 0x47c7c0, 0x3295c00, 0xc00a6f4c80)
        /go/src/github.com/m3db/m3/src/dbnode/generated/thrift/rpc/tchan-rpc.go:2015 +0xc4 fp=0xc013251b30 sp=0xc013251ad8 pc=0xecb1f4
github.com/m3db/m3/src/dbnode/generated/thrift/rpc.(*tchanNodeServer).Handle(0xc09f14ad60, 0x7f7425640580, 0xc0131bfae0, 0xc01321cc26, 0x13, 0x234adc0, 0xc09f2326c0, 0xc0132ac090, 0xc01329ae00, 0x6dcb17, ...)
        /go/src/github.com/m3db/m3/src/dbnode/generated/thrift/rpc/tchan-rpc.go:1162 +0x16c1 fp=0xc013251bb8 sp=0xc013251b30 pc=0xec5ed1
github.com/uber/tchannel-go/thrift.(*Server).handle(0xc09f0fdd10, 0x2306560, 0xc0131bfa50, 0x22f05a0, 0xc09f14ad60, 0x1f990e0, 0xc01321cc26, 0x13, 0xc013296180, 0x0, ...)
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/thrift/server.go:163 +0x3ad fp=0xc013251d40 sp=0xc013251bb8 pc=0xe02b0d
github.com/uber/tchannel-go/thrift.(*Server).Handle(0xc09f0fdd10, 0x2306560, 0xc0131bfa50, 0xc013296180)
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/thrift/server.go:237 +0x182 fp=0xc013251e08 sp=0xc013251d40 pc=0xe03642
github.com/uber/tchannel-go.(*handlerMap).Handle(0xc09f13fa60, 0x2306560, 0xc0131bfa50, 0xc013296180)
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/handlers.go:118 +0xf9 fp=0xc013251f10 sp=0xc013251e08 pc=0xdc7d49
github.com/uber/tchannel-go.channelHandler.Handle(0xc00042ce00, 0x2306560, 0xc0131bfa50, 0xc013296180)
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/handlers.go:126 +0x8c fp=0xc013251f58 sp=0xc013251f10 pc=0xdc830c
github.com/uber/tchannel-go.(*Connection).dispatchInbound(0xc094d43900, 0x1500000005, 0xc013296180, 0xc0a06f5da0)
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/inbound.go:194 +0x362 fp=0xc013251fc0 sp=0xc013251f58 pc=0xdcb1e2
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc013251fc8 sp=0xc013251fc0 pc=0x45f0a1
created by github.com/uber/tchannel-go.(*Connection).handleCallReq
        /go/pkg/mod/github.com/uber/tchannel-go@v1.14.0/inbound.go:125 +0xc97

goroutine 1 [chan receive]:
github.com/m3db/m3/src/x/os.WaitForInterrupt(0xc00059cea0, 0xc00059cd80)
        /go/src/github.com/m3db/m3/src/x/os/interrupt.go:51 +0x9d
github.com/m3db/m3/src/dbnode/server.Run(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/m3db/m3/src/dbnode/server/server.go:993 +0x8685
main.main()
        /go/src/github.com/m3db/m3/src/cmd/services/m3dbnode/main/main.go:90 +0x31c

goroutine 17 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 125 [chan receive, 1 minutes]:
github.com/m3db/m3/src/x/os.NewInterruptChannel.func1(0x2, 0xc00059cd80)
        /go/src/github.com/m3db/m3/src/x/os/interrupt.go:59 +0xe2
created by github.com/m3db/m3/src/x/os.NewInterruptChannel
        /go/src/github.com/m3db/m3/src/x/os/interrupt.go:58 +0x68

goroutine 30 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc00017ba60)
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x80

goroutine 117 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0006de7a0)
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x80

goroutine 96 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0003a4060)
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
        /go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20200824191128-ae9734ed278b/pkg/logutil/merge_logger.go:91 +0x80

goroutine 126 [chan receive, 1 minutes]:
github.com/m3db/m3/src/x/os.WaitForInterrupt(0xc000111620, 0xc00059cd80)
        /go/src/github.com/m3db/m3/src/x/os/interrupt.go:51 +0x9d
github.com/m3db/m3/src/query/server.Run(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/m3db/m3/src/query/server/query.go:600 +0x4a18
main.main.func1(0xc00058abf0, 0xc00059cde0, 0xc00059ce40, 0xc00059cd80, 0xc000478060)
        /go/src/github.com/m3db/m3/src/cmd/services/m3dbnode/main/main.go:78 +0xe4
created by main.main
        /go/src/github.com/m3db/m3/src/cmd/services/m3dbnode/main/main.go:77 +0x3a8

...

goroutine 14662 [select]:
github.com/m3db/m3/src/cluster/etcd/watchmanager.(*manager).Watch(0xc063d23f80, 0xc00592fd40, 0x39)
        /go/src/github.com/m3db/m3/src/cluster/etcd/watchmanager/manager.go:170 +0x490
created by github.com/m3db/m3/src/cluster/kv/etcd.(*client).Watch
        /go/src/github.com/m3db/m3/src/cluster/kv/etcd/store.go:364 +0x20e

goroutine 15005 [chan receive]:
github.com/m3db/m3/src/dbnode/namespace.(*runtimeOptionsManager).RegisterListener.func1(0x22f3d60, 0xc00bd8e630, 0x22bc2c0, 0xc00acb1680)
        /go/src/github.com/m3db/m3/src/dbnode/namespace/namespace_runtime_options.go:244 +0xa5
created by github.com/m3db/m3/src/dbnode/namespace.(*runtimeOptionsManager).RegisterListener
        /go/src/github.com/m3db/m3/src/dbnode/namespace/namespace_runtime_options.go:243 +0xf9

goroutine 17528 [IO wait]:
internal/poll.runtime_pollWait(0x7f7424058f60, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc012dae618, 0x72, 0x0, 0x1, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc012dae600, 0xc013241a21, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc012dae600, 0xc013241a21, 0x1, 0x1, 0x0, 0xc09f1119d0, 0xc0109f2fb0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00edff1e0, 0xc013241a21, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).backgroundRead(0xc013241a10)
        /usr/local/go/src/net/http/server.go:688 +0x58
created by net/http.(*connReader).startBackgroundRead
        /usr/local/go/src/net/http/server.go:684 +0xd4

rax    0xc066657970
rbx    0xc066657970
rcx    0xc005a54780
rdx    0xc01324cc60
rdi    0xc0132503c0
rsi    0xc013250570
rbp    0xc013250278
rsp    0xc013250210
r8     0x0
r9     0x203004
r10    0x14
r11    0xfc
r12    0xfc
r13    0x0
r14    0x227f8d0
r15    0x0
rip    0xd01bf0
rflags 0x10206
cs     0x33
fs     0x0
gs     0x0

Jerry Zhen · Answer 1 · Fri Oct 08 2021 11:19:20 GMT+0800 (China Standard Time)

Is this something related to the support of rdtscp?

Gibbs Cullen · Answer 2 · Thu Oct 21 2021 04:42:22 GMT+0800 (China Standard Time)

@jerryzhen01 I would refer to issue 3659 and 3105 that had a similar issue.

Jerry Zhen · Answer 3 · Thu Oct 21 2021 14:46:02 GMT+0800 (China Standard Time)

@gibbscullen thank you for the reply, will close this issue.