The mon container crashes when bootstrapping cephadm on Raspberry Pi 4.

Question

The mon container crashes when bootstrapping cephadm on Raspberry Pi 4.

penM000 opened this issue 3 years ago · comments

penM000 commented 3 years ago

Is this a bug report or feature request?

Bug Report

Bug Report

What happened:
Cephadm bootstrapping does not complete on raspbeerypi4.
This is caused by a crash of the mon container.

What you expected to happen:
Bootstrapping complete.

How to reproduce it (minimal and precise):

Write ubuntu to MicroSD.
Package update and docker installation.

Follow "https://docs.ceph.com/en/latest/cephadm/install/" to perform bootstrapping.

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm
chmod +x cephadm
sudo ./ cephadm bootstrap --mon-ip 192.168.1.23
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit systemd-timesyncd.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit systemd-timesyncd.service is enabled and running
Host looks OK
Cluster fsid: fec36af4-94de-11eb-bfa3-074bf963131c
Verifying IP 192.168.1.23 port 3300 ...
Verifying IP 192.168.1.23 port 6789 ...
Mon IP 192.168.1.23 is in CIDR network 192.168.1.0/24
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image docker.io/ceph/ceph:v16...
Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...

"Waiting for mon..." too long
If you check with "docker ps -a" while implementing the above, you can see ceph's mon start and fail for a moment.
I have attached the container log as a crash log below.

Environment:

OS (e.g. from /etc/os-release):

ubuntu@rpi-node3:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Kernel (e.g. uname -a):

ubuntu@rpi-node3:~$ uname -a
Linux rpi-node3 5.4.0-1032-raspi #35-Ubuntu SMP PREEMPT Fri Mar 19 20:52:40 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Docker version (e.g. docker version):

ubuntu@rpi-node3:~$ docker version 
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:19:31 2021
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:17:22 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Ceph version (e.g. ceph -v):

ubuntu@rpi-node3:~$ sudo cephadm version
Using recent ceph image ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)

Crush log

ubuntu@rpi-node3:~$ docker logs ceph-93b0da22-93b6-11eb-bd37-a77433dba045-mon.rpi-node3 debug 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7 debug 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 pidfile_write: ignore empty --pid-file debug 2021-04-02T13:26:33.159+0000 ffff8fd88040 0 load: jerasure load: lrc load: isa debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: RocksDB version: 6.8.1

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Compile date Mar 30 2021
debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: DB SUMMARY

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: CURRENT file: CURRENT

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: IDENTITY file: IDENTITY

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-rpi-node3/store.db dir, Total Num: 2, files: 000004.sst 000007.sst

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-rpi-node3/store.db: 000006.log size: 3462 ;

debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: debug 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 Options.error_if_exists: 0
Options.create_if_missing: 0
Options.paranoid_checks: 1
Options.env: 0xaaaab3820a38
Options.fs: Posix File System
Options.info_log: 0xaaaad7a81ec0
Options.max_file_opening_threads: 16
Options.statistics: (nil)
Options.use_fsync: 0
Options.max_log_file_size: 0
Options.max_manifest_file_size: 1073741824
Options.log_file_time_to_roll: 0
Options.keep_log_file_num: 1000
Options.recycle_log_file_num: 0
Options.allow_fallocate: 1
Options.allow_mmap_reads: 0
Options.allow_mmap_writes: 0
Options.use_direct_reads: 0
Options.use_direct_io_for_flush_and_compaction: 0
Options.create_missing_column_families: 0
Options.db_log_dir:
Options.wal_dir: /var/lib/ceph/mon/ceph-rpi-node3/store.db
Options.table_cache_numshardbits: 6
Options.max_subcompactions: 1
Options.max_background_flushes: -1
Options.WAL_ttl_seconds: 0
Options.WAL_size_limit_MB: 0
Options.max_write_batch_group_size_bytes: 1048576
Options.manifest_preallocation_size: 4194304
Options.is_fd_close_on_exec: 1
Options.advise_random_on_open: 1
Options.db_write_buffer_size: 0
Options.write_buffer_manager: 0xaaaad7b17f20
Options.access_hint_on_compaction_start: 1
Options.new_table_reader_for_compaction_inputs: 0
Options.random_access_max_buffer_size: 1048576
Options.use_adaptive_mutex: 0
Options.rate_limiter: (nil)
Options.sst_file_manager.rate_bytes_per_sec: 0
Options.wal_recovery_mode: 2
Options.enable_thread_tracking: 0
Options.enable_pipelined_write: 0
Options.unordered_write: 0
Options.allow_concurrent_memtable_write: 1
Options.enable_write_thread_adaptive_yield: 1
Options.write_thread_max_yield_usec: 100
Options.write_thread_slow_yield_usec: 3
Options.row_cache: None
Options.wal_filter: None
Options.avoid_flush_during_recovery: 0
Options.allow_ingest_behind: 0
Options.preserve_deletes: 0
Options.two_write_queues: 0
Options.manual_wal_flush: 0
Options.atomic_flush: 0
Options.avoid_unnecessary_blocking_io: 0
Options.persist_stats_to_disk: 0
Options.write_dbid_to_manifest: 0
Options.log_readahead_size: 0
Options.sst_file_checksum_func: Unknown
Options.max_background_jobs: 2
Options.max_background_compactions: -1
Options.avoid_flush_during_shutdown: 0
Options.writable_file_max_buffer_size: 1048576
Options.delayed_write_rate : 16777216
Options.max_total_wal_size: 0
Options.delete_obsolete_files_period_micros: 21600000000
Options.stats_dump_period_sec: 600
Options.stats_persist_period_sec: 600
Options.stats_history_buffer_size: 1048576
Options.max_open_files: -1
Options.bytes_per_sync: 0
Options.wal_bytes_per_sync: 0
Options.strict_bytes_per_sync: 0
Options.compaction_readahead_size: 0
rocksdb: Compression algorithms supported:
kZSTDNotFinalCompression supported: 0
kZSTD supported: 0
kXpressCompression supported: 0
kLZ4HCCompression supported: 1
kLZ4Compression supported: 1
kBZip2Compression supported: 0
kZlibCompression supported: 1
kSnappyCompression supported: 1
rocksdb: Fast CRC32 supported: Supported on Arm64
rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-rpi-node3/store.db/MANIFEST-000005

debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]:

debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.merge_operator:
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_filter: None
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_filter_factory: None
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_factory: SkipListFactory
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.table_factory: BlockBasedTable
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0xaaaad7a25c00)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0xaaaad7a5cd10
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0

debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.write_buffer_size: 33554432
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_number: 2
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression: NoCompression
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression: Disabled
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.prefix_extractor: nullptr
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.num_levels: 7
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.level: 32767
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.enabled: false
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.window_bits: -14
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.level: 32767
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.strategy: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.enabled: false
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_stop_writes_trigger: 36
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.target_file_size_base: 67108864
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.target_file_size_multiplier: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_base: 268435456
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_compaction_bytes: 1677721600
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.arena_block_size: 4194304
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.disable_auto_compactions: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.table_properties_collectors:
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.inplace_update_support: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.inplace_update_num_locks: 10000
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_whole_key_filtering: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_huge_page_size: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bloom_locality: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_successive_merges: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.optimize_filters_for_hits: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.paranoid_file_checks: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.force_consistency_checks: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.report_bg_io_stats: 0
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.ttl: 2592000
debug 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.periodic_compaction_seconds: 0
debug 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-rpi-node3/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

debug 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4

debug 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1617369993178201, "job": 1, "event": "recovery_started", "log_files": [6]}
debug 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2
debug 2021-04-02T13:26:33.179+0000 ffff8fd88040 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
*** Caught signal (Illegal instruction) **
in thread ffff8fd88040 thread_name:ceph-mon
ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
1: __kernel_rt_sigreturn()
2: (crc32c_arm64(unsigned int, unsigned char const*, unsigned int)+0x45c) [0xaaaab35d0eb4]
3: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x59c) [0xaaaab362b69c]
4: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaab362b8ec]
5: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaab362be88]
6: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaab362bf04]
7: (rocksdb::BlockBasedTableBuilder::Finish()+0x28) [0xaaaab362e6f8]
8: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBaserocksdb::Slice, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator > > >, rocksdb::FileMetaData, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<unsigned long, std::allocator >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x244c) [0xaaaab35e57e4]
9: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaab34779c4]
10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator > const&, unsigned long*, bool, bool*)+0x14d4) [0xaaaab3479854]
11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaab347aa2c]
12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB**, bool, bool)+0x464) [0xaaaab34758e4]
13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocatorrocksdb::ColumnFamilyHandle* >*, rocksdb::DB**)+0x1c) [0xaaaab34769bc]
14: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0xc98) [0xaaaab33fdc80]
15: (MonitorDBStore::open(std::ostream&)+0x194) [0xaaaab318dc3c]
16: main()
17: __libc_start_main()
18: /usr/bin/ceph-mon(+0x2464a0) [0xaaaab31804a0]
debug 2021-04-02T13:26:33.187+0000 ffff8fd88040 -1 *** Caught signal (Illegal instruction) **
in thread ffff8fd88040 thread_name:ceph-mon

ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
1: __kernel_rt_sigreturn()
2: (crc32c_arm64(unsigned int, unsigned char const*, unsigned int)+0x45c) [0xaaaab35d0eb4]
3: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x59c) [0xaaaab362b69c]
4: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaab362b8ec]
5: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaab362be88]
6: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaab362bf04]
7: (rocksdb::BlockBasedTableBuilder::Finish()+0x28) [0xaaaab362e6f8]
8: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBaserocksdb::Slice, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator > > >, rocksdb::FileMetaData, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<unsigned long, std::allocator >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x244c) [0xaaaab35e57e4]
9: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaab34779c4]
10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator > const&, unsigned long*, bool, bool*)+0x14d4) [0xaaaab3479854]
11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaab347aa2c]
12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB**, bool, bool)+0x464) [0xaaaab34758e4]
13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocatorrocksdb::ColumnFamilyHandle* >*, rocksdb::DB**)+0x1c) [0xaaaab34769bc]
14: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0xc98) [0xaaaab33fdc80]
15: (MonitorDBStore::open(std::ostream&)+0x194) [0xaaaab318dc3c]
16: main()
17: __libc_start_main()
18: /usr/bin/ceph-mon(+0x2464a0) [0xaaaab31804a0]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.

--- begin dump of recent events ---
debug -219> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command assert hook 0xaaaad7a20530
debug -218> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command abort hook 0xaaaad7a20530
debug -217> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command leak_some_memory hook 0xaaaad7a20530
debug -216> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perfcounters_dump hook 0xaaaad7a20530
debug -215> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 1 hook 0xaaaad7a20530
debug -214> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf dump hook 0xaaaad7a20530
debug -213> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perfcounters_schema hook 0xaaaad7a20530
debug -212> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf histogram dump hook 0xaaaad7a20530
debug -211> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 2 hook 0xaaaad7a20530
debug -210> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf schema hook 0xaaaad7a20530
debug -209> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf histogram schema hook 0xaaaad7a20530
debug -208> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf reset hook 0xaaaad7a20530
debug -207> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config show hook 0xaaaad7a20530
debug -206> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config help hook 0xaaaad7a20530
debug -205> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config set hook 0xaaaad7a20530
debug -204> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config unset hook 0xaaaad7a20530
debug -203> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config get hook 0xaaaad7a20530
debug -202> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config diff hook 0xaaaad7a20530
debug -201> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config diff get hook 0xaaaad7a20530
debug -200> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command injectargs hook 0xaaaad7a20530
debug -199> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log flush hook 0xaaaad7a20530
debug -198> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log dump hook 0xaaaad7a20530
debug -197> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log reopen hook 0xaaaad7a20530
debug -196> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command dump_mempools hook 0xaaaad7ac0328
debug -195> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 set uid:gid to 167:167 (ceph:ceph)
debug -194> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7
debug -193> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 pidfile_write: ignore empty --pid-file
debug -192> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) init /var/run/ceph/ceph-mon.rpi-node3.asok
debug -191> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) bind_and_listen /var/run/ceph/ceph-mon.rpi-node3.asok
debug -190> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 0 hook 0xaaaad7a25c10
debug -189> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command version hook 0xaaaad7a25c10
debug -188> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command git_version hook 0xaaaad7a25c10
debug -187> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command help hook 0xaaaad7a205c0
debug -186> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command get_command_descriptions hook 0xaaaad7a201e0
debug -185> 2021-04-02T13:26:33.147+0000 ffff85134200 5 asok(0xaaaad7ad4000) entry start
debug -184> 2021-04-02T13:26:33.159+0000 ffff8fd88040 0 load: jerasure load: lrc load: isa
debug -183> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -182> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option write_buffer_size = 33554432
debug -181> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option compression = kNoCompression
debug -180> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -179> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option write_buffer_size = 33554432
debug -178> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option compression = kNoCompression
debug -177> 2021-04-02T13:26:33.163+0000 ffff8fd88040 5 rocksdb: verify_sharding column families from rocksdb: [default]
debug -176> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: RocksDB version: 6.8.1

debug -175> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
debug -174> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Compile date Mar 30 2021
debug -173> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: DB SUMMARY

debug -172> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: CURRENT file: CURRENT

debug -171> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: IDENTITY file: IDENTITY

debug -170> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes

debug -169> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-rpi-node3/store.db dir, Total Num: 2, files: 000004.sst 000007.sst

debug -168> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-rpi-node3/store.db: 000006.log size: 3462 ;

debug -167> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.error_if_exists: 0
debug -166> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.create_if_missing: 0
debug -165> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.paranoid_checks: 1
debug -164> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.env: 0xaaaab3820a38
debug -163> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.fs: Posix File System
debug -162> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.info_log: 0xaaaad7a81ec0
debug -161> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_file_opening_threads: 16
debug -160> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.statistics: (nil)
debug -159> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.use_fsync: 0
debug -158> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_log_file_size: 0
debug -157> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_manifest_file_size: 1073741824
debug -156> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.log_file_time_to_roll: 0
debug -155> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.keep_log_file_num: 1000
debug -154> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.recycle_log_file_num: 0
debug -153> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.allow_fallocate: 1
debug -152> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.allow_mmap_reads: 0
debug -151> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.allow_mmap_writes: 0
debug -150> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.use_direct_reads: 0
debug -149> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
debug -148> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.create_missing_column_families: 0
debug -147> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.db_log_dir:
debug -146> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-rpi-node3/store.db
debug -145> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.table_cache_numshardbits: 6
debug -144> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_subcompactions: 1
debug -143> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_background_flushes: -1
debug -142> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.WAL_ttl_seconds: 0
debug -141> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.WAL_size_limit_MB: 0
debug -140> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576
debug -139> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.manifest_preallocation_size: 4194304
debug -138> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.is_fd_close_on_exec: 1
debug -137> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.advise_random_on_open: 1
debug -136> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.db_write_buffer_size: 0
debug -135> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.write_buffer_manager: 0xaaaad7b17f20
debug -134> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.access_hint_on_compaction_start: 1
debug -133> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
debug -132> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.random_access_max_buffer_size: 1048576
debug -131> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.use_adaptive_mutex: 0
debug -130> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.rate_limiter: (nil)
debug -129> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
debug -128> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.wal_recovery_mode: 2
debug -127> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.enable_thread_tracking: 0
debug -126> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.enable_pipelined_write: 0
debug -125> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.unordered_write: 0
debug -124> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.allow_concurrent_memtable_write: 1
debug -123> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
debug -122> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.write_thread_max_yield_usec: 100
debug -121> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.write_thread_slow_yield_usec: 3
debug -120> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.row_cache: None
debug -119> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.wal_filter: None
debug -118> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.avoid_flush_during_recovery: 0
debug -117> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.allow_ingest_behind: 0
debug -116> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.preserve_deletes: 0
debug -115> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.two_write_queues: 0
debug -114> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.manual_wal_flush: 0
debug -113> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.atomic_flush: 0
debug -112> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0
debug -111> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.persist_stats_to_disk: 0
debug -110> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.write_dbid_to_manifest: 0
debug -109> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.log_readahead_size: 0
debug -108> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.sst_file_checksum_func: Unknown
debug -107> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_background_jobs: 2
debug -106> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_background_compactions: -1
debug -105> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.avoid_flush_during_shutdown: 0
debug -104> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
debug -103> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.delayed_write_rate : 16777216
debug -102> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_total_wal_size: 0
debug -101> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
debug -100> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.stats_dump_period_sec: 600
debug -99> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.stats_persist_period_sec: 600
debug -98> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.stats_history_buffer_size: 1048576
debug -97> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.max_open_files: -1
debug -96> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.bytes_per_sync: 0
debug -95> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.wal_bytes_per_sync: 0
debug -94> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.strict_bytes_per_sync: 0
debug -93> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Options.compaction_readahead_size: 0
debug -92> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Compression algorithms supported:
debug -91> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kZSTDNotFinalCompression supported: 0
debug -90> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kZSTD supported: 0
debug -89> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kXpressCompression supported: 0
debug -88> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kLZ4HCCompression supported: 1
debug -87> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kLZ4Compression supported: 1
debug -86> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kBZip2Compression supported: 0
debug -85> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kZlibCompression supported: 1
debug -84> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: kSnappyCompression supported: 1
debug -83> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: Fast CRC32 supported: Supported on Arm64
debug -82> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-rpi-node3/store.db/MANIFEST-000005

debug -81> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]:

debug -80> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
debug -79> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.merge_operator:
debug -78> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_filter: None
debug -77> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_filter_factory: None
debug -76> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_factory: SkipListFactory
debug -75> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.table_factory: BlockBasedTable
debug -74> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0xaaaad7a25c00)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 0
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0xaaaad7a5cd10
block_cache_name: BinnedLRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 2
enable_index_compression: 1
block_align: 0

debug -73> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.write_buffer_size: 33554432
debug -72> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_number: 2
debug -71> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression: NoCompression
debug -70> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression: Disabled
debug -69> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.prefix_extractor: nullptr
debug -68> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
debug -67> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.num_levels: 7
debug -66> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
debug -65> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
debug -64> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0
debug -63> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14
debug -62> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.level: 32767
debug -61> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.strategy: 0
debug -60> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0
debug -59> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0
debug -58> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bottommost_compression_opts.enabled: false
debug -57> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.window_bits: -14
debug -56> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.level: 32767
debug -55> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.strategy: 0
debug -54> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
debug -53> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0
debug -52> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compression_opts.enabled: false
debug -51> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
debug -50> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
debug -49> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level0_stop_writes_trigger: 36
debug -48> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.target_file_size_base: 67108864
debug -47> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.target_file_size_multiplier: 1
debug -46> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_base: 268435456
debug -45> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1
debug -44> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
debug -43> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
debug -42> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
debug -41> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
debug -40> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
debug -39> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
debug -38> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
debug -37> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
debug -36> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
debug -35> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_compaction_bytes: 1677721600
debug -34> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.arena_block_size: 4194304
debug -33> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
debug -32> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
debug -31> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
debug -30> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.disable_auto_compactions: 0
debug -29> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
debug -28> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio
debug -27> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
debug -26> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
debug -25> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
debug -24> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
debug -23> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
debug -22> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
debug -21> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
debug -20> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0
debug -19> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.table_properties_collectors:
debug -18> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.inplace_update_support: 0
debug -17> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.inplace_update_num_locks: 10000
debug -16> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
debug -15> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_whole_key_filtering: 0
debug -14> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.memtable_huge_page_size: 0
debug -13> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.bloom_locality: 0
debug -12> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.max_successive_merges: 0
debug -11> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.optimize_filters_for_hits: 0
debug -10> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.paranoid_file_checks: 0
debug -9> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.force_consistency_checks: 0
debug -8> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.report_bg_io_stats: 0
debug -7> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.ttl: 2592000
debug -6> 2021-04-02T13:26:33.167+0000 ffff8fd88040 4 rocksdb: Options.periodic_compaction_seconds: 0
debug -5> 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-rpi-node3/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

debug -4> 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4

debug -3> 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1617369993178201, "job": 1, "event": "recovery_started", "log_files": [6]}
debug -2> 2021-04-02T13:26:33.171+0000 ffff8fd88040 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2
debug -1> 2021-04-02T13:26:33.179+0000 ffff8fd88040 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
debug 0> 2021-04-02T13:26:33.187+0000 ffff8fd88040 -1 *** Caught signal (Illegal instruction) **
in thread ffff8fd88040 thread_name:ceph-mon

ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
1: __kernel_rt_sigreturn()
2: (crc32c_arm64(unsigned int, unsigned char const*, unsigned int)+0x45c) [0xaaaab35d0eb4]
3: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x59c) [0xaaaab362b69c]
4: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x1b4) [0xaaaab362b8ec]
5: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x40) [0xaaaab362be88]
6: (rocksdb::BlockBasedTableBuilder::Flush()+0x64) [0xaaaab362bf04]
7: (rocksdb::BlockBasedTableBuilder::Finish()+0x28) [0xaaaab362e6f8]
8: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBaserocksdb::Slice, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_deleterocksdb::FragmentedRangeTombstoneIterator > > >, rocksdb::FileMetaData, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_deleterocksdb::IntTblPropCollectorFactory > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<unsigned long, std::allocator >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0x244c) [0xaaaab35e57e4]
9: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x804) [0xaaaab34779c4]
10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator > const&, unsigned long*, bool, bool*)+0x14d4) [0xaaaab3479854]
11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, bool, bool, bool, unsigned long*)+0x8ec) [0xaaaab347aa2c]
12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB**, bool, bool)+0x464) [0xaaaab34758e4]
13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle, std::allocatorrocksdb::ColumnFamilyHandle* >*, rocksdb::DB**)+0x1c) [0xaaaab34769bc]
14: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0xc98) [0xaaaab33fdc80]
15: (MonitorDBStore::open(std::ostream&)+0x194) [0xaaaab318dc3c]
16: main()
17: __libc_start_main()
18: /usr/bin/ceph-mon(+0x2464a0) [0xaaaab31804a0]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
281472914375168 / admin_socket
281473095073856 / ceph-mon
max_recent 10000
max_new 10000
log_file
--- end dump of recent events ---
--- begin dump of recent events ---
debug -219> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command assert hook 0xaaaad7a20530
debug -218> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command abort hook 0xaaaad7a20530
debug -217> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command leak_some_memory hook 0xaaaad7a20530
debug -216> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perfcounters_dump hook 0xaaaad7a20530
debug -215> 2021-04-02T13:26:33.123+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 1 hook 0xaaaad7a20530
debug -214> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf dump hook 0xaaaad7a20530
debug -213> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perfcounters_schema hook 0xaaaad7a20530
debug -212> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf histogram dump hook 0xaaaad7a20530
debug -211> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 2 hook 0xaaaad7a20530
debug -210> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf schema hook 0xaaaad7a20530
debug -209> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf histogram schema hook 0xaaaad7a20530
debug -208> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command perf reset hook 0xaaaad7a20530
debug -207> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config show hook 0xaaaad7a20530
debug -206> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config help hook 0xaaaad7a20530
debug -205> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config set hook 0xaaaad7a20530
debug -204> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config unset hook 0xaaaad7a20530
debug -203> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config get hook 0xaaaad7a20530
debug -202> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config diff hook 0xaaaad7a20530
debug -201> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command config diff get hook 0xaaaad7a20530
debug -200> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command injectargs hook 0xaaaad7a20530
debug -199> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log flush hook 0xaaaad7a20530
debug -198> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log dump hook 0xaaaad7a20530
debug -197> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command log reopen hook 0xaaaad7a20530
debug -196> 2021-04-02T13:26:33.127+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command dump_mempools hook 0xaaaad7ac0328
debug -195> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 set uid:gid to 167:167 (ceph:ceph)
debug -194> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7
debug -193> 2021-04-02T13:26:33.143+0000 ffff8fd88040 0 pidfile_write: ignore empty --pid-file
debug -192> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) init /var/run/ceph/ceph-mon.rpi-node3.asok
debug -191> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) bind_and_listen /var/run/ceph/ceph-mon.rpi-node3.asok
debug -190> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command 0 hook 0xaaaad7a25c10
debug -189> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command version hook 0xaaaad7a25c10
debug -188> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command git_version hook 0xaaaad7a25c10
debug -187> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command help hook 0xaaaad7a205c0
debug -186> 2021-04-02T13:26:33.147+0000 ffff8fd88040 5 asok(0xaaaad7ad4000) register_command get_command_descriptions hook 0xaaaad7a201e0
debug -185> 2021-04-02T13:26:33.147+0000 ffff85134200 5 asok(0xaaaad7ad4000) entry start
debug -184> 2021-04-02T13:26:33.159+0000 ffff8fd88040 0 load: jerasure load: lrc load: isa
debug -183> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -182> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option write_buffer_size = 33554432
debug -181> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option compression = kNoCompression
debug -180> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option level_compaction_dynamic_level_bytes = true
debug -179> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option write_buffer_size = 33554432
debug -178> 2021-04-02T13:26:33.163+0000 ffff8fd88040 1 set rocksdb option compression = kNoCompression
debug -177> 2021-04-02T13:26:33.163+0000 ffff8fd88040 5 rocksdb: verify_sharding column families from rocksdb: [default]
debug -176> 2021-04-02T13:26:33.163+0000 ffff8fd88040 4 rocksdb: RocksDB version: 6.8.1