How to properly configure chacheLib to work with Nvme drive

Question

How to properly configure chacheLib to work with Nvme drive

MeirHebrew opened this issue a year ago · comments

Can you please add a fully working example of how to set up a working hybrid Cachelib configuration (i.e., RAM + SSD)?

I try to understand how to do it properly, this is how I configured cache lib:

// Big hash configurations
static int gBigHashSizePct = GetUintFromEnv("NVME_BIGHASH_SIZE_PCT"); // bigHashSizePct (0-100), the % of the file that bigHash will use
static int gBigHashSmallItemMaxSize = GetUintFromEnv("NVME_BIGHASH_SMALL_ITEM_MAX_SIZE"); // bigHashSmallItemMaxSize must be smaller then bucket size which is 4096 by default
static int gBigHashBucketSize = GetUintFromEnv("NVME_BIGHASH_BUCKET_SIZE"); // default 4096
static int gBigHashBucketBfSize = GetUintFromEnv("NVME_BIGHASH_BLOOM_FILTER_PER_BUCKET_SIZE"); //  bloom filter size per bucket, default is 8


// Allow control for admission probability to Nvme
float gAdmissionProbability = std::stof(std::getenv("NVME_ADDMISSION_PROBABILITY")); // should be 0 - 1

// Allow control on max msg in and max msg out sizes
static int gMaxGrpcRecivedSize = GetUintFromEnv("MAX_GRPC_MSG_RECIVED_SIZE"); // Default will be 50 mb (50000000)
static int gMaxGrpcSendSize = GetUintFromEnv("MAX_GRPC_MSG_SEND_SIZE"); // Default will be 50 mb (50000000)


static uint32_t gMaxChainedSize = GetUintFromEnv("MAX_CHAINED_SIZE");
static uint64_t gCacheRamSize = InitgCacheSize("CACHE_RAM_SIZE"); 
static uint32_t gNumbersOfItemsWePlanToCache = GetUintFromEnv("NUMBER_OF_ITEMS_TO_CACHE"); 
static uint64_t gCacheSize =  InitgCacheSize("CACHE_NVME_SIZE"); // the nvme cache size
static uint32_t gBlockSize = GetUintFromEnv("BLOCK_SIZE"); 
static uint32_t gRegionSize = GetUintFromEnv("REGION_SIZE"); 
static std::string gCacheFilePath = std::getenv("CACHE_FILE");

using Cache = cachelib::LruAllocator; // or Lru2QAllocator, or TinyLFUAllocator
using CacheConfig = typename Cache::Config;
using CacheKey = typename Cache::Key;
using CacheReadHandle = typename Cache::ReadHandle;

// Global cache object and a default cache pool
std::unique_ptr<Cache> gCache_;
cachelib::PoolId defaultPool_;

void init()
{
    // Ram config
    CacheConfig config;
    config
    .setCacheSize(gCacheRamSize) // 1GB = 1 * 1024 * 1024 * 1024
    .setCacheName("My Case")
    .setAccessConfig(gNumbersOfItemsWePlanToCache) // assuming caching 20 = {25 /* bucket power */, 10 /* lock power */})
                                                    // million items
    .validate(); // will throw if bad config
            
    // Hybrid config (Ram 6 Nvme)
    //CacheConfig lruConfig;
    Cache::NvmCacheConfig nvmConfig;
    nvmConfig.navyConfig.setBlockSize(gBlockSize); // Default is 4096 - Device block size in bytes (minimum IO granularity)
    nvmConfig.navyConfig.setSimpleFile(gCacheFilePath,
                                gCacheSize,         /*fileSize*/
                                false           /*truncateFile*/);

    // Set admission policy
    nvmConfig.navyConfig.enableRandomAdmPolicy().setAdmProbability(gAdmissionProbability); // a float number that decides Acceptance probability. The value has to be in the range of [0, 1].

    // Engine Settings:

    // Block Cache: - caches objects that are larger than KBs in size 
    // Set the block cache - Large Object Cache - caches objects that are larger than KBs in size 
    nvmConfig.navyConfig.blockCache().setRegionSize(gRegionSize);

    // Set the Small Object Cache - caches objects that are 100s of bytes
    nvmConfig.navyConfig.bigHash()
      .setSizePctAndMaxItemSize(gBigHashSizePct, gBigHashSmallItemMaxSize) // bigHashSizePct (0-100), the % of the file that bigHash will use, bigHashSmallItemMaxSize must be smaller than bucket size which is 4096 by default
      .setBucketSize(gBigHashBucketSize)
      .setBucketBfSize(gBigHashBucketBfSize); // bloom filter size per bucket, default is 8


    //Make the cache actually a hybrid
    config.enableNvmCache(nvmConfig).validate();

    // Try to ease up on the SSD burning rate by adding the admission rate to the cache as a whole
    // This will accept to the cache only if there were 2 assurance in the time frame
    //auto policy = std::make_shared<cachelib::RejectFirstAP<facebook::cachelib::LruAllocator>>();
    //config.setNvmCacheAdmissionPolicy(std::move(policy));

    gCache_ = std::make_unique<Cache>(config);
    //gCache_ = std::make_unique<Cache>(lruConfig);
    
    defaultPool_ =  gCache_->addPool("default", gCache_->getCacheMemoryStats().ramCacheSize);
}

There are the environmental variables:

environment:
      - CACHE_RAM_SIZE=107374182 # 100mb
      - CACHE_NVME_SIZE=858993459200 # 1GB = 1 * 1024 * 1024 * 1024, we will use *800gb*
      - BLOCK_SIZE=4096 #  default 4096
      - NUMBER_OF_ITEMS_TO_CACHE=40000000 # 100 * 100 * 4000 [we plan to support 100 customers with 100 files each]
      - REGION_SIZE=16777216 # Not sure what is this - keep it as default: 16 * 1024 * 1024
      - CACHE_FILE=/nvmeCache/nvmeCache # The path to the file we use for NVME cache
      - MAX_CHAINED_SIZE=1048568 # The size of each chained element, currently we set it to (1024*1024*1)-8 (1MB -8 bytes) which effectively set the max size of cacheable item to ~255gb 
      - MAX_GRPC_MSG_RECIVED_SIZE=50000000 # 50mb
      - MAX_GRPC_MSG_SEND_SIZE=50000000 # 50mb
      - NVME_ADDMISSION_PROBABILITY=0.9 # This set what is the probability we will add the object to NVME cache
      - NVME_BIGHASH_SIZE_PCT=30 # (0-100), the % of the file that bigHash will use
      - NVME_BIGHASH_SMALL_ITEM_MAX_SIZE=4052 # bigHashSmallItemMaxSize must be smaller than bucket size which is 4096 by default
      - NVME_BIGHASH_BUCKET_SIZE=4096 #default 4096
      - NVME_BIGHASH_BLOOM_FILTER_PER_BUCKET_SIZE=8 # bloom filter size per bucket, default is 8

When I test it, I send many chanks of information (larger than 100mb) and expect that the eviction will be to the NVME and I can get all of them back.

This is my test code:

// Test multiple writes
for (int i = 33; i < 125; i ++)
{
    string key = new string((char)i, 32);
    string value = new string((char)i, 50000000 - 50);
    IntType res = await client.PutAsync(new StoreRequest { Key = key, Value = value });
    if (res.Res != 1)
    {
        bool aaa = false;
    }

    // Make sure we X3 hits
    await client.GetAsync(new Request { Key = key });
    await client.GetAsync(new Request { Key = key });
    await client.GetAsync(new Request { Key = key });
}

// Test multiple reads
for (int i = 33; i < 125; i++)
{
    string key = new string((char)i, 32);
    Response res = await client.GetAsync(new Request { Key = key });
    if (res.Value == null || res.Value.Length < 50000000 - 50)
    {
          //I expect to never be here - all the cache should be written to the NVME
        bool error = true;
    }
}

But when I test this, I get only the results of the keys 124,125 (i.e., 100mb is cached only).

What do I do rung? I appreciate any help here.

Jiayue Bao · Answer 1 · Mon May 29 2023 23:38:49 GMT+0800 (China Standard Time)

Hi @MeirHebrew,
you can try just set the file path and file size, and leave all the other settings as default. (This will only enable BlockCache):

nvmConfig.navyConfig.setSimpleFile(gCacheFilePath,
          gCacheSize, /*fileSize*/);

And for further debugging, there are a few things you can check:

check whether nvmCache is enabled: https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocator.h#L1202

gCache_->isNvmCacheEnabled()

check nvmCache StatsMap: https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocator.h#L1186

auto statsMap = gCache_->getNvmCacheStatsMap();
for (const auto& kv : statsMap.getCounts()) {
   cout << "key: " << kv.first << "value: " << kv.second << endl;
}

You may want to check the following keys:

bigHash size: "navy_bh_size"
bigHash items number: "navy_bh_items"
blockCache size: "navy_bc_size"
blockCache items number: "navy_bc_items"

MeirHebrew · Answer 2 · Tue May 30 2023 18:58:58 GMT+0800 (China Standard Time)

Hi @MeirHebrew, you can try just set the file path and file size, and leave all the other settings as default. (This will only enable BlockCache):
nvmConfig.navyConfig.setSimpleFile(gCacheFilePath,
          gCacheSize, /*fileSize*/);
And for further debugging, there are a few things you can check:

check whether nvmCache is enabled: https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocator.h#L1202
gCache_->isNvmCacheEnabled()
check nvmCache StatsMap: https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocator.h#L1186
auto statsMap = gCache_->getNvmCacheStatsMap();
for (const auto& kv : statsMap.getCounts()) {
   cout << "key: " << kv.first << "value: " << kv.second << endl;
}
You may want to check the following keys:

bigHash size: "navy_bh_size"

bigHash items number: "navy_bh_items"

blockCache size: "navy_bc_size"

blockCache items number: "navy_bc_items"

Hi @jiayuebao Thanks for the response, this I my output:

On starting the cache:

gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0

After adding 1 item:

[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0

After 1 item got revoked from RAM (I know this since I added a callback)

Is evicted: 1 0 Item key:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 1
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 1
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0

The only changes I see are navy_max_writer_pool_pending_jobs value: 1 and navy_max_writer_pool_pending_jobs value: 1, navy_bc_items is 0, same as navy_bh_items.
I tried different write rates to the cache, and it didn't help, I tried different numbers of writes, and this didn't help as well.

Jiayue Bao · Answer 3 · Thu Jun 01 2023 00:39:30 GMT+0800 (China Standard Time)

@MeirHebrew:
NvmCache uses job schedule to handle read or write requests. Every time there is an nvm put request, we will enqueue "insert" job to the writer pool. We have 1 for writer_pool_pending_job means there is indeed an insertion request sent. However, both navy_bc_items and navy_bh_items are 0 seems indicate the write job hasn't been processed.

To help further debugging, could you also try

printing out the rate stats? e.g.

for (const auto& kv : statsMap.getRates()) {
   cout << "key: " << kv.first << "value: " << kv.second << endl;
}

add loggings to check whether the job has been processed: https://github.com/facebook/CacheLib/blob/main/cachelib/navy/scheduler/ThreadPoolJobScheduler.cpp#L50

MeirHebrew · Answer 4 · Thu Jun 01 2023 15:05:45 GMT+0800 (China Standard Time)

@jiayuebao
I added prints for the rate stats in addition I added debug printing to the function ThreadPoolExecutor::ThreadPoolExecutor as follows:

ThreadPoolExecutor::ThreadPoolExecutor(uint32_t numThreads,
                                       folly::StringPiece name)
    : name_{name}, queues_(numThreads) {
  cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] Start function with parameters: numThreads: "<< numThreads << " name: " << name << endl;
  XDCHECK_GT(numThreads, 0u);
  cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After XDCHECK_GT(numThreads, 0u);" << endl;
  workers_.reserve(numThreads);
  cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After workers_.reserve(numThreads);" << endl;
  for (uint32_t i = 0; i < numThreads; i++) {
    cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {" << endl;
    queues_[i] = std::make_unique<JobQueue>();
    cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();" << endl;
    workers_.emplace_back(
        [&q = queues_[i],
         threadName = folly::sformat("navy_{}_{}", name.subpiece(0, 6), i)] {
          cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: " << threadName <<  endl;
          folly::setThreadName(threadName);
         cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);" << endl;
          q->process();
          cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: q->process();" << endl;
        });
   cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block" << endl;
  }
  cout << "[DBG][ThreadPoolExecutor::ThreadPoolExecutor] End of function" << endl;
}

Here is the new output blob when I start the server:

I0601 06:55:05.715304  7795 Factory.cpp:337] Cache file: /nvmeCache/nvmeCache size: 858993459200 truncate: 0
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] Start function with parameters: numThreads: 32 name: reader_pool
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After XDCHECK_GT(numThreads, 0u);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After workers_.reserve(numThreads);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_0
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: [DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
navy_reader_1
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_2
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_3
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: [DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
navy_reader_4
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_5
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_6
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_7
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: [DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();navy_reader_8

[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_9
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_10
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_11
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_12
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_13
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_14
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_15
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_16
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_17
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_18
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_19
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_20
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_21
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_22
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_23
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();

[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_24
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_25
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_26
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_27
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_28
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_29
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_reader_30
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] End of function
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] Start function with parameters: numThreads: 32 name: writer_pool
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After XDCHECK_GT(numThreads, 0u);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After workers_.reserve(numThreads);[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: 
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
navy_reader_31
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: [DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
navy_writer_0
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_1
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_2
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_3
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_4
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_5[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();

[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_6
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_7
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_8
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_9
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_10
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_11
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_12
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_13
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_14
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_15
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_16
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_17
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_18
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_19
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_20
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_21
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_22
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_23
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_24
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_25
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_26
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_27
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_28
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_29
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After for (uint32_t i = 0; i < numThreads; i++) {
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: queues_[i] = std::make_unique<JobQueue>();
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_30
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: workers_.emplace_back block
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] End of function
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: threadName = folly::sformat('navy_{}_{}', name.subpiece(0, 6), i)] { ;  Where threadName is: navy_writer_31
[DBG][ThreadPoolExecutor::ThreadPoolExecutor] After: folly::setThreadName(threadName);
I0601 06:55:05.762142  7795 RejectRandomAP.cpp:41] RejectRandomAP: probability 0.8999999761581421
I0601 06:55:05.762191  7795 NavySetup.cpp:239] metadataSize: 4294967296
I0601 06:55:05.762225  7795 NavySetup.cpp:241] Setting up engine pair 0
I0601 06:55:05.762235  7795 NavySetup.cpp:111] bighashStartingLimit: 4294967296 bigHashCacheOffset: 601295421440 bigHashCacheSize: 257698037760
I0601 06:55:05.762241  7795 NavySetup.cpp:255] blockCacheSize 597000454144
I0601 06:55:05.762268  7795 NavySetup.cpp:154] blockcache: starting offset: 4294967296, block cache size: 597000454144
I0601 06:55:05.762343  7795 LruPolicy.cpp:37] LRU policy: expected 35584 regions
I0601 06:55:05.922420  7795 BigHash.cpp:96] BigHash created: buckets: 62914560, bucket size: 4096, base offset: 601295421440
I0601 06:55:05.922488  7795 BigHash.cpp:105] Reset BigHash
I0601 06:55:05.963578  7795 RegionManager.cpp:50] 35584 regions, 16777216 bytes each
I0601 06:55:05.970287  7795 Allocator.cpp:40] Enable priority-based allocation for Allocator. Number of priorities: 1
I0601 06:55:05.970386  7795 BlockCache.cpp:145] Block cache created
I0601 06:55:05.970482  7795 Driver.cpp:71] Max concurrent inserts: 1000000
I0601 06:55:05.970492  7795 Driver.cpp:72] Max parcel memory: 268435456
I0601 06:55:05.970498  7795 Driver.cpp:195] Reset Navy
I0601 06:55:05.970527  7795 BigHash.cpp:105] Reset BigHash
I0601 06:55:06.004157  7795 BlockCache.cpp:684] Reset block cache
gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0
[statsMap.getRates()] key: navy_device_decryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_encryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_write_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_rebuildsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_lookupsvalue: 0
[statsMap.getRates()] key: navy_bh_io_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_writer_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bc_insert_hash_collisionsvalue: 0
[statsMap.getRates()] key: navy_device_read_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaimvalue: 0
[statsMap.getRates()] key: navy_bc_insertsvalue: 0
[statsMap.getRates()] key: navy_reader_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bh_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_writtenvalue: 0
[statsMap.getRates()] key: navy_removesvalue: 0
[statsMap.getRates()] key: navy_writer_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_false_positivesvalue: 0
[statsMap.getRates()] key: navy_lookupsvalue: 0
[statsMap.getRates()] key: navy_rejected_concurrent_insertsvalue: 0
[statsMap.getRates()] key: navy_rejected_bytesvalue: 0
[statsMap.getRates()] key: navy_insertsvalue: 0
[statsMap.getRates()] key: navy_bh_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_rejected_parcel_memoryvalue: 0
[statsMap.getRates()] key: navy_bh_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_failuresvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_acceptedvalue: 0
[statsMap.getRates()] key: navy_req_order_spooledvalue: 0
[statsMap.getRates()] key: navy_bc_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_rejectedvalue: 0
[statsMap.getRates()] key: navy_reader_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_accepted_bytesvalue: 0
[statsMap.getRates()] key: navy_io_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_removesvalue: 0
[statsMap.getRates()] key: navy_bh_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_eviction_lookup_missesvalue: 0
[statsMap.getRates()] key: navy_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_evictions_expiredvalue: 0
[statsMap.getRates()] key: navy_bh_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertionsvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_for_item_destructor_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_remove_attempt_collisionsvalue: 0
[statsMap.getRates()] key: navy_bc_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_timevalue: 0
[statsMap.getRates()] key: navy_bh_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_alloc_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_cleanup_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_removesvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_bytesvalue: 0
[statsMap.getRates()] key: navy_bh_insertsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_readvalue: 0
[statsMap.getRates()] key: navy_bc_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_region_reclaim_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_evictions_expiredvalue: 0
Server listening on 0.0.0.0:50051

Here is the new output blob after adding the first item:

[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0
[statsMap.getRates()] key: navy_device_decryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_encryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_write_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_rebuildsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_lookupsvalue: 1
[statsMap.getRates()] key: navy_bh_io_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_writer_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bc_insert_hash_collisionsvalue: 0
[statsMap.getRates()] key: navy_device_read_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaimvalue: 0
[statsMap.getRates()] key: navy_bc_insertsvalue: 0
[statsMap.getRates()] key: navy_reader_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bh_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_writtenvalue: 0
[statsMap.getRates()] key: navy_removesvalue: 0
[statsMap.getRates()] key: navy_writer_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_false_positivesvalue: 0
[statsMap.getRates()] key: navy_lookupsvalue: 1
[statsMap.getRates()] key: navy_rejected_concurrent_insertsvalue: 0
[statsMap.getRates()] key: navy_rejected_bytesvalue: 0
[statsMap.getRates()] key: navy_insertsvalue: 0
[statsMap.getRates()] key: navy_bh_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_rejected_parcel_memoryvalue: 0
[statsMap.getRates()] key: navy_bh_lookupsvalue: 1
[statsMap.getRates()] key: navy_bc_inmem_flush_failuresvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_acceptedvalue: 0
[statsMap.getRates()] key: navy_req_order_spooledvalue: 0
[statsMap.getRates()] key: navy_bc_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_rejectedvalue: 0
[statsMap.getRates()] key: navy_reader_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_accepted_bytesvalue: 0
[statsMap.getRates()] key: navy_io_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_removesvalue: 0
[statsMap.getRates()] key: navy_bh_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_eviction_lookup_missesvalue: 0
[statsMap.getRates()] key: navy_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_evictions_expiredvalue: 0
[statsMap.getRates()] key: navy_bh_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertionsvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookupsvalue: 1
[statsMap.getRates()] key: navy_bc_lookup_for_item_destructor_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_remove_attempt_collisionsvalue: 0
[statsMap.getRates()] key: navy_bc_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_timevalue: 0
[statsMap.getRates()] key: navy_bh_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_alloc_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_cleanup_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_removesvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_bytesvalue: 0
[statsMap.getRates()] key: navy_bh_insertsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_readvalue: 0
[statsMap.getRates()] key: navy_bc_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_region_reclaim_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_evictions_expiredvalue: 0

Here is the new output blob after adding the second item and the first eviction:

Is evicted: 1 0 Item key:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 1
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 1
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0
[statsMap.getRates()] key: navy_device_decryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_encryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_write_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_rebuildsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_lookupsvalue: 3
[statsMap.getRates()] key: navy_bh_io_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_writer_pool_jobs_donevalue: 1
[statsMap.getRates()] key: navy_bc_insert_hash_collisionsvalue: 0
[statsMap.getRates()] key: navy_device_read_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaimvalue: 0
[statsMap.getRates()] key: navy_bc_insertsvalue: 1
[statsMap.getRates()] key: navy_reader_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bh_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_writtenvalue: 0
[statsMap.getRates()] key: navy_removesvalue: 0
[statsMap.getRates()] key: navy_writer_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_false_positivesvalue: 0
[statsMap.getRates()] key: navy_lookupsvalue: 2
[statsMap.getRates()] key: navy_rejected_concurrent_insertsvalue: 0
[statsMap.getRates()] key: navy_rejected_bytesvalue: 0
[statsMap.getRates()] key: navy_insertsvalue: 1
[statsMap.getRates()] key: navy_bh_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_rejected_parcel_memoryvalue: 0
[statsMap.getRates()] key: navy_bh_lookupsvalue: 2
[statsMap.getRates()] key: navy_bc_inmem_flush_failuresvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_acceptedvalue: 1
[statsMap.getRates()] key: navy_req_order_spooledvalue: 0
[statsMap.getRates()] key: navy_bc_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_rejectedvalue: 0
[statsMap.getRates()] key: navy_reader_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_accepted_bytesvalue: 5.74044e+07
[statsMap.getRates()] key: navy_io_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_removesvalue: 0
[statsMap.getRates()] key: navy_bh_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_eviction_lookup_missesvalue: 0
[statsMap.getRates()] key: navy_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_evictions_expiredvalue: 0
[statsMap.getRates()] key: navy_bh_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertionsvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookupsvalue: 2
[statsMap.getRates()] key: navy_bc_lookup_for_item_destructor_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_remove_attempt_collisionsvalue: 0
[statsMap.getRates()] key: navy_bc_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_timevalue: 0
[statsMap.getRates()] key: navy_bh_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_alloc_errorsvalue: 1
[statsMap.getRates()] key: navy_bc_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_cleanup_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_removesvalue: 1
[statsMap.getRates()] key: navy_bc_reinsertion_bytesvalue: 0
[statsMap.getRates()] key: navy_bh_insertsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_readvalue: 0
[statsMap.getRates()] key: navy_bc_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_region_reclaim_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_evictions_expiredvalue: 0

Here is the new output blob after adding 2 more and evict 2 more:

Is evicted: 1 0 Item key:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 5.74044e+07
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 1
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 1
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 1
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0
[statsMap.getRates()] key: navy_device_decryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_encryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_write_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_rebuildsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_lookupsvalue: 5
[statsMap.getRates()] key: navy_bh_io_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_writer_pool_jobs_donevalue: 1
[statsMap.getRates()] key: navy_bc_insert_hash_collisionsvalue: 0
[statsMap.getRates()] key: navy_device_read_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaimvalue: 0
[statsMap.getRates()] key: navy_bc_insertsvalue: 2
[statsMap.getRates()] key: navy_reader_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bh_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_writtenvalue: 0
[statsMap.getRates()] key: navy_removesvalue: 0
[statsMap.getRates()] key: navy_writer_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_false_positivesvalue: 0
[statsMap.getRates()] key: navy_lookupsvalue: 3
[statsMap.getRates()] key: navy_rejected_concurrent_insertsvalue: 0
[statsMap.getRates()] key: navy_rejected_bytesvalue: 0
[statsMap.getRates()] key: navy_insertsvalue: 2
[statsMap.getRates()] key: navy_bh_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_rejected_parcel_memoryvalue: 0
[statsMap.getRates()] key: navy_bh_lookupsvalue: 3
[statsMap.getRates()] key: navy_bc_inmem_flush_failuresvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_acceptedvalue: 2
[statsMap.getRates()] key: navy_req_order_spooledvalue: 0
[statsMap.getRates()] key: navy_bc_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_rejectedvalue: 0
[statsMap.getRates()] key: navy_reader_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_accepted_bytesvalue: 1.14809e+08
[statsMap.getRates()] key: navy_io_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_removesvalue: 0
[statsMap.getRates()] key: navy_bh_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_eviction_lookup_missesvalue: 0
[statsMap.getRates()] key: navy_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_evictions_expiredvalue: 0
[statsMap.getRates()] key: navy_bh_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertionsvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookupsvalue: 3
[statsMap.getRates()] key: navy_bc_lookup_for_item_destructor_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_remove_attempt_collisionsvalue: 0
[statsMap.getRates()] key: navy_bc_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_timevalue: 0
[statsMap.getRates()] key: navy_bh_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_alloc_errorsvalue: 2
[statsMap.getRates()] key: navy_bc_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_cleanup_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_removesvalue: 2
[statsMap.getRates()] key: navy_bc_reinsertion_bytesvalue: 0
[statsMap.getRates()] key: navy_bh_insertsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_readvalue: 0
[statsMap.getRates()] key: navy_bc_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_region_reclaim_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_evictions_expiredvalue: 0
Is evicted: 1 0 Item key:cccccccccccccccccccccccccccccccc
[After adding item] gCache_->isNvmCacheEnabled(): 1
[statsMap.getCounts()] key: items_tracked_for_destructor value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99 value: 0
[statsMap.getCounts()] key: navy_parcel_memory value: 0
[statsMap.getCounts()] key: navy_bh_size value: 2.57698e+11
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p75 value: 0
[statsMap.getCounts()] key: navy_max_reader_pool_pending_jobs value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_max value: 0
[statsMap.getCounts()] key: navy_bc_external_fragmentation value: 0
[statsMap.getCounts()] key: navy_writer_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p50 value: 0
[statsMap.getCounts()] key: navy_writer_pool_max_queue_len value: 1
[statsMap.getCounts()] key: navy_device_write_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p90 value: 0
[statsMap.getCounts()] key: navy_reader_pool_jobs_high_reschedule value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999 value: 0
[statsMap.getCounts()] key: navy_bc_size value: 5.97e+11
[statsMap.getCounts()] key: navy_device_write_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p75 value: 0
[statsMap.getCounts()] key: navy_bc_inmem_active value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_min value: 0
[statsMap.getCounts()] key: navy_bc_hole_bytes value: 0
[statsMap.getCounts()] key: navy_bc_inmem_waiting_flush value: 0
[statsMap.getCounts()] key: navy_req_order_curr_spool_size value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999 value: 0
[statsMap.getCounts()] key: navy_bh_bf_false_positive_pct value: 0
[statsMap.getCounts()] key: navy_bc_num_regions value: 35584
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p90 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p25 value: 0
[statsMap.getCounts()] key: navy_reader_pool_max_queue_len value: 0
[statsMap.getCounts()] key: navy_concurrent_inserts value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_max value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p50 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_total_usable_size value: 8.54698e+11
[statsMap.getCounts()] key: navy_max_writer_pool_pending_jobs value: 1
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_max value: 0
[statsMap.getCounts()] key: navy_bc_num_clean_regions value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p10 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p75 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_min value: 0
[statsMap.getCounts()] key: navy_bc_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p25 value: 0
[statsMap.getCounts()] key: navy_bc_hole_count value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_min value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p99 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p5 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p25 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_lru_region_hits_estimate_max value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_avg value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_avg value: 0
[statsMap.getCounts()] key: navy_bc_items value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_min value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p50 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p90 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p9999 value: 0
[statsMap.getCounts()] key: navy_bc_item_removed_with_no_access value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p999999 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_avg value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p99999 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p999999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p5 value: 0
[statsMap.getCounts()] key: navy_bh_items value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_insertion_p99 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_bh_used_size_bytes value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p25 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p25 value: 0
[statsMap.getCounts()] key: navy_bh_expired_loop_x100_p99 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p5 value: 0
[statsMap.getCounts()] key: navy_bc_item_hits_p95 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p10 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_min value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p99 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p50 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_p9999 value: 0
[statsMap.getCounts()] key: navy_device_read_latency_us_max value: 0
[statsMap.getCounts()] key: navy_bc_lru_secs_since_access_p75 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p90 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p95 value: 0
[statsMap.getCounts()] key: navy_device_write_latency_us_p99 value: 0
[statsMap.getRates()] key: navy_device_decryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_encryption_errorsvalue: 0
[statsMap.getRates()] key: navy_device_write_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_rebuildsvalue: 0
[statsMap.getRates()] key: navy_bh_bf_lookupsvalue: 7
[statsMap.getRates()] key: navy_bh_io_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_writer_pool_jobs_donevalue: 3
[statsMap.getRates()] key: navy_bc_insert_hash_collisionsvalue: 0
[statsMap.getRates()] key: navy_device_read_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaimvalue: 0
[statsMap.getRates()] key: navy_bc_insertsvalue: 3
[statsMap.getRates()] key: navy_reader_pool_jobs_donevalue: 0
[statsMap.getRates()] key: navy_bh_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_writtenvalue: 0
[statsMap.getRates()] key: navy_removesvalue: 0
[statsMap.getRates()] key: navy_writer_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_bc_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_flush_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_false_positivesvalue: 0
[statsMap.getRates()] key: navy_lookupsvalue: 4
[statsMap.getRates()] key: navy_rejected_concurrent_insertsvalue: 0
[statsMap.getRates()] key: navy_rejected_bytesvalue: 0
[statsMap.getRates()] key: navy_insertsvalue: 3
[statsMap.getRates()] key: navy_bh_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_lookup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_rejected_parcel_memoryvalue: 0
[statsMap.getRates()] key: navy_bh_lookupsvalue: 4
[statsMap.getRates()] key: navy_bc_inmem_flush_failuresvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_entry_header_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_acceptedvalue: 3
[statsMap.getRates()] key: navy_req_order_spooledvalue: 0
[statsMap.getRates()] key: navy_bc_logical_writtenvalue: 0
[statsMap.getRates()] key: navy_rejectedvalue: 0
[statsMap.getRates()] key: navy_reader_pool_reschedulesvalue: 0
[statsMap.getRates()] key: navy_accepted_bytesvalue: 1.72213e+08
[statsMap.getRates()] key: navy_io_errorsvalue: 0
[statsMap.getRates()] key: navy_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_removesvalue: 0
[statsMap.getRates()] key: navy_bh_succ_removesvalue: 0
[statsMap.getRates()] key: navy_bc_eviction_lookup_missesvalue: 0
[statsMap.getRates()] key: navy_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_evictions_expiredvalue: 0
[statsMap.getRates()] key: navy_bh_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertionsvalue: 0
[statsMap.getRates()] key: navy_bc_reinsertion_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_lookupsvalue: 4
[statsMap.getRates()] key: navy_bc_lookup_for_item_destructor_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_remove_attempt_collisionsvalue: 0
[statsMap.getRates()] key: navy_bc_evictionsvalue: 0
[statsMap.getRates()] key: navy_bc_reclaim_timevalue: 0
[statsMap.getRates()] key: navy_bh_succ_insertsvalue: 0
[statsMap.getRates()] key: navy_bc_alloc_errorsvalue: 3
[statsMap.getRates()] key: navy_bc_physical_writtenvalue: 0
[statsMap.getRates()] key: navy_bc_cleanup_value_checksum_errorsvalue: 0
[statsMap.getRates()] key: navy_bc_inmem_cleanup_retriesvalue: 0
[statsMap.getRates()] key: navy_bh_removesvalue: 3
[statsMap.getRates()] key: navy_bc_reinsertion_bytesvalue: 0
[statsMap.getRates()] key: navy_bh_insertsvalue: 0
[statsMap.getRates()] key: navy_device_bytes_readvalue: 0
[statsMap.getRates()] key: navy_bc_succ_lookupsvalue: 0
[statsMap.getRates()] key: navy_bc_region_reclaim_errorsvalue: 0
[statsMap.getRates()] key: navy_bh_evictions_expiredvalue: 0

The changes I see (in rates) between the first 2:

Key						1st	2nd		
navy_bh_bf_lookupsvalue: 		0	1	
navy_bh_bf_lookupsvalue: 		1	3
navy_writer_pool_jobs_donevalue: 	0	1
navy_bc_insertsvalue: 			0	1
navy_lookupsvalue: 			1	2
navy_insertsvalue: 			0	1
navy_bh_lookupsvalue: 			1	2
navy_acceptedvalue: 			0	1
navy_accepted_bytesvalue: 		0	5.74044e+07
navy_bc_lookupsvalue: 			1	2
navy_bc_alloc_errorsvalue: 		0	1
navy_bh_removesvalue: 			0	1

The same fields changes in the next key insertion as well

Jiayue Bao · Answer 5 · Fri Jun 02 2023 05:57:00 GMT+0800 (China Standard Time)

@MeirHebrew
"navy_bc_alloc_errors" might be the reason, it indicates that allocation failed when you tried to insert an item to BlockCache. So the next step would be figuring out why the allocation would fail.

From https://github.com/facebook/CacheLib/blob/main/cachelib/navy/block_cache/BlockCache.cpp#L167, there are two reasons that can cause allocation failure:

size > kMaxItemSize: https://github.com/facebook/CacheLib/blob/main/cachelib/navy/block_cache/BlockCache.cpp#L170
allocator_->allocate returns error: https://github.com/facebook/CacheLib/blob/main/cachelib/navy/block_cache/BlockCache.cpp#L180

I think you can start from here to further investigate the issue :).

MeirHebrew · Answer 6 · Fri Jun 02 2023 18:58:33 GMT+0800 (China Standard Time)

@jiayuebao Thanks! you helped me figure this out, thank you!