Reduce memory pressure of the caching to the minimum

[Marek Prochera]

I'm trying to reduce memory pressure of the caching to the minimum.

I manage to reduce it a lot using the configuration below, but I still see leftovers that have symptoms of the leak.

At the first glance it likes that it comes from the LevelZeroMapCache but I'm not sure - and I cannot find eany reference in the documenetaion.

val cacheKeyValueIdsOverride = false

val memoryCacheOverride = MemoryCache.off

  val mmapDisabled = MMAP.Off(
    ForceSave.Off
  )

  val segmentConfigOverride = DefaultConfigs
    .segmentConfig()
    .copyWithMmap(
      mmapDisabled
    )
    .copyWithCacheSegmentBlocksOnCreate(false)
    .copyWithFileOpenIOStrategy(IOStrategy.AsyncIO(cacheOnAccess = false))
    .copyWithBlockIOStrategy(
      blockIOStrategy = (_) => IOStrategy.AsyncIO(cacheOnAccess = false)
    )

  val fileCacheOverride = FileCache.On(
    0,
    ActorConfig.TimeLoop(
      name = s"${this.getClass.getName} - FileCache TimeLoop Actor",
      delay = 1.seconds,
      ec = DefaultExecutionContext.sweeperEC
    )
  )


      persistent.Map[String, Int, Nothing, Glass](
        dir = File.newTemporaryDirectory("suggestions_topn_base_").deleteOnExit().path,
        mmapMaps = mmapDisabled,
        cacheKeyValueIds = cacheKeyValueIdsOverride,
        segmentConfig = segmentConfigOverride,
        fileCache = fileCacheOverride,
        memoryCache = memoryCacheOverride
      )

[Simer]

Yep detailed documentation on cache configurations is missing. We should really document each cache config with performance benchmarks when turned off vs on.

Everything is configurable, so you should be able to configure caching from being very high to almost no caching.

For now, similar to how you've disabled caching with segmentConfigOverride in your code, you can try the same for sortedIndex, hashIndex, binarySearchIndex, bloomFilter and valuesConfig (find them in DefaultConfigs)

Set the following for all the above blocks.

case action: IOAction.DecompressAction => IOStrategy.SynchronisedIO(cacheOnAccess = false)

Just a note, in your settings above you've disabled caching opened files with this setting.

.copyWithFileOpenIOStrategy(IOStrategy.AsyncIO(cacheOnAccess = false))

So every thread that tries to access a file would open a new java.nio.FileChannel. Unless you have an extreme use-case I would set this to cacheOnAccess = true so there are less concurrently opened FileChannels for a file.