cloudflare / daphne

Implementation of DAP

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

taskprov: Managing KV storage limits

cjpatton opened this issue · comments

We currently store each task config we opt-in to in KV. This incurs a non-negliglbe storage cost. Looking at our current data, the KV key is 77 bytes and the value is 623 bytes. That's 700 bytes per task, which means we will hit our KV storage limit (1GB) after about 1.5million tasks. A few potential solutions to this:

  1. Ideally we avoid storing task configs in KV at all and instead re-generate the task config as needed. This was brought up when we were doing the initial implementation, but we decided there was a blocker that prevented this.
  2. Establish a process for pruning expired tasks.
  3. Establish a process for pruning least recently used tasks.