hodcroftlab / covariants

Real-time updates and information about key SARS-CoV-2 variants, plus the scripts that generate this information.

Home Page:https://covariants.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[superissue] Improve data handling

ivan-aksamentov opened this issue · comments

This superissue tracks issues and PRs related to the improvement of data handling in CoVariants:

Goals:

  • Improve user experience by making web UI more responsive to user actions. Currently, after navigation to pages with plots, there is a few seconds where nothing happens.
  • Speedup rendering of plots. If data is split into pieces, one piece for every plot, we can make some plots rendering earlier, providing quick feedback to the user, while other plots finish rendering later.
  • Reduce amount of downloaded data. Web app can download only the required chunks.
  • Parallelize downloads. Array items can be downloaded concurrently, so that earlier chunks fully arrive and are ready to be rendered, without waiting the whole data array.
  • Reduce web app bundle size. Currently all data is bundled into JS files and they are fully downloaded on first render or on page navigation. The rendering is blocked until the whole thing arrives.
  • Cache app code and data independently. Data is changing more often than the app's code. If they are decoupled, the unmodified code can stay in user's browser's cache longer, saving bandwidth and making things faster.
  • Cache data chunks independently. If some of the elements of the data collection are unchanged during release, then there is no need to redownload it.
  • Manage data chunks separately within the app state. Each element of the data collection can be kept in a separate Recoil atom, and then this Recoil atom can be used only in the component that needs it (alternatively it can be a separate react-query query). This may allow for concurrent rendering of the components that depend only on individual data chunks.
  • Reduce size of the git repository
  • Reduce load and failure rates on CI
  • Improve developer experience
  • Improve or at least preserve scientist experience
  • Preserve correctness and feature-completeness