Support serializing internal state

[Chase Wilson]

I'm working on a user-sided application with the goal of fast response times, and I've really been wanting a way to cache the internal state of dataflows so that it can be quickly recreated across restarts, enabling as fast of a restart time as possible while also skipping work that was already done in previous lifetimes of the program

[Ryan Munro]

@Kixiron have you discovered any way to make this possible? I'm also interested—I would really love a DB that's like SQLite, but differential 😻

I wandered around the code base a bit, I'm not sure if it's possible without patching—or wrapping all the objects to log state because the subgraph fields are private. But these are the areas of interest that I saw from digging around:

timely-dataflow/timely/src/worker.rs

Line 642 in 3671a3b

 let mut operator = subscope.into_inner().build(self); 

timely-dataflow/timely/src/progress/subgraph.rs

Lines 39 to 70 in 79ff074

 pub struct SubgraphBuilder<TOuter, TInner> 
 where 
     TOuter: Timestamp, 
     TInner: Timestamp, 
 { 
     /// The name of this subgraph. 
     pub name: String, 
  
     /// A sequence of integers uniquely identifying the subgraph. 
     pub path: Vec<usize>, 
  
     /// The index assigned to the subgraph by its parent. 
     index: usize, 
  
     // handles to the children of the scope. index i corresponds to entry i-1, unless things change. 
     children: Vec<PerOperatorState<TInner>>, 
     child_count: usize, 
  
     edge_stash: Vec<(Source, Target)>, 
  
     // shared state written to by the datapath, counting records entering this subgraph instance. 
     input_messages: Vec<Rc<RefCell<ChangeBatch<TInner>>>>, 
  
     // expressed capabilities, used to filter changes against. 
     output_capabilities: Vec<MutableAntichain<TOuter>>, 
  
     /// Logging handle 
     logging: Option<Logger>, 
  
     /// Progress logging handle 
     progress_logging: Option<ProgressLogger>, 
 } 

timely-dataflow/timely/src/dataflow/scopes/child.rs

Line 31 in 3671a3b

 pub subgraph: &'a RefCell<SubgraphBuilder<G::Timestamp, T>>, 

[Ryan Munro]

timely-dataflow/timely/src/worker.rs

Line 214 in 3671a3b

 paths: Rc<RefCell<HashMap<usize, Vec<usize>>>>, 

Worker.paths is also looking very interesting!

[Chase Wilson]

Unfortunately not, my hopes are mostly in disk backed differential arrangements but I don't think there's much progress towards that

[Ryan Munro]

disk backed differential arrangements

Sameee, I would love that please — even just applying simple maps would be fine for me right now as well — not that the incremental is hard to write, but I would like to not, if I don't have to. Have you been exploring what it would take for what you're imagining?

I'm wondering what would happen if I just started applying Serialize & Deserialize to things until something interesting happens 🤣

[Chase Wilson]

By and large it's a significantly more complex problem than just adding Serialize to things, dataflow construction isn't the expensive part of reviving a dataflow, the expense lies in rebuilding indices over massive amounts of data data