Live-Reload imported notebooks

[Alex Garcia]

Say you have top.ojs:

import {a} from "./a.ojs"
import {b} from "./b.ojs"

content = md`a=${a}, b=${b}`

a.ojs:

a = 100

b.ojs:

b = 200

If you're running dataflow run top.ojs, then the rendered output will show a=100, b=200 as expected. If top.ojs gets updated, then those changes will be reflected as well.

But, if a.ojs or b.ojs gets updated, then the changes do not immediately get reflected. You would have to refresh the entire page in order to fetch the latest version of the compiled a.ojs or b.ojs file.

This is currently the same case for editor and imports at observablehq.com, but since dataflow works with the filesystem, it should be easier to implement this.

So, when running dataflow run top.ojs, and a.ojs is edited to a = 400, then the live rendered notebook should reload the import and the new 400 value should appear.

[Alex Garcia]

Implementation notes:

On every new top.ojs change, the dataflow dev server will have to parse the code and find every import for local files, then watch those files with chokidar. When an imported file is update, then some websocket message should be pushed to the client, where the client then updates the specific imported/injected cells.

Problems:

1. How does the client update imports when given a message a.ojs has been updated?
2. Sub-imports would be hard. So if a.ojs imports from sub_a.ojs, and if sub_a.ojs gets updated, would we want to update everyything? my guy says yes, but that means we'd have to find every imported .ojs file downstream and watch all of them. Which probably wont be too big of a perf issue, since its not like you're importing 100+ notebooks at a time.

[Oleksandr Nikitin]

My two cents:

• it would be awesome to only update the cells where the text content was actually changed (one of the features Observable does not provide and I immediately started to use is indenting + collapsing parts of file, but currently this causes recompile and resets the state of things like UI inputs)
• yes, it makes sense to update everything changed, thankfully we have a clean dataflow DAG and not "every-other-edge-case-webpack-has-to-handle" :)
• typescript and react live-reload tooling happily watches and incrementally recompiles projects for thousands of files for me, so it's definitely doable
• oh, and i found it really really awesome dataflow updates as-you-type and you don't need to press Enter like Observable requires :)

[Alex Garcia]

it would be awesome to only update the cells where the text content was actually changed (one of the features Observable does not provide and I immediately started to use is indenting + collapsing parts of file, but currently this causes recompile and resets the state of things like UI inputs)

I'd love to hear more about this! Currently, Dataflow should only update cells that have had their contents changed, so if you changed a notebook file like this:

a = 1
-b = 2
+b = 7
c = a + b

x = 3
y = 4
z = x + y

Dataflow will detect that b=2 is outdated and delete that definition, and will only run b=7, causing only b and c to update (meaning a, x, y, and z stay the same). But if you're noticing that you only changed 1 cell's definition but others are updating erroneously, that sounds like a bug!

also, I didn't even think about nested imports! So if your notebook tree looks like this:

top.ojs
  \_ suba1.ojs
    \_ subb1.ojs
    \_ subb2.ojs
  \_ suba2.ojs
  \_ suba3.ojs

When running dataflow run top.ojs, then changing suba1.ojs should cause an update. Changing subb1.ojs should also cause an update, since that changes suba1.ojs, which I didn't consider before... Should definitely be do-able, many bundlers do this already, but definitely adds a layer of complexity to this.

[Oleksandr Nikitin]

I'd love to hear more about this!
So the file looks like

c = a
d = b

a = 1
b = 2

Then I add a "heading" (a comment) and indent lines below so the file looks like this:

c = a
d = b

// various utilities
  a = 1
  b = 2

And after I indent, the cells a and b (and accordingly c and d too) get recalculated, though obviously their values end up the same as before, only the leading whitespace has changed

[Oleksandr Nikitin]

Re: nested imports: yeah :) another potentially tricky case to consider in Observable's javascript (and the one I really wish was available in regular ES modules) to consider is import ... with ... where you can override some cells of the notebook you're importing (and I assume in this case different imports of the same notebook should create different cell instances, parameterized by whatever was in the corresponding with clauses)