Get JavaScript working (table.js, plugins and more)

[Simon Willison]

Relates to plugins challenge:

• #5

The table.js script used by the table page doesn't load at the moment, which means no cog icons on the columns:

[Simon Willison]

The SQL editor JavaScript on this page doesn't load either (well, none of the JavaScript loads): https://simonw.github.io/datasette-lite/#/fixtures?sql=select+sqlite_version%28%29

[Simon Willison]

A few ideas in https://stackoverflow.com/questions/13390588/script-tag-create-with-innerhtml-of-a-div-doesnt-work

I'm going to try scanning the inserted HTML for <script src> elements, extracting the src=, fetching that using a different message to the worker and executing it when it returns.

[Simon Willison]

Potential timing bug here - what if I fire off a message asking for a script, then the user navigates to another page, then I execute the JavaScript that they asked for on that new page?

I can avoid that by attaching some kind of ID attribute to the script element and checking for it in the DOM before running eval().

[Simon Willison]

There's another way I could approach this: the code that runs in the web worker could parse the HTML before sending it back to the client and - if it finds any script elements - could fetch that JavaScript and send it in a script: key.

The index.html page could then execute that JavaScript after inserting the .innerHTML.

This is a kind of filthy hack and I like it. Let's see if it works!

[Simon Willison]

There's another way I could approach this: the code that runs in the web worker could parse the HTML before sending it back to the client and - if it finds any script elements - could fetch that JavaScript and send it in a script: key.

Sadly "Error: DOMParser is not defined" - DOMParser isn't available inside web workers.

[Simon Willison]

I think I need to scan through each <script> and if it has contents eval() that, but if it has a src= attribute fetch that and then eval() it.

A couple of things that worry me:

• Some scripts may attempt to load more scripts by URL, which may break
• Scripts that set global variables may end up polluting each other

Maybe I should render these things in an iframe purely to give them a fresh JavaScript context on each page load?

[Simon Willison]

Here's how to run an HTML parser against the content returned by the Datasette server in the web worker:

   } else if (/^text\/html/.exec(event.data.contentType)) {
+    // Check for any script tags
+    let parser = new DOMParser();
+    let dom = parser.parseFromString(event.data.text, 'text/html');
+    console.log(dom);
     html = event.data.text;

[Simon Willison]

Tip from @Jonty: https://twitter.com/jonty/status/1521947838300762112

Did you consider running pydiode in the webworker, but keeping the serviceworker just for request intercept and passing off to the webworker? It would be a lot cleaner.

I didn't know service workers could talk to web workers - maybe this could help me solve the asset loading challenge?

[Simon Willison]

Another idea I just had: register a service worker for /-/static/ and /-/static-plugins/, then write some code in the web worker which, on startup, loops through ALL of the static files that have been provided by plugins and sends copies of those files to the service worker along with details of their paths.

That way the service worker doesn't have to run Pyodide and doesn't need to ask any questions itself - it just gets primed with the content it will need to serve when Datasette first starts running.

[Simon Willison]

I should definitely explore the idea of running the injected innerHTML content in an <iframe> such that every time a fresh page loads it gets a new, unpolluted JavaScript window object for plugin scripts to operate on.

[Simon Willison]

Relevant: the jQuery.parseHTML() https://api.jquery.com/jquery.parsehtml/ method has a keepScripts option.

Code for that is https://github.com/jquery/jquery/blob/main/src/core/parseHTML.js but I don't really understand what it's doing yet.

This bit is interesting: https://github.com/jquery/jquery/blob/2525cffc42934c0d5c7aa085bc45dd6a8282e840/src/core/parseHTML.js#L24-L33

		// Stop scripts or inline event handlers from being executed immediately
		// by using document.implementation
		context = document.implementation.createHTMLDocument( "" );

		// Set the base href for the created document
		// so any parsed elements with URLs
		// are based on the document's URL (gh-2965)
		base = context.createElement( "base" );
		base.href = document.location.href;
		context.head.appendChild( base );

Here's the issue that references:

• jquery/jquery#2965

[David Humphrey]

I saw your tweet/blog post, and loved what you're doing. This kind of thing is my favourite. Forgive me if you already know about all this, but in case it's helpful, I've done this in the past a few ways:

1. Use JS as strings, and convert to Blob and URL Objects I can attach to <script>s:

const js = new Blob(
  ['alert("hello world")'],
  { type: 'application/json' }
);

const url = URL.createObjectURL(js);

const script = document.createElement('script');
script.src = url;

document.body.appendChild(script);

2. Serve content out of the service worker at real URLs. The service worker can synthesize JS network responses from any source, and your page will happily consume them exactly the same as from a server. For me, I was putting a filesystem in the browser vs. a database, but same idea. I made a web server I could run in a service worker, see https://github.com/humphd/nohost (old, unmaintained code, but might give ideas). Using this I was able to run scripts (or any web content) from within a filesystem managed by a Linux VM running in the browser, mounting that same filesystem, see https://humphd.github.io/browser-shell/

You could probably put a service worker in front of your database, and pull the web assets from there, which would be fun (web site as db).

[Simon Willison]

This is great, thanks! Really useful example code - I'm leaning service worker at the moment but that blob trick looks like a great backup for if I can't get SWs to work.

[Simon Willison]

Now that I've added ?install=package-name a number of plugins work... but not the ones that need their own JavaScript or CSS.

[Cameron Yick]

I've been running a fork for the past few months that gets JS (and other static assets served by datasette, like CSS or SQL query results) working in datasette-lite. Initial support was added here: hydrosquall/datasette-nteract-data-explorer#26

So far, I've tested it successfully with

• The SQL editor / syntax highlighting code
• https://github.com/hydrosquall/datasette-nteract-data-explorer
• https://datasette.io/plugins/datasette-dashboards
• table.js (sort of - the cog menus are OK, but the facet buttons are not). Fixing this systemically may be easier after we offer a datasette plugin API ( simonw/datasette#2052 ) since we can handle redirection at the layer of the datasette client rather than having to intercept every click event on the page.