Slow page loading/construction with 'large' database
kormyen opened this issue · comments
Load divs on scroll?
Delayed div construction?
I believe the slow-ness/lag is due to constructing dom elements rather than dealing with the text data itself (so far...)
Note to self: look at how loading spinner works in https://github.com/kodedninja/orkl
I'm having this issue. It takes a couple of minutes for the entire thing to load. It's mostly because of lots of DOM reflows and string manipulations. Instead of writing directly to the body, you should try appending everything to a document fragment then append that to the body when everything is ready.
I tried to implement it last night but I messed something up lol. I need to further study your code :)
Wow 'couple of minutes'? That is pretty bad. I'm seeing 6 second loads with 555 entries and thought that was bad enough.
I need to look into this properly as I do not understand DOM related stuff. A quick google says
"Reflow occurs when you:
...
resize the window
scroll"
but I once the page (all entry divs constructed) I can freely scroll, resize etc without lag (reflow delay?)
I thought I was doing it right by calculating everything into a string var then applying that all as one update to this.grid.innerHTML... but instead I had a new write to innerHTML for every entry. Just did that now and there is still a delay but it loads in less than a second with 555 entries on my laptop.
I'll push this soon. Completely open to any other improvements or links/resources you recommend.
Pushed
Yeah it takes a couple of minutes (I have a really slow laptop hehe). What if you used Promises? Once a grid item is ready, append it right away to the grid while the rest are still being processed. Similar to what I did for クモ. Everything cascades.
fetch(`${URL}/topstories.json`)
.then(response => response.json())
.then(data => {
for (let i = count; i >= 0; i--) {
fetch(`${URL}/item/${data[count - i]}.json`)
.then(response => response.json())
.then(({title, url}) => {
// append <a> element to the feed
});
}
});
Edit: Looks like most of my latency issues come from hosting the repo in a python SimpleHTTPServer (simply opening index.html gives me some error about CORS and stuff). Your live version loads in about 6 seconds :D
I'll have a look at promises, but wouldn't the append-per-entry-as-ready get me back to the state before the fix above (many edits to innerHTML causing lag?).
Ah, the CORS error (in Chrome?) will be due to the way I am loading the database. It works locally in Firefox however. I did that as a test of just having the database clean text data, rather than including the var DATABASE = '' js. Maybe the Chrome issues make it not worth the hassle.
There might still be some latency but there would also be perceived improvements in speed since the UI will be rendered right away while the database is still being processed. Having above-the-fold content rendered as soon as possible is good enough to satisfy most users
Ok. Makes sense.
I'll give it a go.
I suppose Memex will need pagination or 'load more' (button or automatic as scrolling) eventually.
I've changed to standard js db, now it'll run fine locally in Chrome (no CORS error).
Haven't got the promise stuff done yet but memex now loads the html/css first (with a preloader anim!) THEN loads/runs the js. Lots better than blank white page for 6 seconds from a couple of days ago.
Been trying to get it so that entries are displayed as they are computed but the page stays unresponsive (due to reflows I think) until (DOM?) additions/modifications are complete defeating the purpose.
On top of this it takes much longer than just computing all new html and adding/replacing it once.
Open to advice but my current understanding it that the current master branch is the setup it can be?
- instantly show bg and animated loading anim while db parses and html is computed...
- once all resulting html computed: loading anim removed, html result added to page
Current failed mess of testing here (see main.js start function) 1bb09fe
"Having above-the-fold content rendered as soon as possible is good enough to satisfy most users"
Maybe rendering into two steps could be an option, but I don't know...
- instantly show bg and animated loading anim while db parses and html is computed for first 10 articles...
- once first 10 articles' html computed: loading anim removed, html result added to page, rest of articles computed...
- rest of articles' html added to page
Continuing with promise setup https://github.com/kormyen/memex/commits/promise
Ended up removing promises as they didn't change anything other than code layout (no change to speed or responsiveness).
In the process though I made a simple benchmark-er (seer.js), refactored everything and learned about promises.
The bottleneck now is actually the masonry layout call (20 to 90% of the time loading is spent there, longer for more articles). I could try to write a minimal/faster replacement but not a high priority for me atm. Similarly it would be nice to make my own type icons as font-awesome is comparatively large and takes a moment to load.
Calling this issue complete.
- load times are faster (previously 500~ articles in 6~ seconds, now 1~ second)
- initial load displays background and preload anim immediately (instead of white page while loading)
- for queries that are not 'quick' preload anim displays as immediate response (instead of no change) while loading