More Ideas for Treemap Improvements by @Lithopsian

[Stefan Hundhammer]

Moved from #236 to this new issue to keep user documentation and developer discussions separate.

@Lithopsian writes:

Do you want to take this further? I've improved the algorithm to be faster as well as prettier.

Previously (minTileSize=3):

New algorithm (also minTileSize=3):

Instead of stopping dead at the first small tile, the algorithm carries on, but quietly drops many tiles while creating some at the minimum size to fill the available space.

There are many other performance improvements:

• streamlining the tile and cushion surface constructors
• optimising some of the loops in the tile layout functions
• factoring out the coefficient calculations some more
• precalculations of cushion heights and some pixmap values
• mime categorizer changes to improve the pattern precedence rules (eg. moc*_.cpp takes precedence over *.cpp) and still be a little faster. Patterns with no wildcards (eg. Makefile) now have the highest precedence.

Dropping tiles "invisibly" makes an issue with zooming more obvious: when the selected item is a tile that isn't in the treemap, it isn't possible to zoom. With the new treemap algorithm, a tile that is present at one zoom level may not be at the next and zooming will suddenly stop working. This can be fixed by calculating the zoom actions directly from the DirTree instead of from the treemap.

A Another oddity I came across is the treemap in the config dialog. I didn't realise for ages that it is a real live treemap with a random DirTree. It allows tile selection and also zooming. Unfortunately there is no selection model, so the selections tend to keep disappearing. I could fix up a workaround, but I'm not sure if this is something you really wanted to be in there, or if it just happened. The zooming in particular could be a bit confusing. I have set it up so the treemap just gets re-painted when a colour changes, but not rebuilt. The main treemap also gets re-rendered when category changes are committed.

Lastly, I put the treemap build into a separate thread. That doesn't make it much faster, but it avoids blocking the UI and allows for things like smooth resizing and scaling. I also put the cushion rendering into background threads while the treemap is being built, which does make it faster although not as much as I'd hoped; there is too much overhead in either spawning thousands of threads or sending signals to one and it is heavily bottlenecked in the mime categorizer. I'm still trying to improve it. It seems stable here, but I imagine there might still be uncommon race conditions. I had to mutex the mime categorizer singleton and pay very close attention to being able to cleanly cancel a building treemap since it no longer blocks other actions. Attempts to break the actual tile construction into pieces have all failed miserably; I suspect that QGraphicsRectItem itself is the problem because the tiles themselves are largely independent. I thought the FileInfo iterators might be a problem, but they seemed to be OK with it.

[Stefan Hundhammer]

This sounds very interesting. Some quick answers right away:

• I thought that it was pointless to carry on with the treemap layout algorithm once the first tiles smaller than MinTileSize started appearing: IIRC the algorithm iterates over a list of FileInfo items that are sorted by size. Are really larger tiles appearing again in that same hierarchy level? Why?

  I would very much welcome less empty grey space in the treemap, though, if that does not become a performance killer.

• Yes, the config dialog has a real live treemap. It's crudely made, it uses random numbers for random sizes of FileInfo items that become treemap tiles, and the number of tiles on each hierarchy level is also random. But it doesn't only look real, it is real. You can even click and select tiles, and you can zoom in and out with the mouse wheel. It's a real standalone TreemapView with its own randomly generated DirTree full of FileInfo items. It does not have a SelectionModel or anything like that; why would it? It's just to show what it would look like instantly if you change the colors. That part is already overengineered like hell; no need to add even more bells and whistles.

• Introducing threads for building the treemap? Yikes. No way. QDirStat was never designed as a multi-threaded application; there is nothing in it to ensure thread safety. Threads would mean introducing mutexes everywhere for safe concurrent access. Not doing that would be an accident waiting to happen. Or, more likely, a ton of accidents.

  And for building the treemap, of all things? Why? That is so quick that it happens faster than I can move my mouse scroll wheel. That's a solution (which isn't even a solution because of thread safety and mutexes) in search of a problem. Sorry, no.

OK, that's the quick answers for now. There is more to come.

[Stefan Hundhammer]

Upcoming Release and Stabilization

I mentioned this before, but I'd like to point it out again: I plan to release a new official stable version QDirStat-1.9 in the very near future. So my primary goal right now is to stabilize the code, do some more small bugfixes (like the initial selection and the selection after a cleanup action which typically removes the currently selected item), user documentation like that #236, change log. I already updated most of the screenshots in the screenshots/ subdirectory, and there will be release notes to write.

I am acting main developer, tester, project manager, product manager, release manager, public relations guy, bug tracker contact person, and bug fixer in this project which is mostly a one-man show. And while I like the development part, the other stuff I do out of sheer necessity.

New Feature: Bookmarks

I have one more feature that I think will be useful (I personally missed it while trying to really clean up my home directory that the big fat Internet browsers keep polluting with cruft with every mouse click): Bookmarks. You may have seen the huha-bookmarks branch.

It's still heavily work in progress. There is now a new toggle button at the left of the Breadcrumbs widget (the clickable path) where you can add a directory to the bookmarks list. That list will go to a subdirectory of the "Go" menu, so you can go quickly through directories that keep filling up with tons of stuff that the browser wants to cache.

The file will be kept as plain text in ~/.config/QDirStat/bookmarks.txt so it will be very easy to turn it into a shell script, or to write a script that uses it to, say, iterate over all those directories and clear their content.

While I feel that this will be useful, I'll only know it when I used it for some time. If I am not convinced that it actually is useful, I might decide not to merge it to master.

[Lithopsian]

OK, just a quick safe win then. If you change the signs of lightX and lightY in TreemapView.h (or the calculation with them in TreemapTile.cpp), it will move the lighting direction from below and slightly right to above and slightly left. The top left corner of the treemap, which is currently too dark, will get brighter. Obviously, some tiles towards the bottom right will get darker, but the overall effect is better.

The published paper shows (and describes) lighting from above and slightly right, but I think their formulae apply with y increasing upwards as per mathematical graphs, not downwards as on a computer screen. Anyway, it is trivial to try out the different lighting directions and see which you like best.

[Stefan Hundhammer]

Yes, that's a subtle, but definitive improvement. Thank you!