Improve --sort=due for urgent tasks

[mst]

It would be nice to have a sorted todo list with the most urgent task on the top.

Currently the todo list --sort=due command shows all tasks with a due date on top in the reverse order (least urgent first).
Adding the --no-reverse option shows the tasks with due date in the wanted order, however the tasks without due date are listed on the top.

Expected behavior: The tasks without due date should always be listed below the tasks with due date.

As a general improvement it would be nice to chain sort (and reverse) parameters, so that you could end up with a prioritized list of tasks, for example:

todo list --no-reverse --sort=due --reverse --sort=priority

to show a list sorted by due date and within the same due date by priority (i.e. for tasks without due date).

[Hugo]

I'm not sure this is easy given how our filtering/sorting works right now, it might require some pretty big changes.

I think todo --color=always | tac might serve as a workaround?

[mst]

Yes the tac workaround solves the first problem. Thank you!
I agree, the advanced sorting wouldn't be trivial, but maybe something to consider for a redesign someday. ;)

[balejk]

I think I fail to see how tac differs from --no-reverse, at least in my case it seems to be doing the same thing.

Anyway, to me it would make sense that todos with no due date should behave as having due date equal to infinity with respect to sorting, rather than zero which seems to be the case now. I have not looked into the code but this could be easy to do especially if sqlite supports this or at least allows for some other special value for the timestamp and custom sorting for it. This might then also relate to #125.

[mst]

Ok let me try to explain the differences by the following example:

> todo
[ ] 1 (no due date) minor todo
[ ] 2 2023-12-31 mid todo
[ ] 3 2023-10-01 major todo

To get the most urgent todo on the top I use tac for now:

> todo | tac
[ ] 3 2023-10-01 major todo
[ ] 2 2023-12-31 mid todo
[ ] 1 (no due date) minor todo

because --sort=due show the entries with due date on the top but not in the wanted order (=reversed):

> todo list --sort=due
[ ] 2 2023-12-31 mid todo
[ ] 3 2023-10-01 major todo
[ ] 1 (no due date) minor todo

With --no-reverse the entries are in the wanted order but not on the top:

> todo list --sort=due --no-reverse
[ ] 1 (no due date) minor todo
[ ] 3 2023-10-01 major todo
[ ] 2 2023-12-31 mid todo

Unfortunately this gets more complicated if you use the "Priority" flag on top because of the default sorting behavior of todoman.

Your proposal would definitely make sense for the due dates in my opinion and should solve this issue.

[balejk]

So if I understand correctly, you are rather comparing `todo | tac` with `todo list --sort=due` than `--no-reverse` with `tac` - is that right? I only had the first case in mind and I still think that `tac` is not needed: `todo list --no-reverse` should give exactly the same thing as `todo | tac` (although this is probably easier to type) in your second example. I agree that priorities seem to complicate things. Plain default `todo list` would be almost enough for me regarding the sorting (and for you too with `tac`/`--no-reverse`, if I understood correctly), except that it seems to sort primarily by priority and secondarily by due date. At the same time though, I currently have one task which has the highest priority and has a due date set and yet is not sorted among all other high priority tasks wih due date but instead is further up among high priority tasks with no due date, so I don't really see at the moment what the default sorting algorithm is.

[Hugo]

I think I fail to see how tac differs from --no-reverse, at least in my case it seems to be doing the same thing.

Now that you mention it, they should yield the same results.

To get the most urgent todo on the top I use tac for now:

The defaults keep the most urgent one at the bottom. If you have dozens of todos, the most urgent ones will be visible on screen and the less urgent ones are in your terminal's scroll-back buffer.

If you want most-urgent-on-top, you want --no-reverse (or | tac).

As a general improvement it would be nice to chain sort (and reverse) parameters, so that you could end up with a prioritized list of tasks, for example:

This is possible right now, e.g.:

todo list --sort due,-priority

The minus inverts the order for just that field. See the updated docs in 27c45f0

Regrettably, the default sorting cannot be expressed via the cli:

completed_at DESC,
priority IS NOT NULL, priority DESC,
due IS NOT NULL, due DESC,
created_at ASC

[balejk]

This is possible right now, e.g.: todo list --sort due,-priority

This is great! However I believe my suggestion for no due date being equal to infinity rather than zero still holds - what do you think about it @WhyNotHugo?

``` completed_at DESC, priority IS NOT NULL, priority DESC, due IS NOT NULL, due DESC, created_at ASC ```

This does not seem to correspond to what I'm seeing: `todo list` returns ``` [ ] 428 !!! in 21 days [ ] LOTS OF !!! (no due date) [ ] 498 !!! in 11 months [...] [ ] 502 !!! 19 hours ago ``` for me. I wonder, if it's just some database corruption though. I will try to refresh the cache later.

[balejk]

However I believe my suggestion for no due date being equal to infinity rather than zero still holds - what do you think about it @WhyNotHugo?

So looking at your example, this would translate to something like `todo list --sort=due` going over to `due IS NOT NULL, due DESC` and `todo list --sort=-due` to `due IS NULL, due ASC`. And again inspired by your example, I additionally think that this would make sense for priority also (i. e. no priority set should indeed mean no priority). And regarding the `tac`/`--no-reverse` equivalence, it does not seem to yield the same results for me after all. I don't know why I thought it does before but it seems to me that exactly the NULL conditions remaining the same (i. e. IS NOT NULL for both with and without `--no-reverse`, instead of IS NULL in the latter case) is the reason why it does not.

[mst]

And regarding the tac/--no-reverse equivalence, it does not seem to
yield the same results for me after all. I don't know why I thought it
does before but it seems to me that exactly the NULL conditions
remaining the same (i. e. IS NOT NULL for both with and without
--no-reverse, instead of IS NULL in the latter case) is the reason why
it does not.

Yes as you said the results of tac and no-reverse are not the same. For me it looks like that just tasks with a priority are reverted. That is why I focused on a fixed --sort=due.

Thanks for your hint with the advanced sorting functionality and the docu @WhyNotHugo !

[Hugo]

The advanced sorting was already there, it was just undocumented (and I'd honestly forgotten about it, I don't think that I wrote it).

I think we can add support for something like --sort=due:null-first or --sort=due:null-last.

[balejk]

I think we can add support for something like `--sort=due:null-first` or `--sort=due:null-last`.

That would probably be a good compromise, although personally I would be fine with it being fixed in the way that I described above because it feels natural (to me at least). @mst, what do you think? @WhyNotHugo Regarding `--no-reverse`, as it seems that it does not have the same effect as `tac` but rather seems to globally apply `-` to all sorting conditions individually, perhaps it could be obsoleted and removed after the above NULL suggestions are resolved?

[Hugo]

Once we have support for :null-first we can likely drop --reverse and --no-reverse entirely.