Treating files as entries

Question

Treating files as entries

ahmed-shariff opened this issue a year ago · comments

Hi there,

I've been flushing out trying to get org-roam to work with org-ql (https://github.com/ahmed-shariff/org-roam-ql, #303). I had just realized that org-ql doesn't treat a file as a possible entry, most likely because of the org-agenda integration. Since org-roam treats files as entries, this makes it tricky query when there are file entries. I was wondering if you have any thoughts or insight this? Would it be worth considering making org-ql aware of file entries?

Adam Porter · Answer 1 · Fri Jun 09 2023 04:59:18 GMT+0800 (China Standard Time)

Hi Ahmed,

Well, first let me nitpick about jargon: in Org terms, an "entry" is a heading and the contents of its body, not including any child entries. If we use consistent terminology, it will help us communicate more exactly. :)

So,yes, org-ql is designed to search entries and present each entry as a result.

If you need to see a whole file as a result, well, I think you could do something like set the ACTION argument to one that puts the current buffer's file on a list bound around the call to org-ql-select; and then you'd probably want to abort searching further entries once one is found.

Besides that, it would help if you could provide a bit more detail about the use case. Thanks.

Ahmed Shariff · Answer 2 · Fri Jun 09 2023 05:31:24 GMT+0800 (China Standard Time)

Well, first let me nitpick about jargon: in Org terms, an "entry" is a heading and the contents of its body, not including any child entries. If we use consistent terminology, it will help us communicate more exactly. :)

Thank you for clarifying that, I spent too much time trying to figure that out and seems I still had gotten it wrong 😅

To give a quick background on org-roam, it considers as a node the following (from https://www.orgroam.com/manual.html#The-Org_002droam-Node):

A node is any headline or top level file with an ID.

For example, with this example file content:

:PROPERTIES:
:ID:       foo
:END:
#+title: Foo

* Bar
:PROPERTIES:
:ID:       bar
:END:

We create two nodes:

A file-node “Foo” with id foo.
A headline-node “Bar” with id bar.

To reframe my original question: org-ql works with the entries/headline-nodes. But it's not aware of the file-nodes. Whereas in org-roam file-nodes are very common, in-fact the default capture template that comes with org-roam creates a new file and a file-node without any headings. As is the case with many of my notes. Since org-ql isn't working with them, I can't use any of the org-ql functionalities with these file-nodes.

On the other hand org-roam doesn't have a good way to search through all the nodes, display them and manipulate them, but it does allow using the sql database.
With https://github.com/ahmed-shariff/org-roam-ql, I am trying to use the results from querying the org-roams sql database and use org-ql-view as a front-end of sorts (#303). A quick demo of what that looks like:

I am trying to think of a way to show the file-nodes also in the save view. But I am not sure how to go about it - I was wondering if making org-ql aware of that might an option?

Adam Porter · Answer 3 · Fri Jun 09 2023 07:09:44 GMT+0800 (China Standard Time)

Well, being software, anything is possible. But org-ql's core procedures are designed around searching entries.

As for the org-ql-view library, well, it just displays a list of results, which begin as org-element lists--you can see how that's done in the code. I suppose you could add entries of your own to those results, of whatever type you want, and then adjust the formatting function to handle that type accordingly.

To go much further it would be helpful if you would give a few specific, very simple use cases, from the user's perspective.

Ahmed Shariff · Answer 4 · Sat Jun 10 2023 01:37:44 GMT+0800 (China Standard Time)

Well, being software, anything is possible. But org-ql's core procedures are designed around searching entries.

😁touché

I really appreciate you taking the time to go over this.

I do use the org-ql-view--display function to display the nodes, the issue is when I refresh the buffer, it discards all the file-nodes and only runs the query on the entries.

Consider the following example file:

:PROPERTIES:
:ID:       67a3bc01-6a9b-4c1b-9fc3-fd413badb8ba
:A_PROP: value 1
:END:
#+title: test1 file

* DONE Node a
:PROPERTIES:
:ID:       78fb2d53-4195-49b5-8699-7549d079859c
:A_PROP: value 2
:END:

If a query I run on the org-roam db had both the above ID's in the result, when I display it on a org-ql-view and refresh, only the second id (Node a) will be in the org-ql-view. While alot of functionality in the org-ql-view are built around entries, I think it's still useful to have both the file-nodes, and the headline-nodes displayed in the view. First, so that it is an accurate result of the query. Also, I can still manipulate them from within the org-ql-view buffer. For example, I extensively use a set of functions with the org-agenda-bulk-action.

Currently, I am trying to implement the following as a workaround to allow this:

Advice org-ql--select to avoid returning nil on file-nodes
Show a group on the org-ql-view that shows the file nodes in a separate group which are only filtered based the query run against the org-roam's db

I am trying to figure out if there is a better solution than that.

Adam Porter · Answer 5 · Sat Jun 10 2023 06:48:34 GMT+0800 (China Standard Time)

Well, I'm generally glad to help, but I'm not sure what I can add here. It seems like trying to integrate the two would require a lot of hacky changes, and those hacks would be likely to break in the future when org-ql or org-roam change. So, I wish you luck, but I'd advise caution, because I would hate to see your work go to waste. So I guess my advice would be to understand the problems as clearly as possible and design a robust solution before implementing it.

Ahmed Shariff · Answer 6 · Wed Jun 14 2023 02:59:43 GMT+0800 (China Standard Time)

Thank you for the advice. I guess this thread was more of a way for me to get some feedback. Again appreciate your time on this :)

Adam Porter · Answer 7 · Wed Jun 14 2023 19:06:21 GMT+0800 (China Standard Time)

You're welcome.

Liam Hupfer · Answer 8 · Fri Sep 08 2023 11:57:34 GMT+0800 (China Standard Time)

The latest Org syntax document calls the material before the first heading the zeroth section. It can have a PROPERTIES drawer, though it can’t have a planning line (scheduled, deadline dates).

What do you think about an implementation where Org QL treats this zeroth section as essentially a zero-asterisk heading? The title keyword value could be used as the heading in Org QL views, falling back to the filename if necessary. Of course, many predicates built around tasks would never match zeroth sections, but it would be great to get matches for simple queries on text, tags (via filetags), etc.

I haven’t looked at the code to see how feasible this would be yet, but if this makes sense to you and you would accept patches implementing this, would you consider reopening the issue? As the original discussion indicates, this would be particularly useful to Org-roam users, but it is applicable to Org in general as well.

Adam Porter · Answer 9 · Fri Sep 08 2023 16:37:42 GMT+0800 (China Standard Time)

I don't know. It depends on how invasive the changes would be and how they might affect performance. I expect it would require conditionals in many places, which I would like to avoid. And I wouldn't want performance on the rest of the file (which would be the vast majority of it) to be reduced.

I'm unlikely to experiment with this idea myself, so feel free to do so and show whatever patches you may come up with. I'm willing to offer feedback on them, at least.

Thanks.