[plugin] org-roam.nvim progress & discussion
chipsenkbeil opened this issue · comments
About this task
This issue started discussing a potential bug, and ended up evolving into a rundown of the ongoing work on org-roam.nvim, a plugin to implement Org-roam in neovim.
Scroll further down to see details about the plugin, code pointers and highlights, etc.
The original bug report is kept below for clarity.
Describe the bug
I am writing a plugin that creates some buffers with a buftype of nofile. The purpose of these is to generate some immutable orgmode content that you can navigate. In particular, I want to take advantage of the fantastic folding that orgmode offers.
Unfortunately, when I go to collapse a section, I get an error about the current file is not found or not an org file, caused by
orgmode/lua/orgmode/files/init.lua
Line 106 in 261c987
Steps to reproduce
- Create a scratch buffer via
vim.api.nvim_create_buf(false, true) - Set the filetype to org via
vim.api.nvim_buf_set_option(bufnr, "filetype", "org") - Populate with some org headings via
vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, {"* Heading 1-a", "** Heading 2", "* Heading 1-b"}) - Navigate to the buffer and hit tab to try to collapse or expand a section
Expected behavior
Section is properly folded.
Emacs functionality
No response
Minimal init.lua
-- Import orgmode using the minimal init exampleScreenshots and recordings
No response
OS / Distro
MacOS 14.4
Neovim version/commit
0.9.5
Additional context
Using orgmode on master branch at commit 651078a.
File is considered valid only if it has an org or org_archive extension. Try setting a name for a buffer that has an org extension with nvim_buf_set_name
local bufnr = vim.api.nvim_create_buf(false, true)
vim.api.nvim_buf_set_option(bufnr, 'filetype', 'org')
vim.api.nvim_buf_set_name(bufnr, 'somename.org')
vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, { '* Heading 1-a', '** Heading 2', '* Heading 1-b' })
vim.cmd('b'..bufnr)@kristijanhusak that solved the error about not being a current file! I'm hitting an issue where the plugin still doesn't detect a fold. There may be something wrong with my buffer or setup. Applies on any heading.
What would you expect to be folded here? Did you try doing zx to recalculate folds?
I see you are trying to implement org-roam. I had similar idea but didn't catch time to start working on it.
The first heading. If I create a file manually and reproduce the contents, I can fold it.
I had to make the repo private while I got permission to work on it on personal time. Now that I have it, here's the current plugin: https://github.com/chipsenkbeil/org-roam.nvim
About the plugin
Database
I wrote a simplistic implementation of a database with indexing supported that provides an easy search for links and backlinks. It isn't as full-fledged as SQL and I'm sure will struggle with larger roam collections, but for me this is enough.
Parser
Leveraging the orgmode treesitter parser for me to find the details I need to build the above database.
Completion of node under cursor
Covers both completing within a link and under a cursor. Essentially does the complete everywhere out of the box. Here's a demo:
example-org-roam-completion.mp4
Quickfix jump to backlinks
I like the quickfix list, so while Emacs doesn't have this and uses an org-roam buffer, this was easy to whip up for neovim:
example-org-roam-quickfix.mp4
Org buffer
I was generating an org file instead of writing a custom buffer. I may switch to the custom buffer highlighting and format anyway because I need to capture link navigation to open in a different buffer versus the current one.
Id navigation
Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently. So I'll need to write an org-roam variant to support opening id links. Shouldn't be hard at all. I also built out id generation for a uuid-v4 format in pure Lua. My test nodes aren't using it, though.
Thanks for opening it up. I just skimmed through it, and it seems you did a lot of custom stuff.
A recent refactor was also done to support org-roam with some of the internals. This is what I had in mind:
- Use
orgmode.filesto load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51 - Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though
- Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23
- Use orgmode capture class with custom templates only for org-roam where it would append the properties with id
Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently
In the issue description you wrote you are using commit 651078a. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua
Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.
Thanks for the pointers! When I looked at the plugin's API and data structures, it was (and I believe still is) missing crucial information I'd need for a fully-functional org-roam implementation. Would it make sense to move this discussion to a separate issue? I know you have the plugin API pinned issue, but there's a lot of different questions and dialog I'd want to have about needs and usage that feels better for a separate issue.
Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51
Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under cursor, etc.
Wonder how much of that has changed or was misinformed from my first look.
I also didn't check to see how you're loading files. I tried to keep most of my logic async - in the sense of leveraging neovim's scheduler and callbacks - so I can (in the future) easily report loading a large set of files, monitoring file changes and reloading, etc.
Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though
I don't know on this one. The database I wrote is a way to both track outgoing links (ones you'd find in a file) and incoming links (aka backlinks into a file). I like the design I've got, so I'll most likely keep this.
Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.
Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23
Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.
Use orgmode capture class with custom templates only for org-roam where it would append the properties with id
I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.
In the issue description you wrote you are using commit 651078a. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua
Good to know. I've got different versions on different machines. Given I started fiddling with this a couple of months back, I guess the org id was implemented after.
Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.
I want to reduce the hacks and redundancy where possible. My implementation is meant to build on top of your plugin to supply the org-roam features, but when I started it looked like there was enough missing that I ended up only leveraging the treesitter language tree to get the information I needed.
Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.
We can either have a separate issue or rename this one.
Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under the cursor, etc.
You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files
Files are loaded asynchronously using luv.
Most methods in file and headline provide the element content and the node containing location information (range).
For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.
Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.
This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.
I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.
I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.
Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.
Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.
Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.
I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.
We can either have a separate issue or rename this one.
Renamed this one.
You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using
luv. Most methods infileandheadlineprovide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.
I'll give these a look in a week or two to see how they operate.
This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.
I don't know if it makes sense to add it to that method, but the locations of links within a file are needed for org-roam in order to support listing multiple references to the same backlink and to be able to pull in a sample of the content that linked to a node. I think it's easier to see from this person's tutorial of the Emacs usage: https://youtu.be/AyhPmypHDEw?si=mLGsAdosnKjTZ-1f&t=1690
I think you will be able to use it. You just need to create custom
Templatesclass that gives the list of templates, and you can also create a customOrgRoamTemplateclass while extending Template class here, for stuff like dynamic file name with title, slug and such.
I was really hoping that I could reuse it, so this makes me happy to hear. :) Will be giving that a look as soon as I reach that point.
Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through
add_sourcemethod.
Got it. Yes, I definitely want to use your code to handle omnicompletion. The selector is more like a built-in telescope interface, which I wanted to have similar to what is shown in the emacs tutorial above when it comes to selecting nodes where you can not only select between choices but also filter the choices further. So having both is my plan.
I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.
I don't really mind where my plugin lives, so if you would be interested in taking this in at some point in the future once we remove as much of the custom logic as makes sense, then I'd be happy to hand it over to you. I need something like this for work, and it didn't exist, which is why I'm building it now.
I added range property to links in c4eeb3d that holds the position of the links, including [[ and ]] markers.
We can add a few more things as we go if you find them missing, just let me know.
@kristijanhusak are these indexed starting at 0 or 1?
It's 1.
@kristijanhusak I just updated to c4eeb3d and when trying to open an id link using org_open_at_point (bound to a local leader mapping), I'm still getting the issue about "No headline found with id: ...". I'm assuming this is because having a file-level id is unique to org-roam and not orgmode, which I "think" only has the concept of ids in property drawers at the heading level. But I'm not sure if global ids follow that logic or not.
From https://orgmode.org/manual/Handling-Links.html
If the headline has a ‘CUSTOM_ID’ property, store a link to this custom ID. In addition or alternatively, depending on the value of org-id-link-to-org-use-id, create and/or use a globally unique ‘ID’ property for the link 28. So using this command in Org buffers potentially creates two links: a human-readable link from the custom ID, and one that is globally unique and works even if the entry is moved from file to file. The ‘ID’ property can be either a UUID (default) or a timestamp, depending on org-id-method. Later, when inserting the link, you need to decide which one to use.
Given this fact and your desire to maintain only core orgmode functionality within this plugin, I'm assuming that unless the global ID at a file level is part of core orgmode, I will need to write my own open logic for links that navigates to the correct id that supports file-level ids. The thought process is to have that function fall back to your implementation if the link is not an id link. Thoughts?
I changed the org-roam buffer to more closely match the org-roam variation. Looks like they changed to use magit underneath, which I'm just replicating in style right now.
Also, progress on following node change under cursor. Don't know the performance considerations of this given I've got tiny tests.
cursor-following-buffer.mp4
Id links work fine for me. There's also a test that confirms that it's working. Note that your files need to be part of org_agenda_files otherwise, it won't be found.
@kristijanhusak your test has a property drawer with an id that is within a section with a headline. Does it work with a top-level property drawer?
The case I'm referring to is a top-level property drawer not within a section.
My dotfiles configure every org file (including those of the roam directory) to be within the agenda:
Ah yes, you are correct, there was no support for top-level id. I added that now on the latest master, let me know if it's not working as expected.
@kristijanhusak fantastic! Quick test has it working just fine. :) One less thing I have to manage myself.
@kristijanhusak I haven't been able to find it yet. Do you have a version of org-id-get-create (part of org-id.el)?
This is referenced in org-roam's manual and is used to inject an ID into a head as seen in this demo.
Yes, you can use this https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua
local id = require('orgmode.org.id').new()There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.
There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.
This is what I'm specifically looking for, which both does the work of generating the id and placing it in the headline. Is there a version that also works with a property drawer at the file level?
When I tested org-id-get-create in Emacs, it works with a property drawer at the file level. So, ideally, this would be a command that someone could run whether they're in a heading or not.
Here's an example:
org-id-get-create.mp4
We could add that, but I don't think you will need it. For org-roam you will have custom templates that will already populate this information before showing the capture window. When creating a template, just call the orgmode.org.id to generate an id for you.
function OrgRoamTemplate:compile()
--call parent compile
local content = OrgTemplate:compile()
content = vim.list_extend({
':PROPERTIES:',
':ID: '..require('orgmode.org.id').new(),
':END:'
}, content)
end@kristijanhusak I'm switching back to having the org-roam buffer be an actual org file w/ syntax. I've seen examples of org-roam that use magit as the interface and others where it has a plain org file.
From the discussions I've seen regarding how the buffer is used, people will either use it to open a backlink/citation/unlinked reference directly in the frame (via RET) or they can force it to open in a different frame (via C-u RET).
I know OrgMappings:open_at_point() exists to open the link or date at the point given. Is there any function that lets you open at point while specifying a different window? Was looking at that function and considering if I need to build a wrapper for that function to be able to point to a different window.
Org-roam manual reference
Link: https://www.orgroam.com/manual.html#Configuring-the-Org_002droam-buffer-display
Crucially, the window is a regular window (not a side-window), and this allows for predictable navigation:
- RET navigates to thing-at-point in the current window, replacing the Org-roam buffer.
- C-u RET navigates to thing-at-point in the other window.
Emacs manual reference
Link: https://orgmode.org/org.html#Handling-Links
There's a minor reference to org-link-frame-setup, which appears to let you configure how files and directories are opened including other frames:
There's a minor reference to org-link-use-indirect-buffer-for-internals, which I opened up below:
Other references
We could extend it to accept the command that opens the "point", and default it to edit. Would that work for you?
We could extend it to accept the command that opens the "point", and default it to
edit. Would that work for you?
I think that should work, yeah. As long as there is a way for me to specify the window to use versus the current window.
I think I'd do this by using wincmd:
---@param winnr integer # id of window where we will open buffer
---@return string # to supply to vim.cmd(...)
function make_cmd(winnr)
local cmd = winnr .. "wincmd w" -- goes to window "winnr"
return cmd .. " | edit"
endUpdate
Leveraging loading a singular orgfile per preview in the org buffer. I was reading through how org-roam does this (via changelog and source), and it looks like it handles certain cases specially such as detecting if the link is in a heading and just showing the heading text or detecting if the link is in a list item and showing the entire list.
I'll be switching this back over to an org buffer soon, but here's a preview:
This is my logic for now, although I'm assuming I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version? In my old approach, I would stat each file to check its mtime.sec against a cached version.
-- NOTE: Loading a file cannot be done within the results of a stat,
-- so we need to schedule followup work.
vim.schedule(function()
require("orgmode.files")
:new({ paths = opts.path })
:load_file(opts.path)
:next(function(file)
---@cast file OrgFile
-- Figure out where we are located as there are several situations
-- where we load content differently to preview:
--
-- 1. If we are in a list, we return the entire list (list)
-- 2. If we are in a heading, we return the heading's text (item)
-- 3. If we are in a paragraph, we return the entire paragraph (paragraph)
-- 4. If we are in a drawer, we return the entire drawer (drawer)
-- 5. If we are in a property drawer, we return the entire drawer (property_drawer)
-- 5. If we are in a table, we return the entire table (table)
-- 5. Otherwise, just return the line where we are
local node = file:get_node_at_cursor({ opts.row, opts.col - 1 })
local container_types = {
"paragraph", "list", "item", "table", "drawer", "property_drawer",
}
while node and not vim.tbl_contains(container_types, node:type()) do
node = node:parent()
-- Check if we're actually in a list item and advance up out of paragraph
if node:type() == "paragraph" and node:parent():type() == "listitem" then
node = node:parent()
end
end
-- Load the text and split it by line
local text = file:get_node_text(node)
item.lines = vim.split(text, "\n", { plain = true })
item.queued = false
-- Schedule a re-render at this point
opts.emitter:emit(EVENTS.CACHE_UPDATED)
return file
end)
end)we could also make the argument a function, that would default to:
local edit_cmd = edit_cmd or function(file) return 'edit '..file end
vim.cmd(edit_cmd(filename))That might give you more control over it.
I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version?
Yes, you should go with the single instance. Here I have an Org instance that holds other instances.
Regarding caching, I do same for files that are not loaded, and check the buffer changedtick if the file is loaded inside a buffer. You can see the logic here
orgmode/lua/orgmode/files/file.lua
Lines 108 to 119 in b7c42e6
we could also make the argument a function, that would default to:
local edit_cmd = edit_cmd or function(file) return 'edit '..file end vim.cmd(edit_cmd(filename))That might give you more control over it.
I think either solution should work.
Org roam buffer
I'm still hitting some quirks with creating an org buffer that is of buftype = nofile.
- As you pointed out earlier, I have to supply
.orgat the end of the buffer name. It looks like as a result of this, orgmode updates the buffer name to be relative to the orgfiles directory for me. So now if I want to ensure that I'm not within my own buffer, I have to dovim.endswith(vim.api.nvim_buf_get_name(0), "org-roam-buffer.org")instead of equality. - It looks like my buffer's preferences to be both unlisted and a scratch buffer get overwritten. Maybe you're swapping out the buffer on me (don't know yet). This means that this temporary buffer shows up in the buffer list.
- I still can't get folding to work. Running
zxdoesn't appear to update the folds. - Reloading the buffer clears the contents (this is on me to fix) and requires to jump elsewhere to refresh. But the callout is that the syntax highlighting appears to break partially when the buffers contents are reloaded again. The comment and heading colors.
example-of-org-buffer-issues.mp4
I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.
I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (
vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.
Sure, I can experiment. I think the broader issue is it seems like orgmode really wants org buffers to represent files on disk, while I am making a buffer-only org document with content that I dynamically generate and update on cursor change.
@kristijanhusak still no luck, so there must be something else I'm doing wrong. Even persisting the file to disk, the folds aren't working. Everything else is fine.
The logic is in the NodeView Window, where I'm maintaining an OrgFiles cache across all windows and another cache where I've calculated lines to show in a preview per backlink. All of that works fine, so I must have something wrong set up w/ creating the org buffer.
Maybe there's something else wrong in my setup. I have to hack orgmode a bit to get it to work, including a full reload (edit!) for the current buffer when the orgmode plugin first loads.
It does look like the ftplugin is triggering for org as I can see b:undo_ftplugin is set. For some reason, the fold levels aren't being detected. Everything has a foldlevel of 0.
Seems like if I change from being nomodifiable and nofile to a regular buffer and modifiable and then write the file, I can get folding. I also need to then reload the file via :e to get folds to fully work as some of them don't fold properly. When I took a look at your autocmds, it looks like you just do require("orgmode"):reload(file), which I tried to replicate with the name of the buffer, but so far I can't get it to work unless I make it modifiable, manually write via :w, and then reload via :e.
I investigated a bit what's happening, and I think just additionally doing filetype detect should solve an issue for you. Here's a fuction that I used for testing (simplest to be put in init.lua):
_G.test_org_buffer = function()
local buf = vim.api.nvim_create_buf(false, true)
vim.api.nvim_buf_set_option(buf, 'filetype', 'org')
-- You can maybe use some naming convention like this, but not necessary
vim.api.nvim_buf_set_name(buf, 'org:///org-roam-backlinks.org')
vim.api.nvim_buf_set_lines(buf, 0, -1, false, { '* level 1', '** level 2', '** level 2.2' })
vim.cmd('b'..buf)
vim.cmd('filetype detect')
end- Open nvim
- execute
:lua test_org_buffer()
@kristijanhusak seems like that works. I have it re-detecting the filetype every time I rewrite the buffer. Gotta figure out the best approach to do this though as the buffer can be visible or invisible, and you may not have it selected, either. I wanted to use vim.filetype.match({ buf = buf }), but it doesn't fix the issue. Only filetype detect seems to work.
Right now, trying to see if there is a way for me to set the fold level explicitly when the buffer is updated. I essentially want foldlevel of 1 such that the previews under the links are hidden by default, but can't figure out a way thus far. Balancing act with the orgmode settings getting applied.
@kristijanhusak while filetype detect works, it's too difficult for me to retrigger it when the buffer can get updated elsewhere (e.g. cursor change in another window leads to an update). I'll put this on hold since I can't figure out what exactly about that is causing folding to work.
Will look at tackling the last MVP feature needed which is templating. From there, I can work on leveraging more of this plugin in place of custom code.
Another question for you: are citations supported in parsed files? Orgmode 9.5 added native support for them.
are citations supported in parsed files?
No, not yet. You need those for org-roam?
@kristijanhusak for the minimum viable product? No. For academics that want to use the plugin? Yes.
The org-roam manual references support for citations including orgmode 9.5's built-in variant. For org-roam, they are used through ROAM_REFS, which to my understanding is a way to connect non-id links to nodes; however, orgmode citations only have a singular bracket on either side (youtube example).
[cite:@better_speech]
So alongside your get_links() method, we'd also need a get_citations() method given that they don't match the link syntax of double square brackets.
Technically, org-roam supports org-ref as well, which has the format of just cite:better_speech, but I'm fine just supporting the built-in format.
@kristijanhusak took a look at the template class you referenced to extend.
I see that you have a list of expansions that can be applied and that the _compile_expansions method takes an optional list of expansions as a secondary argument (found_expansions), but it looks like that is never used. Ideally, I'd like to provide additional expansions (e.g. the node id, or references to the node under cursor when capturing), but it looks like there's no way to plug in new expansions on top of the existing ones for a template? If this was available, I think I could avoid having to extend an entirely new class in favor of just providing org-roam themed expansions.
orgmode/lua/orgmode/capture/template/init.lua
Lines 212 to 220 in b7c42e6
-- Creates a template for use with org-roam nodes
local template = Template:new({
template = {
":PROPERTIES:",
":ID: %i",
":ORIGIN: %n",
":END:"
},
-- Provide a mapping of expansions to use that includes default expansions along with new ones
expansions = vim.tbl_extend("keep", Template.EXPANSIONS, {
-- Generates an org id
["%i"] = function()
return require('orgmode.org.id').new()
end,
-- Yields the id of the node under the cursor during capture
["%n"] = function()
return get_id_of_node_under_cursor()
end,
}),
})It could also be handy to support some pattern matching. Org roam's manual references expansion logic that supports looking up user-defined functions to invoke and use for substitution. For example, ${foo} would get translated into foo(node-under-cursor), org-roam-node-foo(node-under-cursor), or use completion to look up the input for the variable.
So having something like %{foo} would be handy, but it looks like your compilation of expansions limits to escaped magic patterns.
I guess we could add something like custom_expansions for this purpose, but you could also use "eval" expansion to call custom stuff (%(return require('orgmode.org.id').new())), here's an example test. Note that %i that you put in the example does something else in Orgmode
That pattern matching seems like an org-roam specific functionality. I'd prefer not to add plugin specific logic here.
That's why I suggested to extend the class and override/extend methods with org-roam specific stuff.
you could also use "eval" expansion to call custom stuff (%(return require('orgmode.org.id').new()))
Oh, I missed that detail. That does what I'd need.
Note that %i that you put in the example does something else in Orgmode
Good to know :) That was just an example, and I hadn't thought of what those would be.
That pattern matching seems like an org-roam specific functionality. I'd prefer not to add plugin specific logic here.
That's why I suggested to extend the class and override/extend methods with org-roam specific stuff.
Fair enough. With the eval expansion you mentioned, I can get by. It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).
It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).
Yes, that is an option, but it is cumbersome. My initial plan was to extend the Template class for org-roam and compile these template variables in there.
We could refactor a template class to accept custom hooks through something like template:on_compile(fn), and then you will be able to hook into it with your own stuff. Current functions could be also treated like builtin hooks. What do you think?
It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).
Yes, that is an option, but it is cumbersome. My initial plan was to extend the Template class for org-roam and compile these template variables in there. We could refactor a template class to accept custom hooks through something like
template:on_compile(fn), and then you will be able to hook into it with your own stuff. Current functions could be also treated like builtin hooks. What do you think?
Yeah, that could work. To confirm, the idea is that passing a function to on_compile would let us modify the content? Would it take the content as input and return the modified version as output?
Yes, that's the idea. Example:
local template = Template:new({ template = 'This is {title}' })
template:on_compile(function(content)
return content:gsub('{title}', 'org-roam')
end)
local content = template:compile() -- Result: This is org-roamAdded in cc1c4c2
Added in cc1c4c2
Great! I would say a minor nit is it would be nice to have on_compile return the template so you could chain like this:
local content = Template
:new({ template = 'This is {title}' })
:on_compile(function(content) return content:gsub('{title}', 'org-roam') end)
:compile()Sure, force pushed in bb89dfc
If I haven't said it already, really appreciate how engaged and supportive you've been 😄
The plugin is shaping up, and I'll be tackling the templating over the next few days (maybe it'll be quick).
From there, I'll be working to flesh out testing a little more at an integration level and remove large portions of custom code (e.g. the custom parser) in favor of the core plugin. It's taking shape, which is exciting to see as I'm badly in need of org-roam capabilities in my day-to-day.
Thank you for the kind words :) I'm glad someone took the initiative to build this. It is not a small project to handle, but I know a lot of people want to use it.
I don't need roam functionalities that often, but I had a few situations where it would be helpful.
As you go let me know if there's something else that might be helpful and we can discuss.
@kristijanhusak I'm just about done with the initial version of the org-roam buffer. I ended up going back to not using an org syntax and instead having a custom buffer (video below) due to technical challenges I couldn't figure out. I like how it works now - I even have a nifty way to select opening backlinks in other windows.
The one thing I don't support is syntax highlighting when previewing the contents from a backlink. I'm putting that as a "nice to have" but was wondering if you had any thoughts on a way to perform syntax highlighting (and link concealing) for a range of text within a non-org buffer that is supposed to represent org syntax.
no-highlight-orgmode.mp4
@kristijanhusak also, one issue I'm facing that is pretty challenging is that OrgRange does not including the starting and ending offset, only the starting and ending line and column positions.
I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information. Is there any way I can re-calculate the offsets from OrgRange? Or better yet, have OrgRange include the offset positions?
Right now, I've written this to see if it would work:
---Converts from an nvim-orgmode OrgFile and OrgRange into an org-roam Range.
---@param file OrgFile #contains lines which we use to reconstruct offsets
---@param range OrgRange #one-indexed row and column data
---@return org-roam.core.parser.Range
function M:from_org_file_and_range(file, range)
local start = {
row = range.start_line - 1,
column = range.start_col,
offset = range.start_col - 1,
}
local end_ = {
row = range.end_line - 1,
column = range.end_col,
offset = range.end_col - 1,
}
-- Reverse engineer the starting offset by adding
-- the length of each line + a newline character
-- up until the line we are on
for i = 1, range.start_line - 1 do
local line = file.lines[i]
if line then
start.offset = start.offset + string.len(line) + 1
end
end
-- Reverse engineer the ending offset by adding
-- the length of each line + a newline character
-- up until the line we are on
for i = 1, range.end_line - 1 do
local line = file.lines[i]
if line then
end_.offset = end_.offset + string.len(line) + 1
end
end
return M:new(start, end_)
endI'm putting that as a "nice to have" but was wondering if you had any thoughts on a way to perform syntax highlighting (and link concealing) for a range of text within a non-org buffer that is supposed to represent org syntax.
AFAIK you could do that only if you used a tree-sitter highlighting for your org-roam buffer and inject org highlighting through treesitter injections.
What technical challenges do you have with using an org filetype in the org-roam buffer? I know there were some issues with filetype detection but I thought we managed to solve those. Also, you mentioned that you have a bunch of buffers to manage and filetype detect needs to be run on the buffer. Did you consider using https://neovim.io/doc/user/api.html#nvim_buf_call() for those types of calls?
I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information
From the code, it seems that you need offset from the start of range. I'm curious why you need that? I understand it's probably hard to explain it concisely but please give it a try.
AFAIK you could do that only if you used a tree-sitter highlighting for your org-roam buffer and inject org highlighting through treesitter injections. What technical challenges do you have with using an org filetype in the org-roam buffer? I know there were some issues with filetype detection but I thought we managed to solve those.
It seemed like using filetype detect worked, but I needed to continually invoke it whenever the buffer was updated, which happens when you move your cursor to a different node.
I had a lot of hack logic to try to get filetype detect to work in different scenarios, and I finally ran into an issue of switching windows to the buffer in order to use filetype detect triggering autocmds that in turn re-ran the buffer render and causing loops. There was also an issue with some options getting overridden, but I can't remember which ones in my pursuit to get the org buffer to work.
Also, you mentioned that you have a bunch of buffers to manage and
filetype detectneeds to be run on the buffer. Did you consider using https://neovim.io/doc/user/api.html#nvim_buf_call() for those types of calls?
I honestly forgot about nvim_buf_call and wonder if it would have helped avoid triggering autocmds when it switches to a window containing the buffer. I think I'm going to leave this as-is for now, but in the future could have a configuration option to let people choose what type of buffer they want.
A nice bonus of the new buffer is that I can lazily look up previews from files instead of pulling them all at once to fill in the org buffer. And I can control the navigation a bit more as the current buffer lets you press <Enter> on a link to open it in a different window only (versus replacing the org-roam buffer). I also have control to jump to not only the line but also the column position of the link versus only being able to have links to lines in org.
If/when I use org for an org-roam buffer, I'd need the feature we talked about in terms of having opening link under cursor support a parameter that lets me replace edit with something like 1234wincmd w | edit.
I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information
From the code, it seems that you need offset from the start of range. I'm curious why you need that? I understand it's probably hard to explain it concisely but please give it a try.
There are a few reasons why I need an offset, some of which can be switched over to line and column:
- I use it for interval trees. I wrote a tree.lua to support fast detection of data structures under cursor.
a. I use this to figure out the node under cursor by mapping a section's starting and ending offset into the tree with the node's data as the value.
b. I use this to quickly figure out which node each link within a file belongs to by putting nodes into the tree and seeing which node is lowest on the tree for a link's offset.
c. I use this to determine if the cursor is over a link. - I use it for ordering of sections, links, and more, which could be switched to line and column position.
- I use it to calculate the range for a section that includes a heading and other elements, which could probably be switched once I use
OrgFilein place and load up headings, but I don't know. I just need section ranges. - I use it for a lot of my parser logic, which will go away if I can fully switch to
OrgFile. - I use it to get the string content of something (link, section, tags, etc) so I can hash them, but I don't use this feature yet so not important.
I think interval tree usage is the biggest blocker.
There are a few reasons why I need an offset, some of which can be switched over to line and column
This seems a lot more complicated than I assumed at first.
I wouldn't calculate offset anything better than you do, considering that it's usually created from TSNode, which only contains the range information. Since I don't always have access to a file and its lines when creating OrgRange, I think you will have to stick with your version.
@kristijanhusak I'm working on the capture templates now. To support org-roam-node-insert and org-roam-node-find, I need to be able to prompt for a template, have the refile happen, and get a callback that tells me:
- Did the refile happen or was it canceled?
- If the refile happened, what file was modified or created?
- If the refile happened, what is the id of the node created?
When using OrgCapture with OrgCapture:prompt(), it doesn't look like I have any control on getting insight regarding whether a capture was completed or not. Even with OrgCapture:open_template() I don't have control over the callbacks for opening, closing, or finishing.
What is your recommendation for how to proceed with this? Is there a way to use the orgmode capture api to get that data? I need to know it to inject links or follow the created/modified file.
You can see some of the functionality in this portion of code https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/capture/init.lua#L34-L85
Basically, CaptureWindow accepts on_open and on_close. When the refile is canceled (default <leader>ok), on_close is not triggered. If the user attempts to do a refile, on_close is triggered. Then, in the callback, you can detect if the buffer is in a modified state and act accordingly, like it's done in the on_refile_close. Regarding the node, there isn't any information at that moment. There's no way for you to hook in into this from the OrgCapture class, so we can add option to Capture:new() to provide custom on_close function that will be additionally triggered when refiling is happening.
Regarding 2., capture window is always a "create" state, so it's not used for modifications. AFAIK org-roam doesn't have any "modify" functionality for captures, but I maybe missed something.
Regarding 2., capture window is always a "create" state, so it's not used for modifications. AFAIK org-roam doesn't have any "modify" functionality for captures, but I maybe missed something.
Oh, by modified, I meant which file is the capture creating or updating.
As for everything else you mentioned, sounds good. Should I wait for OrgCapture to support an on_close function? Or build out my own version of OrgCapture to do the work you mentioned instead?
I added on_close in 3d3dfe3. As first arg you get the whole capture instance, and 2nd argument is refile vars, which contain source and destination file + headline for the capture. Hopefully, this should provide you with enough information. This is triggered before any actual processing of the capture. I don't know if you need some way to stop execution of the capture, but I didn't add support for that. If you need it, we can have false value returned from it as a stop.
@kristijanhusak great! I'm giving that a go, and for some reason neovim itself exits when the refile happens. I've commented out my on_close logic and removed on_close entirely from the capture and this still happens. Any idea what I'm doing wrong?
I can see that the refile works as it shows up in the existing file.
My code is here: https://github.com/chipsenkbeil/org-roam.nvim/blob/main/lua/org-roam/node.lua#L101
Screenflick.Movie.20.mp4
Line 171 doesn't seem properly commented out. Try removing the whole on_close and see if it stops.
Line 171 doesn't seem properly commented out. Try removing the whole
on_closeand see if it stops.
I did after the video and it still happens. Here's another video.
Screenflick.Movie.22.mp4
Try opening some other random file, and then invoke a capture. Does it still happen? This might cause it to quit the whole Neovim because there's only one window https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/utils/init.lua#L436
@kristijanhusak that was it. Here's another video showing that it closes a window and leaves the other open.
Screenflick.Movie.23.mp4
It should happen only if the capture window is the only window at the moment of finishing the capture. Without it, nvim_win_close errors out with cannot close last window.
It should happen only if the capture window is the only window at the moment of finishing the capture. Without it,
nvim_win_closeerrors out withcannot close last window.
I don't know why that isn't happening.
Is this something with my setup? Or an issue with the orgmode plugin? Or both?
Same logic is used for orgmode. If you don't have this issue in orgmode, then it's probably something in org-roam that's causing it.
@kristijanhusak it's now happening for me with org-capture from this plugin as well. I completely removed the org-roam.nvim plugin to test and can verify it was not loaded.
@kristijanhusak working towards narrowing down the issue, but something definitely changed in orgmode, I just don't know when and may have not noticed it.
On commit facd9d5, it works like expected, although I will say that this is not the behavior I would expect (getting an error at all):
[UPDATE] Here's some working commits I'm tracking:
What would you expect to happen if the capture window is the only visible window and you finish capturing it?
What would you expect to happen if the capture window is the only visible window and you finish capturing it?
In my videos, it's not the only window when neovim closes. I have a normal org file open. Then I start a capture, which opens a new window for the capture. When the capture finishes, I would expect to return to my single org window, but it seems like the capture window closes, then the check happens that sees a single window, and neovim closes.
Ah ok I reproduced it. I use <C-c> to finish capture, and it doesn't have that issue. I'll look into it.
Ah ok I reproduced it. I use
<C-c>to finish capture, and it doesn't have that issue. I'll look into it.
Oh, yeah, that is different :) I use :wq or ZZ
Should be fixed now in 606c747
@kristijanhusak after you mentioned your means to finalize the capture buffer, I noticed that you have finalize capture as <C-c>. Emacs uses <C-c><C-c> to finalize, and has a bunch of other bindings that start with <C-c>. Org-roam also encourages a prefix of <C-c>, so all of my default bindings using <C-c> as the prefix.
What are your thoughts on switching finalize to <C-c><C-c> to prevent conflicts by default (breaking change)? Users can override this to just be <C-c>, but avoid having <C-c> take up the entire prefix.
<C-c> is only bound in the capture buffer for finishing the capture. It's not used anywhere else. I'm not sure why I went with it, but I'd rather not change it now.
Do you plan to use <C-c> as a prefix key for all the mappings? I'm not sure what your preferences are, but I wouldn't suggest that. <C-c> prefix is very "emacsy" and I don't think many people would find it appealing. I could easily be wrong though, but I didn't see it as a pattern over the last 10+ years in various configs and plugins.
<C-c>is only bound in the capture buffer for finishing the capture. It's not used anywhere else. I'm not sure why I went with it, but I'd rather not change it now.Do you plan to use
<C-c>as a prefix key for all the mappings? I'm not sure what your preferences are, but I wouldn't suggest that.<C-c>prefix is very "emacsy" and I don't think many people would find it appealing. I could easily be wrong though, but I didn't see it as a pattern over the last 10+ years in various configs and plugins.
Yeah, it's definitely more emacsy, but I was mirroring it such that people moving over from Emacs would feel more comfortable. So all of my prefixes start with <C-c> to mirror org-roam, which means that none of them could be used within a capture buffer today.
but I was mirroring it such that people moving over from Emacs would feel more comfortable
I'm trying to do a similar thing here, but judging from the issues reported, I can tell that at least 90% of the users never used Emacs orgmode before. They start with this and figure things out as they go.
Anyway, you can just add a note to readme that users should override the default capture mapping in the orgmode configuration if they want everything to work. For example:
-- Override `org_capture_finalize` mapping to make org-roam mappings work in capture window
require('orgmode').setup({
mappings = {
capture = {
org_capture_finalize = '<Leader>w',
}
}
})
require('org-roam').setup()@kristijanhusak sounds good to me. I may just look at switching away from <C-c> per your recommendation if 90% of users haven't used orgmode before and just have a guide for those coming from orgmode in terms of how they could configure the keybindings.
I expect to have a version of org-roam.nvim ready for people to use in a week. If I don't get it ready by next Wednesday, it'll be another couple of weeks since I'll frequently be away from my computer.
One aspect of templating I haven't checked yet is how templates handle content that should only be generated when the file is new versus every time. For example, I want to only create the :PROPERTIES: with an :ID: when the file is created, but every time I want to add * SOME CONTENT HERE regardless of the file being new or not.
My first thought is to check if the file exists from on_compile and, if not, prepend the :PROPERTIES: drawer. Or is there a better way in your mind?
Note: I may misunderstand how capturing works. I haven't figured out where the refile writing to file happens.
Another issue I encountered. I turns out that the on_close you provided for capture may not be at the right time. Ideally, I need to re-scan the file that is created or modified, but on_close happens before the refiling takes place and there's no guarantee that the refile even happens if the user is prompted and cancels.
So at the moment I would kick off a task that checks for the destination file to be created or updated and keep polling until I detect that or some time expires.
There's no way to hook into the file update process today. Looking through the source, I think I'd need to hook into post-write of the destination file (maybe add a callback within OrgProcessCaptureOpts triggered from _refile_from_capture_buffer. Or, if there was a way to capture updates to OrgFiles, then I could just hook into that at a global level and update the org-roam database whenever a file is updated.
Is there a way to do this today? Any recommendations here?
I moved the on_close to the end in 5a5b486, just before closing the capture window.
@kristijanhusak testing out captures. Using on_compile, I can see the property drawer and title get injected when a file is new using the approach I described earlier. When the refile happens, it looks like the property drawer and title are missing from the created file.
Am I doing something wrong? I was expecting the content from the capture buffer to be written directly to the file and am not sure why it only has the headline.
capture-missing-content.mp4
Update here. It looks like the refile is happening with the headline. If I move the property drawer and other content after the headline, then it gets included, parsing the file works to find the node, and it gets inserted. This is obviously not the behavior I'd expect for a capture.
Separately, I also saw that navigating by id doesn't work for this newly-created file. Do I need to manually load it within require("orgmode").files for the id to be picked up? Not a big deal if that's the case, just figuring out how to get the main plugin to recognize the newly-created file's id.
has-content.mp4
It looks like the refile is happening with the headline. If I move the property drawer and other content after the headline, then it gets included, parsing the file works to find the node, and it gets inserted. This is obviously not the behavior I'd expect for a capture.
Yeah, it grabs the first headline. It doesn't do that only if the content has no headline.
To circumvent this, I added whole_file option to the template in 63e4c5a, so just provide that additionally when creating templates.
Separately, I also saw that navigating by id doesn't work for this newly-created file. Do I need to manually load it within require("orgmode").files for the id to be picked up?
Doing require('orgmode').files:load(true) should reload everything (it knows how to ignore already loaded and up to date files).
Note that the idea with files was to initialize it with the custom path for org roam, and not rely on the require('orgmode').files. General suggestion around it is to not share paths between org and org-roam.
This is what I had briefly in mind:
function OrgRoam.setup(opts)
-- opts.paths = { '~/org-roam/**/*' }
self.files = OrgFiles:new({ paths = opts.paths })
self.capture = OrgCapture:new({ files = self.files, templates = {} })
end
-- Mappings
vim.keymap.set('n' '<leader>ro', function()
return roam.capture:prompt()
end, { desc = 'Org roam capture' })I just figured out that you cannot rely on files argument from on_close because default mappings for capture point to orgmode specific capture instance so you would get orgmode.files instead of files you provided to your own capture instance. I'll see how to abstract that.
In d160a95 I updated mappings to be bound to the capture instance that is triggering it, so now you should have proper arguments in on_close.
@kristijanhusak does OrgFiles:new(...) load the files from the supplied paths synchronously? Want to make sure this is async, so if that's the case, I guess I'd need to set paths to nothing, create OrgFiles, update the paths field, and then call :load()?
If this is the case, having the ability to do something like OrgFiles:new({ paths = ..., lazy = true }) would be nice to avoid the sync loading by default.
Yeah it loads files synchronously on :new(). I removed that functionality in c79e06c, so you can do it async now.
local files = OrgFiles:new({ paths = ... })
files:load():next()@kristijanhusak hm, I tried using OrgFiles:load_file, but that didn't appear to populate the path into filenames() even if it is within one of the provided globs.
I then tried OrgFiles:load(false) and it didn't load the new file. So I then did OrgFiles:load(true) and it did load the newly-created file, but navigating by opening the id link still fails with the "No headline found with id: ..." error message.
I'm on c79e06c and can confirm that my OrgFiles is being passed to Capture:new and only contains the roam org files and not everything else. I can also confirm that at the time of capture finishing and load succeeding (only the force attempt), the newly-created file shows up in OrgFiles.
Is there something else I need to do? Something I can debug from OrgFiles to see if it's set up correctly?
Was testing out the plugin against https://github.com/jethrokuan/braindump/tree/master, which has over 500 nodes, in order to see how performance works. After a tweak on my plugin, it's doing okay (woo!).
I did notice again that ID navigation does not work. I changed my roam directory to point to a completely separate folder from my main orgfiles such that there is no overlap, and now none of the ids work. So it looks like I still need to update the main OrgFiles instance?
Previously:
# Contains my main orgfiles, which I've configured for nvim-orgmode/orgmode
~/orgfiles
# Contains my org-roam files
~/orgfiles/roam
Now:
# Contains my main orgfiles, which I've configured for nvim-orgmode/orgmode
~/orgfiles
# Contains org-roam files that I'm testing
~/project/braindump/org
For the time being, I got it to work by injecting a wrapper around OrgMappings:open_at_point. Would love your thoughts on this to see if it's a good way forward or if there's something else I should do?
local function modify_orgmode_plugin()
-- Provide a wrapper around `open_at_point` from orgmode mappings so we can
-- attempt to jump to an id referenced by our database first, and then fall
-- back to orgmode's id handling second.
--
-- This is needed as the default `open_at_point` only respects orgmode's
-- files list and not org-roam's files list; so, we need to manually intercept!
---@diagnostic disable-next-line:duplicate-set-field
require("orgmode").org_mappings.open_at_point = function(self)
local link = require("orgmode.org.hyperlinks.link").at_pos(
vim.fn.getline("."),
vim.fn.col(".") or 0
)
local id = link and link.url:get_id()
local node = id and require("org-roam.database"):get_sync(id)
-- If we found a node, open the file at the start of the node
if node then
vim.cmd.edit(node.file)
return
end
-- Fall back to the default implementation
return require("orgmode.org.mappings").open_at_point(self)
end
end
@kristijanhusak another issue I noticed. OrgFile:get_directive_properties() is always empty for me. To make sure I understand, the expectation is that the above function and get_directive_property work with #+NAME: VALUE, right?
:PROPERTIES:
:ID: 3057DD95-697D-4032-90E3-791F95D3B2EC
:END:
#+TITLE: manpage
* some manual documentation
---@type OrgFile
local file -- assume this is set to the above content
-- Should print "manpage", but is "nil" instead
print(vim.inspect(file:get_directive_property("title")))
-- Should print "{'title' = 'manpage'}", but is "{}" instead
print(vim.inspect(file:get_directive_properties()))Unrelated to the two separate challenges I'm facing above, I was also trying to figure out a way to change or expand part of the template target at the point of refile but before the refile happens.
The use case is that org roam has a special expansion called ${slug} that it uses for its target, which itself calls the function org-roam-node-slug to read the title of the node and clean it up for part of a file name.
So what I'd do to offer that is let it be an expansion that I'd fill in as part of the template's target, but only do so once the OrgFile is ready to be refiled so I can grab the title from it.
Pipeline
- Create templates for org roam
- Create
OrgCaptureusing templates - Fill in org roam specific expansions using
on_compile(and on each template's target) - Register an
on_pre_refilecallback to modify the template a bit more (expand target) - Register an
on_post_refile(calledon_closetoday) callback to trigger a reload for my database andOrgFiles
To make sure I understand, the expectation is that the above function and get_directive_property work with #+NAME: VALUE, right?
No, that one is use to get values from #+property directive. So for example for#+property: header-args :tangle no, if you do get_directive_property('header-args') it gives you :tangle no`.
For what you need i exposed get_directive(name) method, so now you can do file:get_directive('title') to get #+TITLE value.
Regarding the capture on_pre/on_post, I added that in 050ed17 commit instead of on_close.
For file loading it's a bit tricky. To load all new files you need to call OrgFiles:load(true) as you did, but the problem is in another place.
All mappings rely on the OrgFiles created in orgmode. So you create your loader, but mapping still uses the orgmode loader, and is unable to find any of your files. I solved this for capture with some custom mappings, but for orgmode mappings I don't want to do that, there's too many of them.
I'll try to think of some solution. Only thing that I have in mind now is to do this:
- Keep a track of all file loaders in
orgmode. For example, by default it would be something like this:
local file_loaders = {
orgmode = OrgFiles:new({ paths = org_agenda_files })
}- Expose a method called
add_file_loaderwhich you would call to append to above loaders
orgmode.add_file_loader('org-roam', OrgFiles:new({ paths = org_roam_files }))- In the buffer, keep track which loader to be used via buffer variable. By default it would be
vim.b.org_file_loader = 'orgmode' - For org-roam, you would override this value and set it to
vim.b.org_file_loader = 'org-roam'( or maybe that can be done automatically through the file loader, I would need to check) - Have all calls figure out the loader from the buffer variable.
I'm still not sure if this idea would work, but I would need some to give it a try. Let me know if you have some ideas how could we have different file loaders depending on the file.
@kristijanhusak I went back and edited one of my comments to include an example of how I'm tackling the mapping today. What are your thoughts on the injection method where I wrap the OrgMappings:open_at_point for the specific require("orgmode").org_mappings instance?
So far, it works pretty cleanly, and would avoid any extra work on your side.
local function modify_orgmode_plugin()
-- Provide a wrapper around `open_at_point` from orgmode mappings so we can
-- attempt to jump to an id referenced by our database first, and then fall
-- back to orgmode's id handling second.
--
-- This is needed as the default `open_at_point` only respects orgmode's
-- files list and not org-roam's files list; so, we need to manually intercept!
---@diagnostic disable-next-line:duplicate-set-field
require("orgmode").org_mappings.open_at_point = function(self)
local link = require("orgmode.org.hyperlinks.link").at_pos(
vim.fn.getline("."),
vim.fn.col(".") or 0
)
local id = link and link.url:get_id()
local node = id and require("org-roam.database"):get_sync(id)
-- If we found a node, open the file at the start of the node
if node then
vim.cmd.edit(node.file)
return
end
-- Fall back to the default implementation
return require("orgmode.org.mappings").open_at_point(self)
end
endAnother thing I'm tackling, and I think I might just have an approach, is applying highlighting and conceals for org filetype against a specific region of a non-org filetype buffer. This is to address the lack of highlighting in the previews of the org-roam buffer that we were tackling earlier. The only limitation I have is that I can only grab the pre-existing extmarks if I disable _ephemeral within require("orgmode").highlighter, which I'm doing today to test things out, but is a private field.
The idea is that I create a scratch buffer, render the subset of text in the buffer, grab the extmarks from that buffer, and apply them to the original range in the org-roam buffer by adjusting the line positions. I know this may lose some context when highlighting.
In terms of your idea of having an API to segment files used and configure the buffer to point to which set it wants to leverage for mappings, I think that sounds fine for a more reliable plugin hook. I'm not sure when the time to configure the buffer variable would be, though. Would plugins need to hook into entering a buffer or when it's first created to figure this out?
My first thought is to use OrgFiles paths to determine which of the loaders the buffer belongs by seeing if its filename is located within an evaluated glob.
The challenge is if more than one loader fits the location of the file, as is my situation where I have orgmode configured to use ~/orgfiles and org-roam configured to use ~/orgfiles/roam. The easiest path is to say that this is unsupported and have an error occur if one set of OrgFiles could be contained in another set. I know that Emacs' org-roam doesn't recommend having the roam directory within the orgfiles directory.
However, I quite like being able to place my roam files mixed in with my orgfiles, and I'd ideally want to support having them mixed. With my current function injection, this actually works (to my knowledge) without any major issues. So if there was a more formalized register for OrgFiles that you've specified, I'd want something to specify order, or prefer the non-standard file set first, or try each loader in turn until one works.
Also, here's an example of the highlighting in action (manually calling function) to refresh on the org-roam buffer:
Example-of-highlighting.mp4
The approach you are using for open_at_point works, but you would need to do that for all the mappings where it relies on loader. I'm not sure how many are there though. Lets see how it goes.
Regarding the highlighting, the highlighter that you are using adds highlights only for some things, like emphasis (bold, italic), links dates, some latex. All other things would not be highlighted. We should figure out a way to somehow inject the real tree-sitter highlighter in that place.
I just merged a branch where dependency on nvim-treesitter was removed. I know you had some failed attempts to use org highlighting in backlinks, but maybe you can give it a 2nd try. Since backlinks buffer is somewhat limited (from what I understand), you could try creating it and doing few things manually that are now done here in ftplugin/org.lua. For example:
_G.test_org_buffer = function()
local buf = vim.api.nvim_create_buf(false, true)
vim.api.nvim_buf_set_option(buf, 'filetype', 'org')
-- You can maybe use some naming convention like this, but not necessary
vim.api.nvim_buf_set_name(buf, 'org:///org-roam-backlinks.org')
vim.api.nvim_buf_set_lines(buf, 0, -1, false, { '* level 1', '** level 2', '** level 2.2' })
vim.cmd('b'..buf)
-- stuff from ftplugin/org.lua
vim.treesitter.start()
vim.opt_local.foldmethod = 'expr'
vim.opt_local.foldexpr = 'v:lua.require("orgmode.org.fold").foldexpr()'
vim.opt_local.fillchars:append('fold: ')
vim.opt_local.foldlevel = 0
vim.cmd('norm!zX')
-- 0.10 only
vim.opt_local.foldtext = ''
endYou probably don't need everything from ftplugin, only things to make it work. I don't know if this would work for you but I think it's worth a try.
Edit: These probably cause the same issues you had previously with different buffers loaded, but some of these can be ran with specific buf number, like vim.treesitter.start(bufnr, 'org')
The approach you are using for open_at_point works, but you would need to do that for all the mappings where it relies on loader. I'm not sure how many are there though. Lets see how it goes.
As I'm not an org expert since I didn't come from Emacs, there may be features I'm not aware of that integrate between org-roam and org when it comes to ids. At the moment, the only need I have is being able to navigate id links, which this solves.
So if there's nothing else and you're fine with the org-roam plugin wrapping that mapping, then this should work fine.
Regarding the highlighting, the highlighter that you are using adds highlights only for some things, like emphasis (bold, italic), links dates, some latex. All other things would not be highlighted. We should figure out a way to somehow inject the real tree-sitter highlighter in that place.
I just implemented the logic into the main branch that does the highlighting as I demonstrated. There are definitely issues that I can't overcome with this approach such as flickering when you first expand a preview, but caching the preview lines (using vim.fn.sha256) and saving extmarks if the lines haven't changed seems to work well enough.
As you said, this doesn't cover all highlighting. The previews are limited to a subset of orgmode elements being extracted, but obviously having full syntax highlighting would be great.
I just merged a branch where dependency on nvim-treesitter was removed. I know you had some failed attempts to use org highlighting in backlinks, but maybe you can give it a 2nd try.
I'm hesitant to try to the org syntax approach again for two reasons: performance and control.
-
Each preview under a backlink needs to be loaded from disk or at least sourced from
OrgFiles, and if there are a lot of backlinks this could be an issue. With the current approach, I only load the preview when you first expand the link. I don't think I can do that with the org filetype. -
When I was using the org filetype, it was easy to change buffers in the same window, open links within the same buffer but not a different buffer, and other problems. While I could remap over the bulk of orgmode bindings for that newly-created buffer, it would be error prone and difficult to keep up-to-date with your plugin. With this current approach, you hit
<Enter>to open in another window and<Tab>to expand/collapse a preview, which is all I really need.We talked about updating open_at_point to support pointing to another window, which would at least solve that issue.
So, I may look into supporting an org buffer again down the line, but there's so much work to get that done and this works well enough for me that I'm hesitant to try right now.
I'm open to trying to get an org buffer to work again, but it'll be a couple of weeks before I'm able to get to something like that.
Preview of implementation in action. The preview highlighting can also be disabled to prevent the flickering if it becomes a problem.
preview-impl.mp4
@kristijanhusak I refactored to use the new on_pre_refile and on_post_refile. It looks like the on_pre_refile may happen too late? My target is %<%Y%m%d%H%M%S>-%[slug].org. I wrote some new expansions to overwrite the target of the template, but they don't get applied before this prompt:
I moved it to be earlier in a0f16dd so it processes these stuff before collecting the destination info.
I moved it to be earlier in a0f16dd so it processes these stuff before collecting the destination info.
Perfect. That fixed my issue with the slug.
At this point, I think I'm about done, except I've got to figure out the performance issue around capturing and needing a full OrgFiles:load(true). I might have to write my own handler for retrieving filenames that includes all files from all_files and see what else needs to be done to use just OrgFiles:load_file. Right now, it's pretty slow, and results in a big delay in capture, insert, and finding of nodes when there are a lot (testing with 500+ from that braindump repo).
The slowness is a mixture of reloading files and extracting out the needed information for the org-roam database.
Thanks again for all of your help on this 😄 I'll be taking screenshots and video clips to put on the README and then it'll be ready to share around.
From my tests initial load of braindump repo loads in 100ms. Doing again with load(true) is a bit faster, but not much.
In dafe433 I added add_to_paths(filename) method to OrgFiles, so you can use that to append a specific file to a path. There's also sync version add_to_paths_sync.
From my tests initial load of
braindumprepo loads in 100ms. Doing again withload(true)is a bit faster, but not much.
In dafe433 I addedadd_to_paths(filename)method toOrgFiles, so you can use that to append a specific file to a path. There's also sync versionadd_to_paths_sync.
I'll give it a try, thanks! Pretty sure this is one of the inefficiencies on my end such as that link offset conversion I mentioned.
Once I get that tackled, I've got a minimum viable product ready.
Some minor annotation comments
While I'm thinking of them, here are a couple of Lua language server annotation issues I've encountered while interfacing with the plugin:
- It looks like
OrgPromise<T>'snext()function is missing a defined return type of anotherOrgPromise<V>: ], which leads to the language server complaining when chaining/returning:orgmode/lua/orgmode/utils/promise.lua
Line 33 in dafe433
- Other methods for
OrgPromise<T>don't show up in the language server, presumably because it gets confused with the parent object literal type. I guess the solution here would be to have a massive object literal with all of the methods re-defined. Most common missing method for me iscatch. OrgFiles:new()is missing a return type annotation and defaults totablefor the language server:orgmode/lua/orgmode/files/init.lua
Lines 19 to 20 in dafe433
OrgCaptureTemplates:new()is missing a parameter and return type annotation and defaults totablefor the return:orgmode/lua/orgmode/capture/templates.lua
Lines 6 to 10 in dafe433
- For some reason,
OrgFile:new()doesn't always resolve toOrgFileas the return for me. I think this is some quirk with my setup, though. Sometimes it has no return and I have to mark it with@type OrgFile.
3, 4, and 5. can be easily fixed, but I don't know what to do with 1 and 2. Generics support is very lacking, this is the best I could figure out to get at least some typings. Do you have an idea how could I solve it?
3, 4, and 5. can be easily fixed, but I don't know what to do with 1 and 2. Generics support is very lacking, this is the best I could figure out to get at least some typings. Do you have an idea how could I solve it?
1 should be straightforward, changing to include the return value, right? Or does that not work?
next: fun(self: OrgPromise<T>, resolve:fun(result:T):V):OrgPromise<V>2 is difficult like you said since generics are very limited. I've done the hack that you've done, and it's the only thing that works. The only way to make it cover all of the methods to my knowledge is to just grow that object literal used as the parent of OrgPromise<T> to include the other methods. Just makes it a veeeerrrryyyy long line.
1 should be straightforward, changing to include the return value, right? Or does that not work?
Unfortunately, it does not work. It starts to split the return value to be either 1 or 2 values, which messes up a lot of things.
For 2, I added catch to the types for now. We can see later for other ones.
@kristijanhusak not sure what you mean by splitting into 1 or 2 values. I tried it out myself, and the only thing I had to do was wrap the entire return value in parentheses:
next: (fun(self: OrgPromise<T>, resolve:fun(result:T):V):OrgPromise<V>)
I've encountered issues where wrapping the function type or other type in an extra set of parentheses works. And in the rare occasion I've actually had to reorder the fields of the object literal because one of the fields was causing issues with the others. Does the above help at all? At least for Lua language server 3.7.4 this works to get next() to return the promise and I can still use wait().