Listen for when a resource changes
Thom1729 opened this issue · comments
Problem description
It's common for packages to want to take some kind of action when a resource changes. There is no simple, reliable way to do this. (Several workarounds are listed under Alternatives.)
Example use cases:
- When a Python file changes, reload the package. (AutomaticPackageReloader does this, but the implementation has bugs and limitations.)
- When a JS Custom syntax extension is modified, rebuild all user syntaxes that use that extension.
- Perform any expensive operation on the contents of a resource, caching the result until the resource changes. (For example, parsing a syntax definition with
ruamel.yaml.)
Preferred solution
A new method EventListener.on_resource_changed_async(name), to be invoked whenever:
- A resource is created, deleted, modified, or moved.
- A package is ignored or un-ignored.
- A resource becomes overridden, or an override is removed.
(I presume that Sublime already monitors the resource directories. That is, all of the work of monitoring the directories across three platforms is already done, so this feature is “merely” to notify the user code. If my assumption is incorrect, then please let me know.)
Example:
class MyListener(sublime_plugin.EventListener):
def on_resource_changed_async(self, name):
if name.startswith('Packages/MyPlugin/')
pass # do somethingPackage operations (such as (un)installing, (un)ignoring) may still cause a large number of events. Instead of calling the listener with the name of a single resource, it may make more sense to call the listener with an iterable of names. (As appropriate, the event could be debounced in addition to this.)
class MyListener(sublime_plugin.EventListener):
def on_resources_changed_async(self, names):
for name in names:
if name.startswith('Packages/MyPlugin/')
pass # do somethingSome events will inevitably be spurious. For instance, most applications won't care if a resource is overridden by an identical copy. But it's probably better to leave this to the listener code to sort out rather than to try to do something clever.
Alternatives
An explicit subscription (like Settings.add_on_change()) could work as well. The downside is that the user would have to manually manage the subscription, e.g. remembering to remove it in plugin_unloaded. (Sublime automatically manages EventListener instances, eliminating this failure mode.) An explicit method could also take a parameter describing which resources to monitor, but it would be more flexible for the user code to handle this directly.
There are several workarounds that packages might use instead of an event:
on_post_save_async
Packages can run a callback when the user manually saves a file. This may work well enough most of the time, but it will fail if a resource is modified by other means, such as Dropbox sync or a build system.
It may also fail if a resource is in a symlinked directory — the user would have to add code to manually check whether the directory is a symlink and get its resource name from there. This is rarely done in practice, and doing it correctly is surprisingly difficult and potentially inefficient. (AutomaticPackageReloader tries to do this, but it's currently buggy.)
Manual modification checks
Rather than waiting for a notification, a package can manually check to see whether a file has been modified. This can be accomplished by loading the entire resource and comparing it to a previous version (or a hash). This has several drawbacks. First, you have to read the entire resource and either compare it byte-by-byte or hash it. Second, in order to have a baseline for comparison you would need to pre-load all of the relevant resources, which could cause problems if many exist but few are expected to change. Third, in order to use this to monitor for changes (as opposed to checking manually as needed), you would have to poll the relevant files at intervals.
Checking modificaton time
As above, but instead of reading the file, check the file modification time. For unpacked resources, this is a filesystem operation; for packed resources, you can check the ZipInfo.date_time. It isn't necessary to read in the entire resource or pre-load resources for comparison, but there are several new brand-drawbacks. For one thing, you have to keep track of every physical file location and figure out the override logic yourself. For another, file modification times are inherently unreliable (for one thing, it's not necessarily monotonic) and ZipInfo.date_time even more so.
inotify (and company)
A package could use inotify to monitor the resource directories. This is in some sense the “correct” way to do it. The package would have to use alternative implementations on OS X and Windows. Python packages to do this are available, but not (yet) as a dependency. Such a dependency would also have to reimplement the resource override logic — in effect, it would be doing most or all of the work that Sublime does to monitor resources (taking up its own set of resources to boot). It is unlikely that such an implementation would behave identically to Sublime's, though it might be good enough for most practical purposes. Another problem is that while this would remove the need for polling, I don't think there's any way to guarantee whether the package or Sublime itself would receive the first notification, which raises the specter of race conditions (especially if it were usually Sublime).
One use case: sublimelsp/LSP#516
Could you summarize the use case here? It isn't clear to me without reading the LSP spec to know what the requirements would be.
I can't summarize more than it is, but I'll try:
- Servers are allowed to run their own file watching mechanism and not rely on clients to provide file events, however this is not recommended.
- The client has to watch the requested files for creation, deletion and changes:
/**
* The file event type.
*/
export namespace FileChangeType {
/**
* The file got created.
*/
export const Created = 1;
/**
* The file got changed.
*/
export const Changed = 2;
/**
* The file got deleted.
*/
export const Deleted = 3;
}- The language server may register for certain change events only:
export namespace WatchKind {
/**
* Interested in create events.
*/
export const Create = 1;
/**
* Interested in change events
*/
export const Change = 2;
/**
* Interested in delete events
*/
export const Delete = 4;
}It is worth checking the workspace/didChangeWatchedFiles section of the spec for more details.
Also useful for the workspace/didChangeWorkspaceFolder notification, where the client would watch for the addition or removal of folders.
v0.9.3 of LSP implemented this in sublimelsp/LSP@96e2897.
The proposed on_resources_changed_async takes a bunch of strings but it's important to get the type of event too.
I propose it should receive a list of these small objects:
class FileSystemEvent:
__slots__ = ('kind', 'path', 'prev_path')
def __init__(self, kind: int, path: str, prev_path: Optional[str] = None) -> None:
self.kind = kind # sublime.FS_CREATED / FS_DELETED / FS_MODIFIED / FS_RENAMED
self.path = path
self.prev_path = prev_path # in case of FS_RENAMED, the old path before the rename
Reading first message, it pretty much describes use case of watching package resources while for LSP it's rather about watching workspace folders instead.
Just wanted to clarify, since potential implementation would most likely handle both cases anyway.
The two use cases are similar, but there are some important differences.
- When a resource changes, you want its resource path (
Packages/…), not its file path. - If an installed package is updated, then only a single file would change (the
.sublime-package), but many resources might. - Resource monitoring should respect overrides. For instance, if a file in an installed package is modified, but you have a custom override for that file that doesn't change, then the resource hasn't changed and there's no need for an event.
- I'm pretty sure that Sublime is already monitoring resource directories, so listening for events would not consume additional OS-level resources. I'm not sure if Sublime is monitoring workspace directories in the same way.
I'm not sure if Sublime is monitoring workspace directories in the same way.
It has to, otherwise it cannot update the side bar or it wouldn't ask the user to reload the view once the underlying file was changed due to an external FS event.
We watch both packages and workspace directories, and there are a lot of edge cases we handle that would take time to get right in an external implementation.
I've discussed this with Jon before, and the blocker has always been the performance implication of serializing, potentially, thousands of FS events (specifically paths) over the plugin_host SHM segment, and the performance of Python finding things that are important to it, and doing something about it.
My understanding is we'd have to try and come up with some way to deal with things like switching branches in Git repositories in a sane way such that it didn't bog the plugin interface down.
I suppose that could be mitigated via subscriptions so that plugins explicitly describe notifications they need (e.g. In X workspace, ending in .js, excluding node_modules). Of course, that's another added layer of complexity, and there's still nothing stopping someone from subscribing to the universe.
I've also thought about collecting FS events, and then batching then into tree structures to try and reduce the raw amount of data being sent.
Note that we have an issue about files in the project/workspace at #626.
Some practical examples with LSP servers:
- vetur (vue language server) wants to watch workspace patterns:
*.js, *.ts - typescript server the same
- eslint server:
**/.eslintr{c.js,c.yaml,c.yml,c,c.json}, **/.eslintignore, **/package.json
All of those don't really want to watch node_modules/ and .git directories either.
So in those cases, a "subscription" model appears better suited and likely more performant.
Note that we have an issue about files in the project/workspace at #626.
Sorry, but I'll keep discussing it here since I think it wouldn't make sense for developers to handle those separately. Both should be taken into consideration when designing the system and likely both would have to be implemented simultaneously.
👋 just wanted to chime in with yet another example, also LSP related.
The official language server for Go is gopls. Today I stumbled upon a problem related to adding new files into a package namespace.
Let's say I have the following directory tree:
.
└── main.go
0 directories, 1 filesThe main.go file contains the following code:
package main
import "fmt"
func main() {
cards := []string{"Ace of Spades", "Five of Diamonds"}
}I introduce a new file, deck.go in the same directory:
.
├── deck.go
└── main.go
0 directories, 2 filesThe deck.go file contains the following code:
package main
type deck []stringIf I now try to swap
cards := []string{"Ace of Spades", "Five of Diamonds"}`in main.go for:
cards := deck{"Ace of Spades", "Five of Diamonds"}`it will error out, saying deck is an undeclared name. This is due to the fact that LSP can't support workspace/didChangeWatchedFiles.
The usage of workspace/didChangeWatchedFiles in gopls is documented here and it ended up in master in January last year. See golang/go#31553 (comment).
This is a fairly big roadblock in the usefulness of gopls when used in Sublime Text as any new file introduced (that introduce new types), requires a restart of the language server itself.
This is a fairly big roadblock in the usefulness of
goplswhen used in Sublime Text as any new file introduced (that introduce new types), requires a restart of the language server itself.
This exact scenario also happens when using the Python language server with sublime-lsp, Pyright. Any time I create a new python file and want to import anything from it in other files, I have to restart the language server manually. It's pretty annoying.
FYI: there is interim solution for LSP: https://github.com/sublimelsp/LSP-file-watcher-chokidar