Improve async observability
Enter-tainer opened this issue · comments
We receive several report of typst preview stopping reacting to user inputs and hang. These bugs are very hard to reproduce and investigate. Even we reproduce it, it is still hard to figure out why because the bracktrace doesn't provide useful info about tokio tasks.
I suggest we adopt https://github.com/risingwavelabs/await-tree, and provide a command in vscode to trigger a async tree dump. This command should send a special ws message to typst-preview, and it send the await tree back. We may also try dump await tree periodically to stderr in a separate thread, so even if all tokio workers are blocked, we can still get useful info.
dump await tree periodically to stderr doesn't make sense imo, since when all tokio workers are blocked, the await tree should almost differ with normal one. If we want to introspect it when there've some error already happened in preview service, We can start a health checking thread that doesn't relate to tokio. A health checking service should also create a separated socket with control panel, so to isolate with ill threads.