Giters

weirdNox / org-noter

Emacs document annotator, using Org-mode

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Enhancement] Expand scope so that it can take notes from video and audio, both online and offline

auroranil opened this issue · comments

commented

I had this idea where you can use org-noter to make notes from videos and audios played by a media player. It would be able to create notes and extract current timestamp of media playing in another window. I am considering making this feature, but it would take me weeks to get to speed with Emacs Lisp, and was wondering if someone else have any ideas in making this possible.

Online media content

For online media content, we could consider interacting with HTMLMediaElement, and use these particular properties/methods/events:

  • HTMLMediaElement.currentTime property
  • HTMLMediaElement.paused property
  • HTMLMediaElement.fastSeek() method
  • HTMLMediaElement.pause() method
  • HTMLMediaElement.play() method
  • timeupdate event

A browser extension would probably be needed so that we can interact with it from outside. I can think of two approaches which I can use for passing messages b/w Emacs and browser:

  1. Native messaging
  2. WebSockets

Offline media content

For offline media content, we could consider using an existing media player such as VLC

  1. via dbus. VLC conforms to MPRIS specification.
  2. via RC interface: https://github.com/mohkale/vlc-rc

Commands

Org Noter commands

We can utilise these following existing commands:

  • org-noter opens media player at timestamp where user viewed video/audio last
  • org-noter-insert-note inserts heading including timestamp in
    :PROPERTIES: drawer, down to seconds resolution
  • org-noter-insert-precise-note inserts heading including timestamp
    but in milliseconds resolution
  • org-noter-sync-current-note jumps to timestamp at current heading
  • org-noter-sync-next-note jumps to timestamp at next heading
  • org-noter-sync-previous-note jumps to timestamp at previous heading
  • org-noter-kill-session kills session, which also closes media player

All other commands are unsupported for this MVP.

Media player commands

Additional commands are needed for interacting with media player within Emacs. I can think of a few so far:

  • org-noter-media-jump-to-timestamp Prompts user to type in timestamp to jump to
  • org-noter-media-toggle-play toggles play/pause media
  • seek back or forward slightly

Does this software already exist?

To my knowledge, there is no org note taker for videos/audios. This section from "How to annotate literally everything" article says that no such note taking software exists.

These links are relevant and may be able to get this feature working:

commented

Example

* Big Buck Bunny - Blender Short Film - YouTube video
  :PROPERTIES:
    :NOTER_LINK: https://www.youtube.com/watch?v=aqz-KE-bpKQ
    :NOTER_CSS_SELECTOR: .video-stream
    :NOTER_TIMESTAMP: 2:40
  :END:
** Bunny picks up fallen apple
   :PROPERTIES:
     :NOTER_TIMESTAMP: 1:28
   :END:
** Credits
   :PROPERTIES:
     :NOTER_TIMESTAMP: 8:15
   :END:
commented

I had almost the same idea last week, though I'd focus on offline video

I'd use the mpv player with some modification, so you can press a keybind to cature the position/area of a video and annotate it in emacs

Since mpv is easily scriptable, this should be the easiest way.

I guess as you can open web videos in mpv as well, online functionality comes for free as well :)

commented

MPV sounds like the best media player to use for this use case.

From #82 (comment), I don't know about hooks, but it seems like if I can use these hooks to add support for video and audio files, then I can build on top of it and create a new repo called org-noter-media which will incorporate ideas I have above as well as a synthesis of ideas coming from alternative packages. If this is not a good approach (like what the author said regarding content that does not have pages), then a completely separate package will be better, and this issue will be closed.

Hooks can be found here:

org-noter/org-noter.el

Lines 207 to 210 in 9ead81d

(defcustom org-noter-insert-heading-hook nil
"Hook being run after inserting a new heading."
:group 'org-noter
:type 'hook)

org-noter/org-noter.el

Lines 226 to 271 in 9ead81d

;; --------------------------------------------------------------------------------
;; NOTE(nox): Integration with other packages
(defcustom org-noter--check-location-property-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--parse-location-property-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--pretty-print-location-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--convert-to-location-cons-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--doc-goto-location-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--note-after-tipping-point-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--relative-position-to-view-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--get-precise-info-hook nil
"TODO"
:group 'org-noter
:type 'hook)
(defcustom org-noter--doc-approx-location-hook nil
"TODO"
:group 'org-noter
:type 'hook)

Hooks are being used from org-pdftools package: https://github.com/fuxialexander/org-pdftools

https://github.com/fuxialexander/org-pdftools/blob/a5b61bca3f8c91b0859bb0df1a929f9a31a57b99/org-noter-pdftools.el#L264-L274

commented

I'd like to inform everyone that I implemented a working prototype which depends on org-media-note and mpv.el borrowing from @auroranil's idea here. For it to work, I have to modify org-noter to accept a link as a document source so it only works with my fork of org-noter, org-noter-plus-djvu.

For how to install read its README here.

commented

May be a good idea to migrate this to https://github.com/org-noter/org-noter

commented

I have no plans for this project. Feel free to fork this project and make changes to it.