eikek / dlm

download manager for the command line

dlm - a download manager

What and why

Dlm is a basic download manager for the command line: a wrapper around tools like curl or youtube-dl. It maintains a sqlite database with metadata of files that have been downloaded with dlm. It is written in Common Lisp on a Linux platform. It might work on other platforms, but it’s not tested.

This metadata includes a sha hash of the file, its length, its location on disk and custom metadata like the url and a lifetime. This lifetime specifies after which time (default is 30 days) dlm can delete the file from disk. The metadata will not be deleted so files can be redownloaded again if needed.

Dlm wants to address the ever growing download folder. But instead of just deleting all old files, dlm keeps track of the metadata, including the source url. This allows to run consitency checks and also to re-download a file that has been deleted already.

Dlm keeps track of the following metadata:

• remote url
• path to the file
• file size
• sha256 checksum of the file
• download time
• keep, a flag indicating to dlm that it should never delete this file
• lifetime: duration after which the file can be deleted
• re-downloads: the number of times this file has been downloaded

Installation

Checkout the code:

git clone https://github.com/eikek/dlm

and build it. There is a prebuild binary for linux can be downloaded here.

Usage

Dlm accepts commands which may take options. You can find out what is available using dlm help and dlm help <command>. If the command is not found, a default command is used, which is (by default) fetch.

The ‘info’ command

Usage:

dlm [--]info [OPTIONS] FILES

Options are:

-r, --raw-values  Print raw db data values instead of human readable form.
-s, --structure   Print information as a lisp data structure.
-h, --help        Prints this summary

Show information about downloaded files.

For a given file the information from the database is displayed in key-value form.

The ‘pending’ command

Usage:

dlm [--]pending [OPTIONS] ACTION

Options are:

-k, --keep       Search files that are specified to not be deleted.
-H, --no-header  Don't print search parameters header.
-s, --source=    Search in the url.
-l, --limit=     Apply the action to the first 'n' items only.
-h, --help       Prints this summary

Show pending downloads.

This queries the pending downloads. All downloads that have not finished yet are listed separately with this command. You can use few search options and actions:

Each search option that takes an argument is compared against the corresponding field. You can use % character as a wildcard (matching many characters).

Actions are then applied to each file in the result. Actions are

• list (the default): print it to stdout
• fetch: resume the download. This can be used if a download has been cancelled. Note that it does not check whether a current process is running regarding this download. It just starts a new one.
• clear: deletes the pending download entry from the db

Action names can be abbreviated to the shortest non-ambiguous name. Only one action can be given at a time.

The ‘query’ command

Usage:

dlm [--]query [OPTIONS] ACTION

Options are:

-k, --keep         Search files that are specified to not be deleted.
-e, --existing     Show only files that exists on disk.
-r, --nonexisting  Show files that do not exist on disk.
-c, --valid        Show files where its metadata (sha256, size, lastmod
                   timestamp) matches the db entry.
-C, --invalid      Show files where its metadata (sha256, size, lastmod
                   timestamp) does not match the db entry.
-H, --no-header    Don't print search parameters header.
-n, --name=        Search in the filename.
-s, --source=      Search in the url.
-l, --limit=       Apply the action to the first 'n' items only.
-h, --help         Prints this summary

Query the database for files that have been downloaded.

Each search option that takes an argument is compared against the corresponding field. You can use % character as a wildcard (matching many characters).

Actions are then applied to each file in the result. Actions are

• list: (the default) print information to stdout. By default output is ansi-colored; this can be suppressed by setting the environment variable DLM_COLOR=0.
• short-list: print only the local filename to stdout
• structure: print a lisp data structure
• fetch: download the file again, if the file exists and the metadata matches the db, it is not downloaded again
• delete: deletes the file on disk, but not the record in the database
• prune: deletes the file and removes the record from the database
• clear: deletes the record from the db but leaves the file on disk
• keep: set the keep flag to true
• nokeep: set the keep flag to false
• move [dir]: moves the file to the given directory
• set-lifetime [secs]: set a new lifetime in seconds or use “2d10M” strings (you can use m,d,h and M)

Action names can be abbreviated to the shortest non-ambiguous name. Only one action can be given at a time. Note that checking file metadata (the -C|c option) involves computing a checksum of the file which may take some time depending on its size.

The ‘collect-garbage’ command

Usage:

dlm [--]collect-garbage [OPTIONS]

Options are:

-s, --silent  Don't print info messages.
-d, --dry     Don't actually perform the action.
-h, --help    Prints this summary

Deletes expired files.

Checks the last access time of each file with expired lifetime in the db and deletes it, if it is older than the specified lifetime for this file.

The ‘fetch’ command

Usage:

dlm [--]fetch [OPTIONS] URLS

Options are:

-k, --keep       Flag the file to never be deleted by dlm
-t, --target=./  The target directory
-l, --lifetime=  The lifetime to set for this file. Default is   1.0m.
-u, --user=      The username to authenticate with.
-p, --pass=      The password to authenticate with.
-h, --help       Prints this summary

Download a file at some url.

This will call other download programs like curl in order to download a file at the given url. If multiple urls are specified they are downloaded sequentially. What download program to use is determined by the url. If, for example, the youtube-dl tool is available, certain urls to video portals are downloaded using this tool. Otherwise curl is the default fallback (unless configured otherwise).

The ‘–user’ and ‘–pass’ options are simply delegated to the real download programs. Thus it depends on whether the program supports these options.

If URL is a path to a local file, it is simply added to the database and the source and location properties are both set to the same path. Those files don’t have a remote source and are treated a little different. First, ‘deleteing’ them causes the db entry to be removed, too. Then, obviously, they cannot be redownloaded and will be skipped if tried.

The ‘help’ command

Usage:

dlm [--]help [OPTIONS] COMMAND

Options are:

-h, --help  Prints this summary

Shows a short help for this program.

If called without arguments, a little help is shown, listing all commands with a short description. If a command is given, a more detailed help to this command is shown (if provided).

Configuring

Dlm reads a configuration file at $HOME/.config/dlm/config.lisp. It is a normal lisp file that is loaded at the beginning. You can specify another configuration file by setting the environment variable DLM_CONFIG to another file.

Settings

A few variables can be set:

The database is usually at $HOME/.config/dlm/dlm.db, which can be overriden with:

(setq *database* "/path/to/a/file")

If you like to change the default command:

(setq *default-command* "query")

The default lifetime of a file is 30 days or you set it (in seconds) via:

(setq *file-lifetime* (* 60 60)) ;; 1 hour, or
(setq *file-lifetime* (parse-duration "5d10h"))

The curl program is used to download files. If it’s not in your path you can set it:

(setq *fetch-default-bin* "/path/to/curl")

The options to curl are stored in *fetch-default-args*, it is -O#C - ~a where ~a is replaced with the url. This string must contain one ~a.

Likewise the youtube-dl and scp program is configured:

(setq *youtube-dl-bin* "/path/to/youtube-dl")
(setq *youtube-dl-args "--option1 ~a")
(setq *scp-bin* "scp")
(setq *scp-args* "~a .")

As with curl the single ~a in the line is replaced with the url.

If the --target options is not specified, the file is downloaded to the current directory. You can specify a folder instead:

(setq *default-target* "/home/eike/Downloads")

For more settings look at the beginning of the cli.lisp and dlm.lisp files.

Extending

customize fetching files

The variable *fetch-configs* is a list of fetch-config objects. A fetch-config contains two functions, where the first is a predicate that given an url returns true if the second function should be used to download the file. This can be used to customize how certain urls are downloaded. For example, the youtube-dl tool is setup using this mechanism.

A config file may look like this:

(defun my-program? (url)
  ;; return true if this url can be downloaded with the function below
  ...)

(defun my-program-download (url &optional user pass)
  ;; download and return the filename
  ...)

(add-fetch-config
  :can-fetch? #'my-program?
  :fetch #'my-program-download)

By default, this list contains a fetch-config for the youtube-dl command, for the scp command and one for curl.

be notified when download is done

The variable *download-notify-hook* can contain a list of functions that are all called with the metadata of a newly downloaded file. A second boolean argument specifies whether this file was downloaded or already existed. You can add functions like for example:

(push (lambda (md existed)
        (when (and (not (getf md :error)) (not existed))
          (external-program:run
           "stumpish"
           `("echo" ,(format nil "Download ~a finished." (getf md :source))))))
       *download-notify-hook*)

The package external-program is available that you can use to call out to other programs. The example raises a notification in the stumpwm windowmanager.

Or you could run garbage collection after each download:

(push (lambda (md existed)
        (declare (ignore md) (ignore existed))
        (dlm-collect-garbage))
      *download-notify-hook*)

custom query actions

The query command applies actions to each item in the result set. An action is a function of two arguments: the metadata plist and the db handle. There are several actions provided, but you can add your own.

You need to create a factory function that, given a number of arguments, creates the action function. The arguments are those given after the action on the command line. For example, the move action needs a target directory where to move each file to:

dlm query move /new/path

This last argument is passed to the factory function which then creates the action function to use for each item in the result.

Add this factory function to the variable *query-actions-alist*.

(push (lambda (&rest args)
        ...)
      *query-actions-alist*)

To ease this a little, the function make-action-fn can be used. First define your custom action function by declaring all extra arguments at the beginning and the metadata and db argument at last and then use make-action-fn to create the factory function.

(defun my-custom-action (arg1 arg2 md db)
   ...)

(push `("myaction" . ,(make-action-fn #'my-custom-action))
      *query-actions-alist*)

The make-action-fn function creates a factory function that curries my-custom-action with its given arguments – which are two. This creates another function that only takes the remaining two arguments. If there are no arguments given to the factory, it just returns the my-custom-action function. For this to work reliably the factory function must always be called with exactly two arguments, such that the currying leaves us with a correct action function. How to achieve this is described below. Other cases must use a custom factory function that checks the argument list itself.

To allow the user of the custom command to specify arguments in the first place, you need to add a hint to *query-actions-argn* alist defining how many the action expects:

(push '("myaction" . 2)
      *query-actions-argn*)

This tells dlm to check for 2 arguments and it prints an error message if there are not exactly two specified on the command line. If there is no entry (or nil) in *query-actions-argn* for some action, it is assumed to not take any arguments. Instead of a plain number, you can specify a function that takes the argument list and would return nil to signal an error. With the example above in place, a call to the custom action would look like this:

dlm query myaction arg1 arg2

The same is used for actions to the pending command. Just use the variables *pending-actions-alist* and *pending-actions-args* instead.

Building

https://travis-ci.org/eikek/dlm.svg

Using Nix

There are some lisp packages needed, which must be installed. Using the nix package manager, this happens automatically:

nix-build build

This creates an fresh environment with the required packages and sbcl installed to build dlm. The resulting executable can be found following the newly created result link.

To build it manually but use the provided environment from nix, you can run:

nix-shell --pure build

This drops you in the same shell that nix-build uses when building. Thus you can cd into the build directory and run make to build the executable and make test to run the tests.

Manually

See the dlm.asd file for the dependencies. It should be fiveam, sqlite, ironclad, external-program, unix-options and cl-ansi-text. These must be installed (fiveam only for running the tests). Then you can use the build/build.lisp lisp file. Load it and call for example (make-image) to create the executable.

All dependencies can be installed via quicklisp. Install it and execute:

$ sbcl --eval '(ql:quickload (quote (:unix-options \
                     :ironclad :external-program \
                     :sqlite :cl-ansi-text :fiveam)))'

Browser integration

Conkeror

I use the following in my .conkerorrc:

function dlm_fetch (url, open) {
    var cmd = "dlm fetch '" + url + "'";
    if (open) {
        cmd = cmd + " && xdg-open $(dlm query -Hs '" + url + "' short)";
    }
    shell_command_blind(cmd);
};

function dlm_download (open) {
    return function(I) {
        var mb = I.window.minibuffer;
        bo = yield read_browser_object(I);
        link = load_spec_uri_string(load_spec(bo));
        mb.message("Downloading " + link);
        dlm_fetch(link, open);
    };
}

interactive("dlm-download",
    "Download the url using dlm.",
    alternates(dlm_download(true), dlm_download(false)),
    $browser_object = browser_object_links);

define_key(content_buffer_normal_keymap, "C-d", "dlm-download");

Instead of s I now press C-d to download and open the file and C-u C-d to download without opening afterwards.

Firefox

Use the addon flashgot that lets you easily add a custom download manager.

Chromium

Feedback

… is always most welcome. You can use email {firstname.lastname}@posteo.de, the issue tracker or pull requests.

License

Copyrighted by me 2015-, distributed under GPLv3 or later.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with GNU Emacs; see the file COPYING. If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.