Import System Proposal

[Armin Ronacher]

Terms

• module: a single .q file (nicknamed scrolls)
• book: metadata + a bunch of .q files

Filesystem Layout

README          an optional readme or other data
book.toml       the book's metadata (toml because of comments)
src/            all sources to here
  index.q       the entrypoint file.  If you import `my_book` it loads this
  submodule/    an example submodule
    index.q     with more code here
lib-native/     holds native dependencies for the ffi
  myobject-Darwin-10_9_intel
  myobject-manylinux1_x86
  myobject-manylinux1_x86_64
dependencies/
  other_book    this holds references (dependencies) to other books.

Dependencies are not defined in the book.toml but instead in the
dependencies folder. Each dependency there can either be directly placed
there, as a symlink or as a toml file.

For instance to depend on a book named curl this would work:

version = "1.x"

Or local references:

path = "/path/to/some/book"

Dependency Resolution

Books are combined in a process called "collation" into a graph. Each
book can import other books that it declared as dependencies through the
import system. If it did not declare a dependency the import will fail
even if it was reachable otherwise.

Because module scopes are immutable items can be collapsed into a common
namespace. For instance a book my_book can ship a structure named
com/example/mybook/index.js and then other code would import it from
com.example.mybook. To avoid having too many nested folders the main
book.toml file can declare a common prefix that is added in front of
all code in src.

There are two dependencies in Quill: public and private dependencies. The
default is a public dependency which means that it becomes part of the API
of a library. Dependencies can also be set private to be contained within
the library. In that case one has to be careful with throwing and
catching exceptions when they cross book boundaries.

The book names do not show up in import paths but each module knows which
book it comes from through the global __book__ constant. This one is
also used by the import system to refer to dependencies.

Importing

All the books in a book's dependencies are considered for searching. By
default the "standard books" are added automatically. So all items in the
standard library can be loaded. The books of the standard library however
can be individually versioned and individually distributed if necessary.

Import syntax:

use std.io							# adds a binding named "io"
use std.io.BufReader				# adds a binding named BufReader
use std.io.{BufReader, BufWriter}	# adds two bindings (BufReader and BufWriter)

When a book imports from std.io then all the book references in the
dependencies folder are considered for this. This means that book B could
import a different verison of std.io from book B if they have different
dependencies set up. This should be the exception not the rule however.
By default packages are required to set up broad versions (eg: depends on
std-io 1.x) and the latest version across all books is used. This
means that all books using std-io 1.x as pin will get the same version
std-io 2.x.

A library can also depend on both std-io 1.x and std-io 2.x in which
case the book needs to be given a different name to keep them apart:

dependencies/
  stdio1:
    book = "std-io"
    version = "1.x"
  stdio2:
    book = "std-io"
    version = "2.x"

Then the import can explicitly declare which books it wants to import
from:

@import_spec(books=['stdio1']) use std.io as iov1
@import_spec(books=['stdio2']) use std.io as iov2

Living in a Multi Library World

Because in this world different versions of a module can live together
a type has a fully qualified name that defines it. The fully qualified
ID there is in this form:

{book@version}dotted.path

So for instance BufReader imported from std.io might be fully
qualified like this: {std-io@1.0.0}std.io.BufReader. Fully qualified
names are generally shown in stacktraces for exceptions:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
{std-io@1.0.0}FileNotFound: No such file or directory: '/tmp/asdfasdfsa'

This is to ensure that users are not confused if they fail to catch
an exception thrown across version boundaries.

Forcing Versions

A book can downwards enforce versions of a book in the dependency chain.
For instance it can place a stdio1-forced file into its dependencies
folder with contents like this:

book = 'std-io'
version = "1.0.1"
force_for = "1.x"

This forces that for all pins downwards from the current book all
users of std-io at 1.x will use the 1.0.1 version. This should only
ever be used toplevel or for private dependencies.

Accessing Metadata

At any point a module can refer to __book__ to refer to an object
that gives access to the current book. The following API is
available to work with it:

• __book__.name: Returns the name of the book
• __book__.version: Returns the version of the book
• __book__.dependencies: Allows one to inspect the associated dependencies of the book
• __book__.open_resource: returns a binary reader to a dependency
• __book__.metadata: the full parsed metadata of the book (book.toml)

[Armin Ronacher]

Before books can be loaded all the sources within a book are assembled. That process scans through all stuff reachable from src/ and performs necessary ast transformations. The end result is then cached and that cache is loaded by the actual importers. (What's the cache? .quillcache next to src? Folder / zip archive?)

Upon loading the __book__ is assembled from that cache and modules are created to refer to modules within the cache.

Abstract loader API:

# this looks for path/to/book/.quillcache and if it does not exist, compiles
# path/to/book/src into the cache based on the information in
# path/to/book/book.toml
var book = loader.load_book('path/to/book')
var module = book.load_module('std.io')