Import System Proposal
mitsuhiko opened this issue · comments
Terms
- module: a single .q file (nicknamed scrolls)
- book: metadata + a bunch of .q files
Filesystem Layout
README an optional readme or other data
book.toml the book's metadata (toml because of comments)
src/ all sources to here
index.q the entrypoint file. If you import `my_book` it loads this
submodule/ an example submodule
index.q with more code here
lib-native/ holds native dependencies for the ffi
myobject-Darwin-10_9_intel
myobject-manylinux1_x86
myobject-manylinux1_x86_64
dependencies/
other_book this holds references (dependencies) to other books.
Dependencies are not defined in the book.toml but instead in the
dependencies folder. Each dependency there can either be directly placed
there, as a symlink or as a toml file.
For instance to depend on a book named curl
this would work:
version = "1.x"
Or local references:
path = "/path/to/some/book"
Dependency Resolution
Books are combined in a process called "collation" into a graph. Each
book can import other books that it declared as dependencies through the
import system. If it did not declare a dependency the import will fail
even if it was reachable otherwise.
Because module scopes are immutable items can be collapsed into a common
namespace. For instance a book my_book
can ship a structure named
com/example/mybook/index.js
and then other code would import it from
com.example.mybook
. To avoid having too many nested folders the main
book.toml
file can declare a common prefix that is added in front of
all code in src.
There are two dependencies in Quill: public and private dependencies. The
default is a public dependency which means that it becomes part of the API
of a library. Dependencies can also be set private to be contained within
the library. In that case one has to be careful with throwing and
catching exceptions when they cross book boundaries.
The book names do not show up in import paths but each module knows which
book it comes from through the global __book__
constant. This one is
also used by the import system to refer to dependencies.
Importing
All the books in a book's dependencies are considered for searching. By
default the "standard books" are added automatically. So all items in the
standard library can be loaded. The books of the standard library however
can be individually versioned and individually distributed if necessary.
Import syntax:
use std.io # adds a binding named "io"
use std.io.BufReader # adds a binding named BufReader
use std.io.{BufReader, BufWriter} # adds two bindings (BufReader and BufWriter)
When a book imports from std.io
then all the book references in the
dependencies folder are considered for this. This means that book B could
import a different verison of std.io
from book B if they have different
dependencies set up. This should be the exception not the rule however.
By default packages are required to set up broad versions (eg: depends on
std-io
1.x
) and the latest version across all books is used. This
means that all books using std-io 1.x
as pin will get the same version
std-io 2.x
.
A library can also depend on both std-io 1.x
and std-io 2.x
in which
case the book needs to be given a different name to keep them apart:
dependencies/
stdio1:
book = "std-io"
version = "1.x"
stdio2:
book = "std-io"
version = "2.x"
Then the import can explicitly declare which books it wants to import
from:
@import_spec(books=['stdio1']) use std.io as iov1
@import_spec(books=['stdio2']) use std.io as iov2
Living in a Multi Library World
Because in this world different versions of a module can live together
a type has a fully qualified name that defines it. The fully qualified
ID there is in this form:
{book@version}dotted.path
So for instance BufReader
imported from std.io
might be fully
qualified like this: {std-io@1.0.0}std.io.BufReader
. Fully qualified
names are generally shown in stacktraces for exceptions:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
{std-io@1.0.0}FileNotFound: No such file or directory: '/tmp/asdfasdfsa'
This is to ensure that users are not confused if they fail to catch
an exception thrown across version boundaries.
Forcing Versions
A book can downwards enforce versions of a book in the dependency chain.
For instance it can place a stdio1-forced
file into its dependencies
folder with contents like this:
book = 'std-io'
version = "1.0.1"
force_for = "1.x"
This forces that for all pins downwards from the current book all
users of std-io at 1.x will use the 1.0.1 version. This should only
ever be used toplevel or for private dependencies.
Accessing Metadata
At any point a module can refer to __book__
to refer to an object
that gives access to the current book. The following API is
available to work with it:
__book__.name
: Returns the name of the book__book__.version
: Returns the version of the book__book__.dependencies
: Allows one to inspect the associated dependencies of the book__book__.open_resource
: returns a binary reader to a dependency__book__.metadata
: the full parsed metadata of the book (book.toml)
Before books can be loaded all the sources within a book are assembled. That process scans through all stuff reachable from src/
and performs necessary ast transformations. The end result is then cached and that cache is loaded by the actual importers. (What's the cache? .quillcache
next to src
? Folder / zip archive?)
Upon loading the __book__
is assembled from that cache and modules are created to refer to modules within the cache.
Abstract loader API:
# this looks for path/to/book/.quillcache and if it does not exist, compiles
# path/to/book/src into the cache based on the information in
# path/to/book/book.toml
var book = loader.load_book('path/to/book')
var module = book.load_module('std.io')