FileIO

FileIO aims to provide a common framework for detecting file formats and dispatching to appropriate readers/writers. The two core functions in this package are called load and save, and offer high-level support for formatted files (in contrast with julia's low-level read and write). To avoid name conflicts, packages that provide support for standard file formats through functions named load and save are encouraged to extend the definitions here. Supported Files

Installation

Install FileIO via Pkg.add("FileIO").

Usage

If your format has been registered, it might be as simple as

using FileIO
obj = load(filename)

to read data from a formatted file. Likewise, saving might be as simple as

save(filename, obj)

If you just want to inspect a file to determine its format, then

file = query(filename)
s = query(io)   # io is a stream

will return a File or Stream object that also encodes the detected file format.

Adding new formats

You register a new format by adding add_format(fmt, magic, extension) to the registry. To do so, please just open a pull request (you can just edit the file in Github). fmt is a DataFormat type, most conveniently created as format"IDENTIFIER". magic typically contains the magic bytes that identify the format. Here are some examples:

# A straightforward format
add_format(format"PNG", [0x89,0x50,0x4e,0x47,0x0d,0x0a,0x1a,0x0a], ".png")

# A format that uses only ASCII characters in its magic bytes, and can
# have one of two possible file extensions
add_format(format"NRRD", "NRRD", [".nrrd",".nhdr"])

# A format whose magic bytes might not be at the beginning of the file,
# necessitating a custom function `detecthdf5` to find them
add_format(format"HDF5", detecthdf5, [".h5", ".hdf5"])

# A fictitious format that, unfortunately, provides no magic
# bytes. Here we have to place our faith in the file extension.
add_format(format"DICEY", (), ".dcy")

You can also declare that certain formats require certain packages for I/O support:

add_loader(format"HDF5", :HDF5)
add_saver(format"PNG", :ImageMagick)

These packages will be automatically loaded as needed. You can also define the loaders and savers in a short form like this:

add_format(format"OFF", "OFF", ".off", [:MeshIO])

This means MeshIO supports loading and saving of the off format. You can add multiple loaders and specifiers like this:

add_format(
    format"BMP",
    UInt8[0x42,0x4d],
    ".bmp",
    [:OSXNativeIO, LOAD, OSX],
    [:ImageMagick]
)

This means, OSXNative has first priority (gets loaded first) and only supports loading bmp on OSX. So on windows, OSXNativeIO will be ignored and ImageMagick has first priority. You can add any combination of LOAD, SAVE, OSX, Unix, Windows and Linux.

Users are encouraged to contribute these definitions to the registry.jl file of this package, so that information about file formats exists in a centralized location.

Implementing loaders/savers

In your package, write code like the following:

using FileIO

# See important note about scope below
function load(f::File{format"PNG"})
    open(f) do s
        skipmagic(s)  # skip over the magic bytes
        # You can just call the method below...
        ret = load(s)
        # ...or implement everything here instead
    end
end

# You can support streams and add keywords:
function load(s::Stream{format"PNG"}; keywords...)
    # s is already positioned after the magic bytes
    # Do the stuff to read a PNG file
    chunklength = read(s, UInt32)
    ...
end

function save(f::File{format"PNG"}, data)
    open(f, "w") do s
        # Don't forget to write the magic bytes!
        write(s, magic(format"PNG"))
        # Do the rest of the stuff needed to save in PNG format
    end
end

Note that these are load and save, not FileIO.load and FileIO.save. Because a given format might have multiple packages that are capable of reading it, FileIO will dispatch to these using module-scoping, e.g., SomePkg.load(args...). Consequently, packages should define "private" load and save methods, and not extend (import) FileIO's.

load(::File) and save(::File) should close any streams they open. (If you use the do syntax, this happens for you automatically even if the code inside the do scope throws an error.) Conversely, load(::Stream) and save(::Stream) should not close the input stream.

Help

You can get an API overview by typing ?FileIO at the REPL prompt. Individual functions have their own help too, e.g., ?add_format.

evetion / FileIO.jl