What? An FFI implementation of the Linux, NetBSD, FreeBSD and OSX kernel ABIs for LuaJIT. This means you will be able to program all the functionality the Unix kernel provides to userspace directly in Lua. You can view it as a high level language equivalent of the Busybox project in a way, although the functionality it provides is somewhat different, and the interface very different.
Why? First it provides a comprehensive set of system call APIs for programming sockets, files and so on, including the more obscure things (eg file change notifications). Second it provides higher level interfaces such as network interface configuration, so your application can control its entire runtime interface including IP addresses routing and so on. Third it provides tools for added security, such as support for Linux namespaces (containers), system call filtering (seccomp type 2), capabilities and so on, all with a script language interface that is much simpler to use than the C interface. As it is Lua based it can easily be embedded in another language; in the future ports to other scripting languages are planned. It also serves as a way of learning how the operating system interfaces work in a more forgiving environment than C.
There is support for the NetBSD rump kernel under other operating systems and natively without an operating system under Xen.
This code is beta. Interfaces will change in future. The code is riddled with TODOs. On the other hand it does work, and the changes at this stage will be smaller than in the past.
The video of my FOSDEM 2013 talk here, and the video of my Lua Workshop 2013 talk is here.
The stable release is now available in the luarocks repository, so you should be able to run luarocks install ljsyscall. There will be a ljsyscall-rump rock soon, but I need to fix the install for the rump libraries.
For simple uses, you just need to put the .lua files somewhere that LuaJIT will find them, eg typically in /usr/local/share/lua/5.1/. Keep the directory structure there is. You can safely remove files from architectures and operating systems you do not use.
You can install using luarocks; this is not in the repo but you can install with luarocks install rockspec/ljsyscall-scm-1.rockspec for the HEAD version, or one of the specific version files in that directory.
If you are using Lua rather than LuaJIT you need to install luaffi first; this is largely working now, but there will be more support for standard Lua coming soon.
There is more information in the INSTALL file.
Requirements: Needs LuaJIT 2.0.2 or later.
The code does not currently portably support the main Lua implementation, only LuaJIT. It now runs again with luaffi under standard Lua 5.2 (and probably 5.1, untested). Because the function calls in luaffi use dynasm they are not fully portable, so I will be adding support for Lua C interface function calls as well shortly. This is not yet integrated into the test suite.
On Linux ARM (soft or hard float), x86, AMD64 and PPC architectures are supported; MIPS support is partial, with some issues still. Either glibc/eglibc, Musl libc or uClibc should work on Linux. Note that uClibc has had little testing. For full testing (as root) a recent kernel is recommended, eg Linux 3.5 or Ubuntu 12.04 is fine, as we use many recent features such as network namespaces to test thoroughly; tests should be skipped if unsupported, raise an issue if not.
Android (ARM tested so far) currently passes all the non root tests now; some tests are be skipped as the kernel does not ship with some functionality (aio, mq). Generally much functionality is available even though it is not in Bionic libc.
For the NetBSD support all platforms should work in principle; more test targets will be added soon, currently tests being run on x86 and amd64, an ARM test target will be added.
FreeBSD support is currently for FreeBSD 9 and 10, and only tested on amd64. Other architectures should work, but are currently untested.
OSX support is currently tested only on amd64, and on a recent release (currently testing on Mountain Lion). It should work on other recent releases, and should work on ARM, ie iOS, although some tweaks.
There will not be Windows support (although in principle Cygwin and similar platforms could be supported). If you want to do similar things on Windows you should try TINN.
For the (optional) rump kernel functionality, the easiest way at present to install it is usually using the buildrump.sh project, which is now included as a git submodule. The rump kernel is a way of running parts of the NetBSD kernel in userspace as libraries. At the moment support is partially implemented, planning to add more soon, in particular to be able to script the backend "hypervisor" part. There are some additional examples in examples/rump which is a port of the tests in buildrump. The rump kernel runs on many elf/Posix OS and architectures, currently tested on Linux x86, x64, ppc, arm and NetBSD x86, x64, FreeBSD 10 (with gcc). It does not currently run on OSX or Windows.
netfilter, dhcp, selinux, arp, better sockopt handling, cgroups support, more NetBSD, FreeBSD and OSX support, rump kernel hypercall API, OSv support, Lua support, more introspection. Plus whatever users request.
Apart from the tests, there are now some examples at in the examples directory; more to come.
There will be proper documentation before the 1.0 release, apologies for it not being available sooner. I understand how important it is and it is planned shortly.
This project is being used ina variety of places, such as for testing the Linux compatibility code in NetBSD, among other things. If your project is not listed here please let me know or send a pull request.
- ljlinenoise a Lua implementation of the linenoise (readline) library.
- nodish a lightweight Lua equivalent to Node.js.
- lua-mmapfile a simple Lua interface to mmap
The test script is fairly comprehensive. Tested on ARM, amd64, x86, with various combinations of libc. I run long test runs as LuaJIT makes random choices in code generation so single runs do not necessarily show errors. Also tested with Valgrind to pick up memory errors, although there are some issues with some of the system calls, which are being gradually resolved (I use Valgrind SVN).
Some tests need to be run as root, and will not be run otherwise. You cannot test a lot of system calls otherwise. Under Linux the testing is now done in isolated containers so should not affect the host system, although on old kernels reboot in a container could reboot the host.
Some tests may fail if you do not have kernel support for some feature (eg namespacing, ipv6, bridges). Starting to add feature testing to work around this, but the way this works needs improving.
The test script is a copy of luaunit. I have made some modifications to allow tests to be skipped, which are not really general enough to push upstream, although I would like a nicer solution.
I have added initial coverage tests (now need fixing), and a C test to check constants and structures. The C test is useful for picking up errors but needs a comprehensive set of headers which eg is not available on most ARM machines so it can be difficult to run. I am putting together a set of hardware to run comprehensive tests on to make this less of an issue.
There is now Travis CI support, although this will only test on one architecture (x64, glibc) at present. You can see the test results here. If you fork the code you should be able to run these tests by setting up your own Travis account, and they will also be run for pull requests.
I have used the LuaJIT reflect library for checking struct offsets.
Adding buildbot tests for a wider variety of architectures, as Travis is limited to Linux/Ubuntu. Currently building on Linux ARM, PowerPC, x64 and x86, NetBSD x86 and x64, FreeBSD x64; more targets to come soon. The buildbot dashboard is now up.
This project is in beta! Much stuff is still missing, this is a work in progress! The majority of syscalls are now there, let me know if you need some that are not.
As well as syscalls, there are interfaces to features such as proc, termios and netlink. These are still work in progress, and are being split into separate modules.
Work on the Linux netlink API is progressing. You can now do print(S.get_interfaces()) to get something much like ifconfig returns, and all the raw data is there as Lua tables. You can then modify these, and add IP addresses, similarly for routes. There is also a raw netlink interface, and you can create new interfaces. There is a lot more functionality that netlink needs to provide, but this is now mostly a matter of configuration. The API needs more work still. Netlink documentation is pretty bad. Useful resources: blog post
There is also a lot of the ioctl, getsockopt and fcntl interfaces to implement, which are very miscellaneous. Mostly you just need some constants and typecasting, but helper functions are probably useful. These are being improved so they understand the underlying types and functionality, making them less error prone.
The aim is to provide nice to use, Lua friendly interfaces where possible, but more work needs to be done, as have really started with the raw interfaces, but adding functionality through metatypes. Where possible the aim is to provide cross platform interfaces for higher level functionality that are as close as possible at least in a duck-typing sort of way.
Under Linux, lots of system calls have glibc wrappers, some of these are trivial some less so, and some are broken. In particular some of them expose different ABIs, so we try to avoid these, just using kernel ABIs as these have long term support and we are not trying to be compatible as we are using a different language. strace is your friend, although strace is buggy in the nasty edge cases (at some point ljsyscall will implement ptrace so it can debug itself; it implements ktrace in NetBSD). Therefore under Linux the project is gradually moving to calling system calls directly, bypassing the libc, just keeping directly to the kernel ABI.
As well as eglibc and glibc, everything now runs on Musl libc. I use sabotage as a build environment, which now includes luajit, although you may need to update to git head. Musl is much smaller than libc (700k vs 3M), while still implementing everything we need in easy to understand code. It is also MIT licensed, which may be useful as it matches the other licenses for LuaJIT and ljsyscall. Occasionally I find small bugs and missing features which I feed back to the developers. The Android libc, bionic, is also supported now, mainly by bypassing it and calling the system calls directly.
Under NetBSD it is much simpler, the only thing we need to be careful of is versioned system calls in libc, where we directly call a specific version as the plain name will always refer to the old version for compatibility. FreeBSD does a more transparent versioning, so the syscall names stay the same.
This will be documented properly soon, once it stabilises.
All functions return two or more values, the return value, or true if there is not one other than success, then an error value. This makes it easy to write things like assert(fd:close()). The error type can be converted to a string message, or you can retrieve the errno, or test against a symbolic error name. Some functions then return additional return values, such as pipe() that returns the two file descriptors for the pipe. A few functions return a Lua iterator, such as epoll_wait().
File descriptors are returned as a type not an integer. This is because they are garbage collected by default, ie if they go out of scope the file is closed. You can get the file descriptor using the fileno field. To disable the garbage collection you can call fd:nogc(), in which case you need to close the descriptors by hand. They also have methods for operations that take an fd, like close, fsync, read. You can use this type where an fd is required, or a numeric fd, or a string like "stderr".
String conversions are not done automatically, you get a buffer back, you have to force a conversion. This is because interning strings is expensive if you do not need it. However if you do not supply a buffer for the return value, you will get a string in general as more useful.
Many functions that return structs return metatypes exposing additional methods, so you get the raw values eg st_size and a Lua number as size, and possibly some extra helpful methods. As these are (ffi) metamethods they have no overhead, so more can be added to make the interfaces easier to use.
Where there are variable length arrays, these are bundled together into a structure that has an array and a count, so you do not need to keep passing around the size. These provide iterators, which helps hide the fact that they are 0-based.
Constants should all be available, eg c.SEEK.SET etc, note they are namespaced into Lua tables rather than underscore seperated like in C. The constant tables will also let you combine flags where appropriate and you can use lower case, so c.O["rdonly, create"] is the same as the bitwise or of c.O.RDONLY and c.O.CREAT. When you call a function, you can just pass the string, as fd = S.open("file", "rdonly, creat") which makes things much more concise.
You do not generally need to use the numbered versions of functions, eg dup can do dup2 or dup3 by adding more arguments (not fully consistent yet).
Types are key, as these encapsulate a lot of functionality, and easy to use constructors and helpful methods. For example you can create the in_addr type with addr = t.in_addr("127.0.0.1") or addr = t.in_addr("loopback").
The test cases are good examples until there is better documentation!
A very few functions have arguments in a different order to make optional ones easier. This is a bit confusing sometimes, so check the examples or source code.
There is now an expreimental luafilesystem interface available, use lfs = require "syscall.lfs". It would be nice to be API compatible with other projects, especially Luaposix, luasocket, nixio. Unfortunately none of these seem to have good test suites, and there interfaces are problematic for some functions, so this has been put on hold, although basic luasocket support is planned fairly soon.
If you want the highest performance, allocate and pass your own buffers, as obviously allocation is expensive. It is now fine to use the string flags for functions, as these are memoized. Check the output of luajit -jv to see what is going on and let me know if there are any issues that need fixes for NYI functions. You should be able to get native C like performance.
There is an example epoll script that you can test with Apachebench in the examples directory. On my machine apachebench uses more CPU time than the script so the results are a bit low.
If you wish to port to an unsupported platform, please get in touch for help. All contributions welcomed.
Porting to different Linux processor architectures is a matter of filling in the constants and types that differ. The ctest tests will flag issues with these, although many platforms are also missing headers which makes it more complex. If you can provide qemu target information that would be helpful as the platform can be added to the test suite, or easily available hardware.
Porting to different OSs is a fair amount of work, but can generally be done gradually. The other BSDs are very similar to NetBSD, FreeBSD and OSX. Solaris has an ABI defined by libc not the kernel ABI, which would mean that you should probably target that. The first thing to do is check the base shared types, and work out if there are issues with large file support if it is a 32 bit platform. There is a fair amount of shared code that should just work once the types and constants are defined.
If you want to port this to a different language, then get in touch, as I have some ideas and plans along this route, though I am trying to get a good fairly stable interface in Lua first. Pypy and Ruby ought to be suitable targets as they have an ffi; I also intend to do a classic Lua port using the C API. I intend to use reflection to generate more generic data for the ports, and rework how files are included. The first thing to do is just prototype some basic functions to see what is needed. Get in touch if you are interested in a port!
There will no doubt be bugs and missing features, please report them if you find them. Also API design issues. You can use the github issue tracker to report issues.
All the ljsyscall code is under the MIT license. The ABI definitions are considered to be non copyrighted or CC0 if you need an official disclaimer. See LICENSE file for further details.