FFI?
technomancy opened this issue · comments
I would like to write some scheme that calls out to existing C libraries. I don't see anything in the examples about this. How much work would it be to add, and what would it look like?
I think that the benefits of an FFI are considerable... I guess there are two main tasks for this:
- Construct some primitive (call-c-func fptr args ...) which does the following:
- Evaluate args and push them onto the stack
- Do some type conversions on args
- Prepare a C-style call frame on the stack
- Save the microscheme special registers
- Jump to code address fptr
- Restore the microscheme special registers
- POP the call frame
- Convert the result into some ms type
- Work out:
- How to determine, statically, the return type for any C function being called, since C values don't carry around any type information...
- How to compile the desired C code and link it with ms programs
- How to make sure the C and ms code share the heap space safely
I have a pretty good idea how to do (1). As for the rest, It's going to take a lot of experimentation (or someone with a deep understanding of the avr-gcc pipeline) to hash out the details...
It seems like if we make a few concessions most of the tricky bits could be
cut out. For instance, if the extern declaration had to include type
declarations, you wouldn't have to do any static analysis of the C. Also while
it's handy for beginners to have a single command that can do all the
linking and uploading, I think it's better to assume advanced use
cases will be handled by external tools. So I think that part should be left
out of Microscheme anyway.
It would be cool to have extern emit a procedure named after the C
function so you could call that instead of a separate extern-call,
but I'm not sure how much additional work that would be.
I'd like to take a shot at this, but I couldn't find much in the way
of details about how to place a call frame on the stack. There is
https://gcc.gnu.org/wiki/avr-gcc#Frame_Layout but it's pretty sparse
on details.
Ok, I when I saw your message this morning I had a sudden wave of inspiration, and I've managed to get something working...
I worked out a lot of the details by just compiling some C functions using avr-gcc, disassembling and inspecting the output. Also, there are a lot of clues here: http://www.nongnu.org/avr-libc/user-manual/FAQ.html#faq_reg_usage
It seems that avr-gcc always allocates the first 10 arguments to any function in registers, and most of the call frame is actually constructed by the 'prologue' and 'epilogue' of the subroutine itself.
As you said, we can make concessions to get this moving. For now, I've assumed that we'll be calling a function which takes only integers as arguments, and returns an integer. This works out well because the microscheme number type representation is exactly the same as a C unsigned 16-bit integer (uint16_t). So, we can forget all type conversion stuff for now... I have written the primitive (call-c-func fname args...) which will work for any function taking and returning unsigned integers...
Here is the assembly generated for (call-c-func "external_func" arg1 arg2 arg3)
PUSH CCPl ; Save the special registers...
PUSH CCPh
PUSH HFPl
PUSH HFPh
PUSH AFPl
PUSH AFPh
PUSH r1
CLR r1 ; Because C expects r1 always to contain zero
{evaluate arg1}
MOV r24, CRSl ; Move the arguments into place
MOV r25, CRSh
{evaluate arg2}
MOV r22, CRSl
MOV r23, CRSh
{evaluate arg3}
MOV r20, CRSl
MOV r21, CRSh
RCALL external_func ; Call the function
MOVW CRSl, r24 ; Move the result into CRS
POP r1 ; Restore the special registers
POP AFPh
POP AFPl
POP HFPh
POP HFPl
POP CCPh
POP CCPl
It turns out that the second part, linking the code, is quite easy if we just compile the C code to assembly, and include it at the assembly level.
Check out ffi_stuff/ in master. Read ffi-test.ms and ffitest.c, and I think you'll see what's going on...
To try it yourself, these are the steps:
Compile the C to assembly:
$ avr-gcc -S -o ffi_stuff/ffitest.s ffi_stuff/ffitest.c
Compile and upload the ms
$ microscheme -m UNO -a -d SERIAL PORT -u ffi_stuff/ffi-test.ms
Watch the serial connection for the output:
$ python ffi_stuff/dump.py SERIAL PORT
(You may need to press the reset button while the python is running to see it)
For me, it outputs 7, so something is working right...
Does this example work for you?
If so, then the next steps are:
- Test the (call-c-func) primitive in different contexts to check that it is saving and restoring the runtime system state correctly...
- Decide whether we will ever need to call a C function with more than 10 arguments? :)
- Decide which microscheme types we want to be passable to C code
- Write type conversion code
- Decide how we want the microscheme code to look. Provide more primitives to clean up the external definition?
Also, I haven't thought about global or local variables in the C code. This is pretty important...
This is great! I've actually been able to get a basic keyboard firmware working in microscheme on my atmega32u4:
(define (loop)
(if (low? 11)
(call-c-func "usb_send" 0 4 0 0 0 0 0)
(call-c-func "usb_send" 0 0 0 0 0 0 0))
(loop))
(define (init)
(input 11)
(high 11) ; activate pullup resistor
(call-c-func "usb_init")
(pause 200))
(init)
(loop)For my purposes I have no problem writing C shims to make it work with the limited microscheme FFI, so not being able to set variables or pass other types isn't a big deal. I agree that it needs a bit more work to be nicely polished, but this is a great start! I'm running into a problem that seems unrelated and will discuss that in another issue.
The fixes for #6 have made the exceptions go away, but it looks like
it's also caused the USB keycodes to no longer be sent. Even the
minimum example in the comment fails now, though the device still
shows up in lsusb.
Anyway, I've rolled back to microscheme rev 0b3832b, which worked a
couple days ago, and that no longer works either. I've cleaned, tried
swapping out the device for a fresh one, tried it on a Teensy 2, tried
with a fresh checkout of microscheme, tried compiling on a different
computer, and have attempted to try bisecting all the commits since
FFI was added, but I can't get a successful build anywhere. I'm
completely stumped.
I'll have to step away from the project for a few days to clear my mind before
I try to look for further things to try, but if you have any suggestions they'd
be appreciated.
I'm very sorry, I think I left the repo' in a bit of a mess...
Would you minding pulling now and giving it another go?
I'm pretty sure this isn't from any of your recent changes; I get the same kinds of problems if I roll back to some of the revisions right after the FFI was added or the latest master. I feel like it must be some local state on my own machine; I just can't think of what it could be right now.
Up until my latest push, the FFI has had one serious problem or another at every point. Initially, it worked but the ISRs would mess up ms code. 0b3832b was a bad fix: interrupts would be enabled only for the duration of the very first C function call. (This would cause the USB device to show up, but then not send any data, as you described.) Now, interrupts are disabled whenever ms code is running, but restored before every C call...
I don't want to sound patronising, but did you remember to $make hexify?
Yeah, I had definitely run make hexify previously, even with the current master, and I got nothing. I deleted my checkouts of both microscheme and my own codebase and recompiled everything from scratch today, and I'm getting keycodes again. So... still no idea what the root cause was, but I'm back in business. Thanks.
Phew!
I'm happy to close this out as it's got everything I need, but maybe we could replace it with some other issues that have more specific features, like doing type conversions on FFI calls.
Sure.
Have you seen ffi_stuff/microscheme_types.c and ffi_stuff/ffitest.c?
microscheme_types.c defines the microscheme types precisely in C. ffitest.c shows how you can then recurse through microscheme data structures easily from the C side...
Now we can say that any C function called through the FFI should take up to 10 arguments of type ms_value, and return one ms_value. All type conversion is done on the C side via microscheme_types.c. I think this is a very nice arrangement, because the C type checker will enforce all necessary type conversion.
Oh gotcha; very cool. That wasn't exactly what I was thinking, but I'm happy with that arrangement too.