User space threading library

This is a micro project of a x86-64 asm cooperative user space threading library written in C.

API

User space threads are mapped N:1, i.e. N user space threads are mapped to a single kernel space process. The implementation uses x86-64 inline assembly and relies on setjump. It is only tested under GNU/Linux.

struct thread *thread_create(void (*fun)(void *), void *arg);
void thread_add_runqueue(struct thread *t);
void thread_yield(void);
void thread_start_threading(void);

A user space thread is created with thread_create(...). It is given a function to execute and function arguments. A thread can be added to the run queue with thread_add_runqueue(...). A round-robin scheduling is implemented. Threading can be started with thread_start_threading(). This function returns if all user space threads returned. thread_yield() interrupts execution of a thread and runs the scheduler to dispatch another thread. User land threads are supposed to cooperate and explicitly yield when they no longer require CPU time. They are not interrupted by other mechanisms.

Thread descriptor

Upon creation, a thread is assigned a new stack region. Among other things, a thread descriptor keeps track of the life-cycle state of a thread, and when yielded stores CPU context in a jmp_buf.

struct thread {
    int64_t id;
    int8_t state;
    void (*fun) (void *);
    void *arg;
    void *sp_base;
    int32_t s_size;
    jmp_buf jmp_buf;
    struct thread *next;
};

Dispatching a thread

When a thread is dispatched for the first time, its stack and base registers are set. When it ran previously, its execution context is restored with longjmp.

if (does thread run for first time ) {
     - emit mov <thread-specific-stack>, %%rbp;
     - emit mov <thread-specific-stack>, %%rsp;
     - call main function of thread
     - we reach here only when main function of thread terminates.
     - longjmp to scheduler
} else {
     - longjmp back to where execution last yielded
}

Yield execution

Threads need to cooperate and yield their execution to let other threads execute. Upon yield we are still on the stack of the thread. We need to store its execution environment and longjmp back into our scheduler to schedule and dispatch another thread.

Limitations

This implementation uses inline assembly and is not portable.
The nature of cooperative multitasking allows threads to be selfish.

Improvements

Stack size fixed and may not be enough. Scheduler does not catch SIGSEGV when user space thread runs out of stack space.
Implement preemptive multitasking in user space with signal handlers. Use SIGALARM and make kernel call into user space periodically. Use Linux APIs makecontext() / swapcontext() to switch execution environments.
No deadlock detection, if a thread deadlocks scheduler deadlocks too.

Resources

Portable-Multithreading - The Signal Stack Trick For User-Space Thread Creation: http://www.gnu.org/software/pth/rse-pmt.ps
https://stackoverflow.com/questions/20448817/how-is-preemptive-scheduling-implemented-for-user-level-threads-in-linux

abertschi / userspace-threading