Add pthread.h in wasi sysroot to support pthread
xujuntwt95329 opened this issue · comments
Hi,
The pthread library has been supported in wasm micro runtime. The pthread related APIs in wasm application should be imported from the runtime, such as pthread_create, pthread_join, etc. so the runtime can create an actual thread to execute the wasm function.
For example, suppose we have such a C code:
#include <pthread.h>
void *func(void *arg)
{
}
int main()
{
pthread_t tid;
pthread_create(&tid, NULL, func, NULL);
return 0;
}The pthread_create should be compiled as an import function:
(import "env" "pthread_create" (func $pthread_create (type $t1)))
......In this situation, I think there should be a simple way to support pthread in wasi-sdk as we only need a few things:
- type definition for pthread_t, pthread_mutex_t and so on
- function prototype for pthread APIs such as pthread_create, pthread_join, pthread_self, etc.
- undefined-symbol lists
To achieve 1 and 2, we just need a pthread.h under sysroot/include, just like WAMR's pthread.h, we also need to undef some macros like __NEED_pthread_t to avoid duplicate definition with sysroot/include/bits/alltypes.h
And to achieve 3, we need to add pthread API symbols into sysroot/share/wasm32-wasi/undefined-symbols.txt just like wasi APIs
In this way, we don't need to provide libpthread.a in wasi-libc, as the pthread related APIs should be implemented by the runtime.
Do you think this is a reasonable way to support pthread ?
Furthermore, I'm trying to build some C++ workload which use std::thread, but I get this error:
error:
<atomic>is not supported on this single threaded system
I noticed that the std::thread depends on pthread. In this case, if we have supported pthread, is there any other thing to do to support std::thread ?
Thanks a lot !
The current situation is that there is a consensus among the people working on threads support in the WebAssembly core spec is that the current semantics are not yet suitable for use outside of Web contexts. See the following issues for background:
WebAssembly/threads#8
WebAssembly/threads#95
WebAssembly/threads#138
Following this guidance, WASI libc does not yet attempt to support pthreads. This guidance is of course open for discussion.
If you're looking to do something WAMR-specific and don't need anything in libpthread.a, then you don't need WASI libc to help; you can just define those APIs and provide the header file yourself.
The lack of pthread headers becomes problematic when attempting to use higher-level APIs in WASI SDK, such as C++ <atomic>. While users can provide the header, this still requires recompiling WASI SDK from scratch, so I wonder if it would be possible to ship pthread headers in WASI libc after all with a no-op implementation to allow compiling C++ code that relies on <atomic>.
BTW, with llvm 11 there will be no need for undefined-symbols.txt at all anymore since symbols can be marked as imported in the source code.. so using a recent build of llvm should make that issue at least go away.
It is a shame that <atomic> doesn't just work.. since at the llvm we lower them all away in single threaded mode.
Are you sure it would require a rebuild of libc rather than just the addition headers? I think this issue (allowing atomics to work in single-threads builds) is somewhat of a separate issue to actually add pthreads and/or compiling with threading/shared memory.
Could you elaborate please what do you mean by "the addition headers" here? This is the error that I get currently without the pthread header, building wasi-libc with THREAD_MODEL=POSIX and without building the WASI SDK with LIBCXXABI_ENABLE_THREADS=ON, LIBCXXABI_HAS_PTHREAD_API=ON and COMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF:
wasi-sdk/share/wasi-sysroot/include/c++/v1/atomic:560:3:
error: <atomic> is not supported on this single threaded system
When we initially set up all these builds, we configured them for "no threads" mode, because we care about code size and have since the beginning, and some libraries have smaller code size than if they are built in threads mode and we just lower atomics. I don't know if that's specifically true of libcxx though, so if someone could investigate that, that'd be a good start.
If it does turn out that libcxx is bigger when compiled in threads mode, a possible option would be to produce two builds, one where everything is built with threads disabled, and one where threads are ostensibly enabled but atomics are lowered.
Circling back to this issue, It appears that thread-local storage is needed to activate multithreading at this time. As a POSIX newbie, what does it take to generate a single-threaded pthreads implementation that doesn't need TLS?
AFAIK clang supports -femulated-tls, but I don't know if it's working for the Wasm target, and I haven't tried it with WASI. But that's where I'd start looking first.
Thanks for the prompt reply!
fatal error: error in backend: only -ftls-model=local-exec is supported for now on non-Emscripten OSes: variable errno
is what I get. Will it be possible to mock up a single threaded pthreads lib?
Did you try adding the suggested -ftls-model=local-exec flag then?
I didn't know the two could work together. I'll try again. Thanks a bunch!
I got the same error as before.
I think LLVM/clang would need some modifications to support -femulated-tls. I can't say anything on behalf of maintainers of wasi-libc, but I tried to mock TLS and submit patches to other repositories, and review feedback was that this should be resolved with -femulated-tls in clang/LLVM, instead of adding mock TLS implementations in client code.
Ugh. Ok, thanks.
To clarify, the motivation is that TLS support for WebAssembly should be implemented once on clang/LLVM level, instead of adding it separately every time in all client code that depends on TLS, which would ultimately save us a lot of effort in the long run.
Oh ok.
I found out I added the flags to the wrong spot. Now I got farther.
I'm really close. It compiles and links but fails the test with:
build/wasi-libc.BUILT: build/llvm.BUILT
$(MAKE) -C $(ROOT_DIR)/src/wasi-libc \
WASM_CC=$(BUILD_PREFIX)/bin/clang \
SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \
WASM_CFLAGS="-U_REENTRANT -femulated-tls -ftls-model=local-exec -O2 -DNDEBUG" \
THREAD_MODEL=posix
touch build/wasi-libc.BUILT
what kind of error messages do you see?
2 errors:
/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/signal.h:2:2: error: "wasm lacks signal support; to enable minimal signal emulation, compile with -D_WASI_EMULATED_SIGNAL and link with -lwasi-emulated-signal"
#error "wasm lacks signal support; to enable minimal signal emulation, \
^
In file included from /home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/share/wasm32-wasi/include-all.c:63:
/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/aio.h:21:18: error: field has incomplete type 'struct sigevent'
struct sigevent aio_sigevent;
^
/home/samuraicrow/Documents/wasi-sdk/build/install/opt/wasi-sdk/share/wasi-sysroot/include/aio.h:21:9: note: forward declaration of 'struct sigevent'
struct sigevent aio_sigevent;
^
2 errors generated.
I'm really close! Adding -D_WASI_EMULATED_SIGNAL \ to the test and #define _WASI_EMULATED_SIGNAL 1 to the expected results clears those errors. Now it's found a symbol called emutls_v.errno that I'm sure is also supposed to be there but needs to be stripped from the test results so it doesn't look compiler-specific.
Adding that symbol to the expected results clears the test! Now I just need to test it and update the submodules in my fork of wasi-sdk to use my fork of wasi-libc!
After wrestling with submodules all day, I just discovered that I forgot to set THREAD_MODEL=posix when building earlier. Now aio.h is trying to build an aiocb structure around a sigevent structure that doesn't exist on the WASI version of the headers. I think I'll try excluding aio.h to see what else breaks.
I get over 75% of the way through compiling libC++ only to run into uninitialized functions in pthread.h:
#if defined(__wasilibc_unmodified_upstream) || defined(_REENTRANT)
#define PTHREAD_MUTEX_INITIALIZER {{{0}}}
#define PTHREAD_RWLOCK_INITIALIZER {{{0}}}
#define PTHREAD_COND_INITIALIZER {{{0}}}
#else
#define PTHREAD_MUTEX_INITIALIZER 0
#define PTHREAD_RWLOCK_INITIALIZER 0
#define PTHREAD_COND_INITIALIZER 0
#endif
#define PTHREAD_ONCE_INIT 0_REENTRANT is currently not possible AFAICT and __wasilibc_unmodified_upstream is false because the WASI headers are used rather than MUSL. It doesn't appear that any of them do anything one way or the other. Now the fun begins as I start to implement single-threaded PThreads.
Update
By allowing the block of code listed in my previous post to execute, along with the code that was flagged as not reentrant, the LibC++ code compiled fully but didn't execute. I've been researching how to proceed.
Sched.h and Futex.c
The requirements for sched.h are implemented in libc-top-half/musl/include/sched.h but has most of the code left out due to the __wasilibc_unmodified_upstream macro being undefined. My goal is to get all the code needed to make WASI-LibC to be reentrant and multitasking.
The only other function needed to implement multitasking is a single-threaded version of a futex. The header is defined at libc-top-half/musl/src/internal/futex.h and I've investigated instances of futexes implemented by Fuchsia's Zircon kernel and OpenBSD but I've decided to implement it myself. The other instances were made extra difficult by the nature that they were designed for real multithreading. OpenBSD had a flag for single-threaded multitsking but the support subroutines were mostly in the 3-clause BSD License which requires that all derived binaries cite the source, making it inappropriate for use in a statically linked runtime library.
It looks as if the only scheduler that doesn't require a timer based interrupt is the FIFO scheduler. While a counter based "event" could theoretically replace the need for such an interrupt, that would require modification to the bytecode processing to generate such a counter so it could be updated after every instruction executed.
UGH! There are 375 uses of the macro __wasilibc_unmodified_upstream scattered throughout the WASI-LibC codebase. Most of them are still needed but anything that asserts that threads and processes don't exist on WASI need to be reevaluated once I add the FIFO scheduler and primitive cooperative multitasking.
I made an attempt to guard the code needed by threads with #if defined(__wasilibc_unmodified_upstream) || defined(_REENTRANT), to distinguish it from code excluded for other reasons which uses plain #if defined(__wasilibc_unmodified_upstream). It's likely not perfect, but it may be helpful.
Thanks! I've been looking through the _REENTRANT code bits and trying to mutex sections of them to get them reentrant already. Maybe I've already dealt with enough of them to find what works and am getting lost in the other stuff. Knowing why you did what you did helps though.
Using a multifile search reveals more of those than I thought there were! Thanks @sunfishcode !
Update
I found a corner case in that after a time-slice yield, the same task's time slice in the next iteration would start at the beginning of the function that yielded instead of the address after the yield subroutine call. This means that without a customized runtime, multitasking won't directly be possible in WASI. Hopefully there will be some experimental patches added to Wasmer soon that look like they might help. I don't know what WasmTime is going to do though. Does anybody know what would be necessary to make a stack swap possible and in the process freeze one task to start the next? If it's not possible at all yet, I may need to apply a patch somewhere else.
I'm not quite sure what you goals are, but have you looked and Ayncify? https://emscripten.org/docs/porting/asyncify.html. This is a binaryen tool that allows wasm binaries to be suspended and resumed at certain points. emscripten uses this to suspend threads and also implements its fibre API this way: https://emscripten.org/docs/api_reference/fiber.h.html.
Asyncify only works in the browser as far as I'm aware. I want an equivalent function for WASI.
Asyncify can work in any wasm environment - nothing in it is specific to the Web. For example, it is used in the CRuby port of WASI and in TinyGo.
(The Asyncify blogpost starts with an example called "Pure WebAssembly" which shows how to do that.)
In order to work in Wasmer, the WASI code needs to be a "clean implementation" of WebAssembly. I was told by the guy who assigned the project, not to use Emscripten because its WASI implementation was not clean somehow. I need something that works from Clang using the LLVM WebAssembly backend. Wasmer is my primary target.
Using asyncify doesn't require using emscripten. It just happens to be one of the primary use cases for asyncify. As I said, I'm not quite sure what problem you are trying to solve, but if it involves unwinding and re-winding wasm call stacks (in order to suspend and continue them later) and asyncify sounds like a good place to start.
It's a custom transformation pass applied at the compiler level. I'll have to port it to Clang to get it to work. It's as clean of an implementation as I'm likely to find using a compiler hack.
I may have identified a rare bug in asyncify: If there are two yields in the same function does resuming the second yield call return to the correct place?
@SamuraiCrow That should work in Asyncify. The mechanism is that we serialize an ID for the call location in the function, so we know how to return to the right place. We have tests for this that pass, but perhaps you've found a corner case somehow where we are buggy - if you can get a testcase that would be good.
Perhaps I was looking at an oversimplified example of how Asyncify was supposed to work. If its design already accounts for the corner cases, it should meet my needs.
Emscripten has a PThread stub that may accomplish what I was trying to do here instead of Asyncify. Asyncify looks rather involved.
Does you code actually need to do threading? Or can it run in a single thread if needs be (i.e. can run on a system where pthread_create() always fails)? The pthread stubs in emscripten are a version of the pthread API that doesn't do anything and doesn't allow for the creation of any threads.. just FYI. Its uses in the case where you compile a thread-aware program but you are targeting a single-threaded runtime.
There's a conditional compilation flag that allows single-threaded execution but I didn't know about it until recently.
Any news to this issue? I am trying to figure out as well how to compile code with threads with WASI.
@sbc100
I tried using wasi-sdk with -pthread and wasm-ld reported the error:
--shared-memory is disallowed by errno.o because it was not compiled with 'atomics' or 'bulk-memory' features.
I found https://stackoverflow.com/questions/70180309/how-to-link-wasi-libc-with-shared-memory-flag, where you mentioned this error stems from the fact that the wasi-libc distribution isn't build with -pthread.
With #311 merged, does that mean it is now possible to build a wasi-libc with -pthread? If that is the case, is make THREAD_MODEL=posix all that is needed?
Looks like make THREAD_MODEL=posix works, exciting :-)
I also just saw #326, which mentions pointing wasi-sdk to the thread-enabled sysroot.
However, when trying to compile a simple pthread test program, I get errors for many undefined symbols:
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __unlock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __thread_list_lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __thread_list_lock
wasm-ld: error: .../sysroot/lib/wasm32-wasi/libc.a(pthread_create.o): undefined symbol: __wasilibc_pthread_self
...
Do the thread-enabled sysroot's libraries assume that the user defines these?
cc @abrown
With #311 merged, does that mean it is now possible to build a wasi-libc with -pthread?
I don't think the "threads in wasi-libc" story is quite ready for use yet; there are still big pieces that need additional work. However, progress is being made: #311 is a start and the recent merging of #325 is an important step forward. Some of the pieces that still need work: finish compiling in the remainder of the pthread_*.c implementation in wasi-libc, fix some duplicate symbol definitions of __get_tp that were introduced at some point, #326, rust-lang/rust#102372, figure out some way to test all of this, etc.
Do the thread-enabled sysroot's libraries assume that the user defines these?
Probably not; I would guess that this is part of finishing out the compilation of more pthread_*.c files in the wasi-libc THREAD_MODEL=posix build. If you're interested in helping out with that, feel free to contact me on Zulip; otherwise, I think the best approach is to wait a bit longer for more parts to be finished.
@abrown Thank you for the update, that helps understanding what to expect. That's really great work!
Over the weekend, I had done some research into how pthreads support could be added to https://github.com/turbolent/w2c2 and tried to piece together the puzzle of what the libc would need to do, how the runtime could implement it, etc. just to realize there actually is a WASI proposal now, and that an implementation is in progress. Exciting! I'm looking forward to implementing the wasi-threads proposal once the libc has support for it. Unfortunately I'm not really familiar with the implementation details on that side.