pthread_join test hangs occasionally
prp opened this issue · comments
A CI run of basic-pthread_join-(debug)-(run-hw)-(8-ethreads) led to the following hanging behaviour in the middle of the test:
Execution timeout: 60
Will run tests for run-hw
/opt/sgx-lkl-debug/bin/sgx-lkl-disk create --size=5M --docker=./Dockerfile sgx-lkl-rootfs.img
Using detected host DNS server in /etc/resolv.conf: 168.63.129.16
Creating sgx-lkl-rootfs.img from Dockerfile ./Dockerfile...
Building Docker image...
Creating disk image file from Docker container...
Succesfully created sgx-lkl-rootfs.img.
SGXLKL_VERBOSE=1 SGXLKL_KERNEL_VERBOSE=1 /opt/sgx-lkl-debug/bin/sgx-lkl-run-oe --hw-debug sgx-lkl-rootfs.img pthread_join-test
[ SGX-LKL ] nproc=4 ETHREADS=8 CMDLINE="mem=32M"
[ SGX-LKL ] HW TLS support: econf->fsgsbase=1
[ SGX-LKL ] Registering disk 0 (path='sgx-lkl-rootfs.img', mnt='/', [RW])
[ SGX-LKL ] No tap device specified, networking will not be available.
[ SGX-LKL ] get_signed_libsgxlkl_path... result=/opt/sgx-lkl-debug/bin/../lib/libsgxlkl.so.signed
[ SGX-LKL ] oe_create_enclave...
[ SGX-LKL ] Enclave config: {"args":["pthread_join-test"],"clock_res":[{"resolution":"0000000000000001"},{"resolution":"0000000000000001"},{"resolution":"0000000000000000"},{"resolution":"0000000000000000"},{"resolution":"0000000000000001"},{"resolution":"00000000003d0900"},{"resolution":"00000000003d0900"},{"resolution":"0000000000000001"}],"cwd":"/","env":[],"esleep":"16000","espins":"500","ethreads":"8","exit_status":"full","format_version":"1","fsgsbase":true,"host_import_env":[],"hostname":"lkl","hostnet":false,"image_sizes":{"num_heap_pages":"262144","num_stack_pages":"1024"},"io":{"block":true,"console":true,"network":true},"kernel_cmd":"mem=32M","kernel_verbose":true,"max_user_threads":"256","mmap_files":"shared","mode":"hw_debug","mounts":[],"net_gw4":"10.0.1.254","net_ip4":"10.0.1.1","net_mask4":"24","oe_heap_pagecount":"8192","root":{"key":"","key_id":null,"overlay":false,"readonly":false,"roothash":null,"roothash_offset":"0"},"stacksize":"524288","swiotlb":true,"sysctl":null,"tap_mtu":"0","verbose":true,"wg":{"ip":"10.0.2.1","key":null,"listen_port":"56002","peers":[]}}
result=0 (OE_OK)
[ SGX-LKL ] sgxlkl_enclave_init(ethread_id=0)
[[ SGX-LKL ]] sgxlkl_enclave_init(): enter
[[ SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave base=0x7fd000000000 size=64.0 GiB
[[ SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave heap base=0x7fd000b03000 size=1024.0 MiB end=0x7fd040b03000
[[ SGX-LKL ]] sgxlkl_enclave_init(): calling _dlstart_c()
[[ SGX-LKL ]] _register_enclave_signal_handlers(): Registering OE exception handler...
[[ SGX-LKL ]] __libc_init_enclave(): calling enclave_mman_init()
[[ SGX-LKL ]] __libc_init_enclave(): calling _lthread_sched_init()
[[ SGX-LKL ]] lkl_start_init(): calling register_lkl_syscall_overrides()
[[ SGX-LKL ]] lkl_start_init(): calling initialize_enclave_event_channel()
[[ SGX-LKL ]] lkl_start_init(): kernel command line: 'mem=32M console=hvc0'
[[ SGX-LKL ]] lkl_start_init(): Disk 0: Disk encryption: no
[[ SGX-LKL ]] lkl_start_init(): Disk 0: Disk is writable: yes
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() called
[ 0.000000] Linux version 5.3.0+ (AzDevOps@cisca24ba0002X9) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #1 Fri Aug 7 17:18:52 UTC 2020
[ 0.000000] memblock address range: 0x7fd03e882000 - 0x7fd040882000
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 8080
[ 0.000000] Kernel command line: mem=32M console=hvc0 virtio_mmio.device=256@0x1000000:1
[ 0.000000] Dentry cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory available: 32176k/32768k RAM
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS: 4096
[ 0.000000] lkl: irqs initialized
[ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000000] lkl: time and timers initialized (irq2)
[ 0.000003] printk: console [hvc0] enabled
[ 0.000003] pid_max: default: 4096 minimum: 301
[ 0.000003] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.000004] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.000015] devtmpfs: initialized
[ 0.014980] random: get_random_bytes called from 0x00007fd0006319de with crng_init=0
[ 0.014981] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.015487] futex hash table entries: 16 (order: -4, 384 bytes, linear)
[ 0.015487] xor: automatically using best checksumming function 8regs
[ 0.015489] NET: Registered protocol family 16
[ 0.018930] clocksource: Switched to clocksource lkl
[ 0.018932] NET: Registered protocol family 2
[ 0.019434] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.019434] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.019434] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.019434] TCP: Hash tables configured (established 512 bind 512)
[ 0.019937] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.019938] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.019940] NET: Registered protocol family 1
[ 0.019940] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x10000ff, IRQ 1.
[ 0.020443] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[ 0.021503] NET: Registered protocol family 38
[ 0.021503] io scheduler mq-deadline registered
[ 0.021504] io scheduler kyber registered
[ 0.116550] software IO TLB: mapped [mem 0x7fe2a5d69000-0x7fe2a9d69000] (64MB)
[ 0.120132] loop: module loaded
[ 0.128338] device-mapper: ioctl: 4.40.0-ioctl (2019-01-18) initialised: dm-devel@redhat.com
[ 0.138323] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[ 0.138879] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[ 0.139384] NET: Registered protocol family 10
[ 0.139387] Segment Routing with IPv6
[ 0.139388] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 0.141403] This architecture does not have kernel memory protection.
[ 0.141403] Run /init as init process
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() finished
[[ SGX-LKL ]] lkl_start_init(): creating LKL termination thread
[[ SGX-LKL ]] create_lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_start_init(): calling lkl_mount_virtual()
[[ SGX-LKL ]] lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_termination_thread(): Performed LKL syscall to get host task allocated (pid=36)
[[ SGX-LKL ]] lkl_start_init(): calling init_random()
[[ SGX-LKL ]] init_random(): Adding entropy to entropy pool
[ 0.143472] random: crng init done
[[ SGX-LKL ]] init_enclave_clock(): Setting enclave realtime clock
[[ SGX-LKL ]] wg0 has public key 09RO378oOw/vQVjwoMrIsao7lxjWvX+MYScNrz1qHwY=
[ 0.147519] virtio_blk virtio1: [vda] 10240 512-byte logical blocks (5.24 MB/5.00 MiB)
[[ SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vda", mnt="/mnt/vda", ro=0)
[ 0.150256] EXT4-fs (vda): mounted filesystem without journal. Opts:
[[ SGX-LKL ]] lkl_mount_disks(): Set working directory: /
[[ SGX-LKL ]] libc_start_main_stage2(): Calling app main: pthread_join-test
Creating worker thread (run=0)
Joining worker thread...
Thread joined (ret=0, run=0)
Creating worker thread (run=1000)
Joining worker thread...
Thread joined (ret=0, run=1000)
Makefile:29: recipe for target 'run-hw' failed
make: *** [run-hw] Terminated
make run-hw: TIMED OUT after 60 secs
Test run completed with EXIT CODE 124
Here is a captured stack trace with PR #751:
================================== run 163 ==================================
SGXLKL_VERBOSE=1 SGXLKL_KERNEL_VERBOSE=1 SGXLKL_ETHREADS=16 SGXLKL_TRACE_SIGNAL=1 /home/v-pepiet/devel/sgx-lkl/build/sgx-lkl-run-oe --hw-debug sgx-lkl-rootfs.img pthread_join-test
[ SGX-LKL ] nproc=8 ETHREADS=16 CMDLINE="mem=32M"
[ SGX-LKL ] HW TLS support: econf->fsgsbase=1
[ SGX-LKL ] Registering disk 0 (path='sgx-lkl-rootfs.img', mnt='/', [RW])
[ SGX-LKL ] No tap device specified, networking will not be available.
[ SGX-LKL ] get_signed_libsgxlkl_path... result=/home/v-pepiet/devel/sgx-lkl/build_musl/./libsgxlkl.so.signed
[ SGX-LKL ] oe_create_enclave...
[ SGX-LKL ] Enclave config: {"args":["pthread_join-test"],"clock_res":[{"resolution":"0000000000000001"},{"resolution":"0000000000000001"},{"resolution":"0000000000000000"},{"resolution":"00000000000$
0000"},{"resolution":"0000000000000001"},{"resolution":"00000000003d0900"},{"resolution":"00000000003d0900"},{"resolution":"0000000000000001"}],"cwd":"/","env":[],"esleep":"16000","espins":"500","ethrea$
s":"16","exit_status":"full","format_version":"1","fsgsbase":true,"host_import_env":["SGXLKL_TRACE_SIGNAL"],"hostname":"lkl","hostnet":false,"image_sizes":{"num_heap_pages":"262144","num_stack_pages":"1$
24"},"io":{"block":true,"console":true,"network":true},"kernel_cmd":"mem=32M","kernel_verbose":true,"max_user_threads":"256","mmap_files":"shared","mode":"hw_debug","mounts":[],"net_gw4":"10.0.1.254","n$
t_ip4":"10.0.1.1","net_mask4":"24","oe_heap_pagecount":"8192","root":{"key":"","key_id":null,"overlay":false,"readonly":false,"roothash":null,"roothash_offset":"0"},"stacksize":"524288","swiotlb":true,"$
ysctl":null,"tap_mtu":"0","verbose":true,"wg":{"ip":"10.0.2.1","key":null,"listen_port":"56002","peers":[]}}
result=0 (OE_OK)
[ SGX-LKL ] sgxlkl_enclave_init(ethread_id=0)
[[ SGX-LKL ]] sgxlkl_enclave_init(): enter
[[ SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave base=0x7f3000000000 size=64.0 GiB
[[ SGX-LKL ]] _sgxlkl_enclave_show_attribute(): enclave heap base=0x7f3000b04000 size=1024.0 MiB end=0x7f3040b04000
[[ SGX-LKL ]] sgxlkl_enclave_init(): calling _dlstart_c()
[[ SGX-LKL ]] _register_enclave_signal_handlers(): Registering OE exception handler...
[[ SGX-LKL ]] __libc_init_enclave(): calling enclave_mman_init()
[[ SGX-LKL ]] __libc_init_enclave(): calling _lthread_sched_init()
[[ SGX-LKL ]] lkl_start_init(): calling register_lkl_syscall_overrides()
[[ SGX-LKL ]] lkl_start_init(): calling initialize_enclave_event_channel()
[[ SGX-LKL ]] lkl_start_init(): kernel command line: 'mem=32M console=hvc0'
[[ SGX-LKL ]] lkl_start_init(): Disk 0: Disk encryption: no
[[ SGX-LKL ]] lkl_start_init(): Disk 0: Disk is writable: yes
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() called
[ 0.000000] Linux version 5.3.0+ (v-pepiet@msrc-cc06) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #1 Sat Aug 8 11:20:20 UTC 2020
[ 0.000000] memblock address range: 0x7f303e883000 - 0x7f3040883000
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 8080
[ 0.000000] Kernel command line: mem=32M console=hvc0 virtio_mmio.device=256@0x1000000:1
[ 0.000000] Dentry cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory available: 32176k/32768k RAM
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS: 4096
[ 0.000000] lkl: irqs initialized
[ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000000] lkl: time and timers initialized (irq2)
[ 0.001065] printk: console [hvc0] enabled
[ 0.001066] pid_max: default: 4096 minimum: 301
[ 0.001066] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.001067] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.002724] devtmpfs: initialized
[ 0.007131] random: get_random_bytes called from 0x00007f3000633541 with crng_init=0
[ 0.007131] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.007132] futex hash table entries: 16 (order: -4, 384 bytes, linear)
[ 0.007680] xor: automatically using best checksumming function 8regs
[ 0.007683] NET: Registered protocol family 16
[ 0.012099] clocksource: Switched to clocksource lkl
[ 0.013207] NET: Registered protocol family 2
[ 0.013759] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.013760] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.013760] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[ 0.013760] TCP: Hash tables configured (established 512 bind 512)
[ 0.014310] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.014310] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[ 0.014311] NET: Registered protocol family 1
[ 0.014312] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x10000ff, IRQ 1.
[ 0.014867] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[ 0.016524] NET: Registered protocol family 38
[ 0.016525] io scheduler mq-deadline registered
[ 0.016526] io scheduler kyber registered
[ 0.137802] software IO TLB: mapped [mem 0x7f481e886000-0x7f4822886000] (64MB)
[ 0.142935] loop: module loaded
[ 0.142937] device-mapper: ioctl: 4.40.0-ioctl (2019-01-18) initialised: dm-devel@redhat.com
[ 0.176863] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[ 0.177595] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[ 0.177598] NET: Registered protocol family 10
[ 0.178669] Segment Routing with IPv6
[ 0.178670] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 0.179313] This architecture does not have kernel memory protection.
[ 0.179313] Run /init as init process
[[ SGX-LKL ]] lkl_start_init(): lkl_start_kernel() finished
[[ SGX-LKL ]] lkl_start_init(): creating LKL termination thread
[[ SGX-LKL ]] create_lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_start_init(): calling lkl_mount_virtual()
[[ SGX-LKL ]] lkl_termination_thread(): enter
[[ SGX-LKL ]] lkl_termination_thread(): Performed LKL syscall to get host task allocated (pid=36)
[[ SGX-LKL ]] lkl_start_init(): calling init_random()
[[ SGX-LKL ]] init_random(): Adding entropy to entropy pool
[ 0.179869] random: crng init done
[[ SGX-LKL ]] init_enclave_clock(): Setting enclave realtime clock
[[ SGX-LKL ]] wg0 has public key 4InRpY9PIt9r/giqRcMggmJyu2T0F0jdOywXBHqOKng=
[ 0.186185] virtio_blk virtio1: [vda] 10240 512-byte logical blocks (5.24 MB/5.00 MiB)
[[ SGX-LKL ]] lkl_mount_disk(): lkl_mount_disk(dev="/dev/vda", mnt="/mnt/vda", ro=0)
[ 0.187878] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended
[ 0.188429] EXT4-fs (vda): mounted filesystem without journal. Opts:
[[ SGX-LKL ]] lkl_mount_disks(): Set working directory: /
[[ SGX-LKL ]] libc_start_main_stage2(): Calling app main: pthread_join-test
Creating worker thread (run=0)
Joining worker thread...
Thread joined (ret=0, run=0)
Creating worker thread (run=1000)
Joining worker thread...
Thread joined (ret=0, run=1000)
Creating worker thread (run=2000)
Joining worker thread...
Thread joined (ret=0, run=2000)
Creating worker thread (run=3000)
Joining worker thread...
Thread joined (ret=0, run=3000)
Creating worker thread (run=4000)
Joining worker thread...
Thread joined (ret=0, run=4000)
Creating worker thread (run=5000)
Joining worker thread...
Thread joined (ret=0, run=5000)
Creating worker thread (run=6000)
Joining worker thread...
Thread joined (ret=0, run=6000)
Creating worker thread (run=7000)
Joining worker thread...
Thread joined (ret=0, run=7000)
Creating worker thread (run=8000)
Joining worker thread...
Thread joined (ret=0, run=8000)
Creating worker thread (run=9000)
Joining worker thread...
Thread joined (ret=0, run=9000)
Creating worker thread (run=10000)
Joining worker thread...
Thread joined (ret=0, run=10000)
Creating worker thread (run=11000)
Joining worker thread...
Thread joined (ret=0, run=11000)
Creating worker thread (run=12000)
Joining worker thread...
Thread joined (ret=0, run=12000)
Creating worker thread (run=13000)
Joining worker thread...
Thread joined (ret=0, run=13000)
Creating worker thread (run=14000)
Joining worker thread...
Thread joined (ret=0, run=14000)
Creating worker thread (run=15000)
Joining worker thread...
Thread joined (ret=0, run=15000)
Creating worker thread (run=16000)
Joining worker thread...
Thread joined (ret=0, run=16000)
Creating worker thread (run=17000)
Joining worker thread...
Thread joined (ret=0, run=17000)
Creating worker thread (run=18000)
Joining worker thread...
Thread joined (ret=0, run=18000)
[ SGX-LKL ] Dumping thread stack traces from enclave...
[[ SGX-LKL ]] sgxlkl_debug_dump_stack_traces(): Dumping all stack traces from threads...
[[ SGX-LKL ]] =============================================================
[[ SGX-LKL ]] Stack traces for all lthreads:
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 1: tid=1 (7f3000b04520) [pthread_join-test]
[[ SGX-LKL ]] #0: 7f30005bfbe3 in lthread_join(...)
[[ SGX-LKL ]] #1: 7f300059a25e in __pthread_join(...)
[[ SGX-LKL ]] #2: 7f303daffa96 in <unknown>(...)
[[ SGX-LKL ]] #3: 7f3000532904 in libc_start_main_stage2(...)
[[ SGX-LKL ]] #4: 7f300053285e in __libc_start_main(...)
[[ SGX-LKL ]] #5: 7f303daff7b6 in <unknown>(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 2: tid=2 (7f3000b06c00) [vio-0]
[[ SGX-LKL ]] #0: 7f30005acac9 in vio_wait_for_host_event(...)
[[ SGX-LKL ]] #1: 7f30005acc99 in vio_enclave_process_host_event(...)
[[ SGX-LKL ]] #2: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 3: tid=3 (7f3000b06e10) [vio-1]
[[ SGX-LKL ]] #0: 7f30005acac9 in vio_wait_for_host_event(...)
[[ SGX-LKL ]] #1: 7f30005acc99 in vio_enclave_process_host_event(...)
[[ SGX-LKL ]] #2: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 4: tid=4 (7f3000b07000) [vio-2]
[[ SGX-LKL ]] #0: 7f30005acac9 in vio_wait_for_host_event(...)
[[ SGX-LKL ]] #1: 7f30005acc99 in vio_enclave_process_host_event(...)
[[ SGX-LKL ]] #2: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 5: tid=5 (7f3000b07210) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414fbd in schedule_idle(...)
[[ SGX-LKL ]] #8: 7f30000b0dea in do_idle(...)
[[ SGX-LKL ]] #9: 7f30000b1045 in cpu_startup_entry(...)
[[ SGX-LKL ]] #10: 7f3000412d2e in rest_init(...)
[[ SGX-LKL ]] #11: 7f300062867e in arch_call_rest_init(...)
[[ SGX-LKL ]] #12: 7f3000628a21 in start_kernel(...)
[[ SGX-LKL ]] #13: 7f300062930c in lkl_run_kernel(...)
[[ SGX-LKL ]] #14: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 6: tid=6 (7f3000b07450) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005d65f5 in timer_callback(...)
[[ SGX-LKL ]] #4: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 7: tid=7 (7f3000b07640) [kernel]
[[ SGX-LKL ]] #0: 7f30005bfa3b in lthread_join(...)
[[ SGX-LKL ]] #1: 7f30005d6369 in thread_exit(...)
[[ SGX-LKL ]] #2: 7f300008a70e in lkl_run_init(...)
[[ SGX-LKL ]] #3: 7f300010ffcf in search_binary_handler(...)
[[ SGX-LKL ]] #4: 7f30001105d5 in __do_execve_file(...)
[[ SGX-LKL ]] #5: 7f300011070d in do_execve(...)
[[ SGX-LKL ]] #6: 7f300008a0b2 in run_init_process(...)
[[ SGX-LKL ]] #7: 7f3000412d6c in kernel_init(...)
[[ SGX-LKL ]] #8: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #9: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 8: tid=8 (7f3000b07830) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a95b5 in kthreadd(...)
[[ SGX-LKL ]] #9: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #10: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 9: tid=9 (7f3000b07a20) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 10: tid=10 (7f3000b07c10) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 11: tid=11 (7f3000b07e00) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 12: tid=12 (7f3000b07ff0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 13: tid=13 (7f3000b081e0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000ac705 in smpboot_thread_fn(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 14: tid=14 (7f3000b083d0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f3000255196 in devtmpfsd(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 15: tid=15 (7f3000b085c0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 16: tid=34 (7f3000b087b0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 17: tid=35 (7f3000b089a0) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 18: tid=36 (7f3000b08b90) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000ee5c4 in kswapd(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 19: tid=37 (7f3000b08d80) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 20: tid=38 (7f3000b08f70) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f300023f285 in khvcd(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 21: tid=39 (7f3000b09160) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 22: tid=40 (7f3000b09350) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 23: tid=41 (7f3000b09590) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008b62f in idle_host_task_loop(...)
[[ SGX-LKL ]] #6: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #7: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 24: tid=42 (7f3000b09780) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f30005d9bfc in lkl_termination_thread(...)
[[ SGX-LKL ]] #6: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 25: tid=43 (7f3000b09c20) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 26: tid=44 (7f3000b09a30) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 27: tid=45 (7f3000b09e60) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a3d73 in worker_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 28: tid=46 (7f3000b0a050) [kernel]
[[ SGX-LKL ]] #0: 7f30005b30e3 in __do_futex_sleep(...)
[[ SGX-LKL ]] #1: 7f30005b3173 in futex_wait(...)
[[ SGX-LKL ]] #2: 7f30005b32f3 in enclave_futex_timedwait(...)
[[ SGX-LKL ]] #3: 7f30005b333d in enclave_futex_wait(...)
[[ SGX-LKL ]] #4: 7f30005d5f3f in sem_down(...)
[[ SGX-LKL ]] #5: 7f300008aa93 in __switch_to(...)
[[ SGX-LKL ]] #6: 7f3000414cfd in __schedule(...)
[[ SGX-LKL ]] #7: 7f3000414f22 in schedule(...)
[[ SGX-LKL ]] #8: 7f30000a4926 in rescuer_thread(...)
[[ SGX-LKL ]] #9: 7f30000a8a44 in kthread(...)
[[ SGX-LKL ]] #10: 7f300008a928 in thread_bootstrap(...)
[[ SGX-LKL ]] #11: 7f30005be4fa in _exec(...)
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 29: tid=19027 (7f3000b0a220) [pthread_join-test]
[[ SGX-LKL ]] #0: 7f30005be551 in _exec(...)
[[ SGX-LKL ]] =============================================================
2020-08-08T21:00:04.000000Z [(H)ERROR] tid(0x7f4824681300) | :OE_ENCLAVE_ABORTING [/home/v-pepiet/devel/sgx-lkl/openenclave/host/calls.c:oe_call_enclave_function_by_table_id:91]
[ SGX-LKL ] FAIL: Aborting after stack trace dump
This is really helpful @prp!
Most kernel threads seem to be sleeping and the main thread tid=1 has joined on the child thread.
The child thread's stacktrace seems interesting though -
[[ SGX-LKL ]] -------------------------------------------------------------
[[ SGX-LKL ]] 29: tid=19027 (7f3000b0a220) [pthread_join-test]
[[ SGX-LKL ]] #0: 7f30005be551 in _exec(...)
@vtikoo Isn't the problem that we're not releasing the lt lock after _exec has run? Normally we would release the lock atomically by providing a callback when returning control to the scheduler.
Hmm, no, that seems fine.
Ok, it seems that the parent thread is not acquiring the lthread lock when it's destroying the child thread. The child thread may therefore still be exiting concurrently on another ethread. I'm testing a fix and so far had 1000 successful runs of the pthread_join test.