GrieferAtWork / KOSmk3

Chaos operating system 300

KOS (Hooby) Operating System & Kernel

The third rendition of the KOS Operating System & Kernel, this time with a lot more emphasis on integration, as well as streamlining. Additionally, many optional features have been implemented this time around, including actually working true SMP and IPI, support for the x86 sysenter instruction, and making use of APIC interrupt timers for preemption (all things that KOSmk2 couldn't do)

Much emphasis has also been put on implementing posix signal() behavior correctly without any shortcuts. Similarly, the kernel now implements much better support for termios flags, supporting numerous options that were completely unsupported before. A lot of work was also put into speeding up system calls such as fork(), making use of lazy copy-on-write, rather than duplicating data immediatly.

The most important difference between this and KOSmk2 is probably the idea that initially convinced me to start over from scratch, that idea being: Exceptions. But not just any exceptions. Oh no. I'm not using some C++ compiler now. It's all still 100% pure C code, making use of inline assembly as well as DWARF's CFI meta-instruction set, alongside a lot of preprocessor magic.

Also, a large number of the core concepts already found in KOSmk2 have been re-invisioned using the new conceptual facilities that, among other things, completely remove functions such as task_crit(), now instead relying on RPC functionality that could not be seen anywhere in KOSmk2, and allowing for much easier to use scheduling semantics, as well as new mechanisms for defining per-cpu, per-task or per-vm variables, rather than having to cram everything into one humonguous and overcomplicated control structure. E.g: to terminate a thread, you just schedule and RPC function that throws an E_THREAD_EXIT exception.

KOSmk3 is a much more stable platform than its predecessors, relying less than ever on code that hasn't been written by me personally, including my very own heap implementation with support for GC-based memory leak detection that scans registers, global data, as well as stack memory for pointers in order to determine what memory is actually reachable, similar to what the linux kernel does (presumably), leading to my bold claim that the KOS kernel should currently be leakless.

Like everything, this is another one-person project, with development (of this rendition) having started on 10.02.2018.

Chaos|KOS - You probably got here through the definition of chaoticity (which totally is a real word).

State

• Able to run busybox (using the same binary that was originally compiled for KOSmk2)

Features

• i486+ (Yes. Your machine can't be older than time itself...)
  • Protected-mode ring #3
• Exceptions
  • try-catch-finally
  • Stack unwinding into user-space (Exceptions thrown by the kernel can be caught by user-applications)
  • Highly robust kernel, even in the face of faulty kernel code (of which there probably is a lot)
  • Zero-effort exceptions using guard-descriptors invoked by unwinding the stack with the help of CFI instructions. In other words, I'm not going the windows route of the uglyness that is SEH and its stack-based linked list.
  • Implemented using pure, unmodified gcc (these aren't c++ exceptions, which is why I do actually have a finally statement)
• QEMU
• multiboot/multiboot2
• Paging
  • Page directory self-mapping
  • %cr3 is only changed during preemption
  • dynamic memory
  • ALLOA (ALLocateOnAccess)
  • LOA (LoadOnAccess)
  • COW (CopyOnWrite)
  • memory-mapped files for reading/writing
• Heap
  • GC-based memory leak detection
  • heap_alloc()
  • kmalloc()
  • malloc()
  • New malloc functions such as memalign_offset()
• syscall
  • Highly linux-compatible using int $0x80 and same ids/registers
  • Support for system call restarting (SA_RESTART)
  • Support for the x86 sysenter instruction
  • Support for system calls using page faults
    • It's quite useful and allows signal handlers to return by executing the sigreturn system calls without the need of any trampoline code
• Unix-compliant user-space interfaces/APIs
  • ANSI-compliant Terminal (needs a rewrite though. - I just copied the old one over)
  • fork()/exec()/wait()/pipe()
  • mmap()/munmap()/mremap()/brk()/sbrk()
    • Full support for file to memory mappings
  • signal()/raise()/kill()/sigprocmask()
    • Terminate/suspend/resume support for SIGKILL, SIGSTOP, SIGCONT
    • Signal-based exception handling (SIGSEGV, etc.)
    • Greatly improved conformance to POSIX behavior.
    • Fully functioning, bash-compatbile job control using SIGTTIN and SIGTTOU
  • open()/read()/write()/lseek()
  • fcntl()/ioctl()/openpty()
  • mount()/umount()
  • main()/argc/argv/environ
  • dlopen()/dlclose()/dlsym()
    • Still needs an emulation library, but all the system calls are already there
  • clone()/futex()/exit_group()
    • Linux-compatible multi-threading support
  • Can run busybox
• Custom-made Libc
  • Binary (ABI) compatibility with both MSVCrt and GLibC
    • For the most part. Compatibility will further be improved over time.
  • System headers designed with portability and clean namespaces
    • Use of <features.h> emulating GLibC, allowing for control of available features using *_SOURCE macros
    • Option to warn about use of non-portable functions
    • Redundant fallbacks of (some) KOS-specific extensions allows for use on other platforms
    • You can actually use KOS's /include folder to spoof MSVCrt and get a much more portable library at the same time!
  • Support for future-proof 64-bit time_t types and functions
• multitasking/scheduler
  • PID, including PID namespaces
  • CLONE_* flags compatibility allows for individual thread components to either be shared with other threads, or be unique to that thread alone.
  • Multi-core SMP support
  • Use of hlt when no threads are actively running
• ELF binaries/libraries (no extension / .so)
• Disk I/O
  • ATA driver supporting PIO28, PIO48 and CHS addressing
  • Filesystem
    • FAT-12/16/32
    • /dev
• Drivers
  • PS/2 driver api
    • Using a miniature interpreter, programs can be run for comunicating with devices.
    • Support for an arbitrary number of connected data handles.
    • PS/2 keyboard driver
      • Supports all scansets (#1, #2 and #3)
      • Support for LEDs, as well as various KOS-specific keyboard ioctls
• Monolithic kernel design with (untested & unused) support for modules.

Fixes / Improvements when compared against KOSmk2

• Fix Unicode conversion
  • I had no idea how UTF-8 actually worked when I wrote the KOSmk2 API
• POSIX signal support
  • KOSmk2 only half-a$$ed its implementation to the point where busybox started working, at which point I simply stopped implementing what was left, or fix what I did wrong.
• Return errors when invoking system calls with unsupported flags
  • KOSmk2 used to just ignore invalid flags
• Complete conceptual re-work of the core concepts driving the entire kernel using exceptions
• Use of exceptions to deliver error-states absurdly reduces the number of check that would otherwise need to be done to deal with error cases (imagine an malloc() that never returns NULL, and what you get is Xmalloc(), or any of the underlying heap-functions)
• Keep arbitrary page directory switches to an absolute minimum
  • Physical memory allocators no longer use a linked list, but rather a large bitset from which pages are allocated using atomic instruction. Aka. a lock-less page_malloc() function
  • Also make use of the x86 GLOBAL flag in page directories, meaning that memory access remains as fast as possible, even after page directories have been switched.
• The new KOS has shown a tendency towards becoming a monolithic kernel, despite the fact that facilities for kernel modules already exists, though have yet to be tested.
• Fixed many conceptual problems related to how the old KOS dealt with signal delivery to tasks in different situations, including numerous situations that could have resulted in dead locks caused by something as simple as pressing a button
  • Introduction of the concept of async-safety, as well as highly detailed rules to-be followed for code running in the context of interrupts and the like.
• Actually working SMP support (not just conceptually), including IPIs and RPCs.
• Fully redesigned filesystem with almost all emphasis put on caching
  • KOSmk2 though it was a good idea to not include any way of caching directory contents, or have some way of loading INodes using only their number
  • Implementation of a pread() and pwrite() operator as basis of any INode now negates the need of intermediate and file streams (the FAT driver simply caches disk positions of file chunks)
  • Much better support for O_* flags, such as O_NONBLOCK
• Using exceptions, dealing with hardware-generated exceptions such as divide-by-zero, or overflow is possible

Interesting, but potentially useless functions

• VIO
  • By analyzing assembly surrounding the return address of a #PF interrupt, as well as emulating a considerable portion of the X86 instruction set, the KOS kernel is able to emulate behavior that is usually associated with memory-mapped devices
  • Basically, KOS is able to create memory mappings that allow high-level read() / write() operators to be invoked whenever any piece of code tries to access that memory.
  • It's somewhat similar to what was discussed over here: https://forum.osdev.org/viewtopic.php?f=15&t=22521
  • For an example use, look at include/kos/ushare.h under struct ushare_procinfo

Known problems

• You can't just compile the kernel with -O3. That'll break exception support due to GCC re-arranging code.
  • Related to this, I have yet to fully trace down which optimization flags actually cause problems.
  • As far as I can tell, the following optimization flags cannot be enabled:
    • -fcrossjumping, -freorder-blocks, -freorder-blocks-and-partition

Planned

• Rewrite the VGA terminal to run in kernel-space as a real TTY supporting job control (CTRL + ALT + F1-F12)
• Support for DMA in the ATA driver
• Support for USB
• Support for bios-calls done through vm86 (including a fallback disk driver)
• Support for /proc
• Support for /sys
• A user-space pthread library (Although threading can already be done using fork() and clone())
• TCP/IP-stack
• Start testing the thing on real hardware again
• WLAN support for "Atheros AR2427 Wireless" (That one's inside my test machine)

Build Requirements

• Havn't gotten around to really setting up a new toolchain. - You're on your own till then...