UPX compressed arm64/amd64/x86_64 executable crashes on macOS Ventura 13.0
chchwy opened this issue · comments
This issue tracker is ONLY used for reporting bugs.
Please use stackoverflow for supporting issues.
What's the problem (or question)?
The UPX compressed amd64 executable crashes on macOS 13.0.
I made a small CMake project which merely print out a few lines of text for reproducing the crash.
Here are the commands I used to compress the executable. Then it crashed immediately after I ran the compressed one.
% upx -vk ./HelloUPX4 -o./HelloUPX4-compressed
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2020
UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
File size Ratio Format Name
-------------------- ------ ----------- -----------
82304 -> 12304 14.95% macho/amd64 HelloUPX4-compressed
Packed 1 file.
% ./HelloUPX4-compressed
zsh: segmentation fault ./HelloUPX4-compressed
%lldb
% lldb ./HelloUPX4-compressed
(lldb) target create "./HelloUPX4-compressed"
Current executable set to '/Users/matt/HelloUPX4/build/Debug/HelloUPX4-compressed' (x86_64).
(lldb) run
Process 14499 launched: '/Users/matt/HelloUPX4/build/Debug/HelloUPX4-compressed' (x86_64)
Process 14499 stopped
* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x21090c000)
frame #0: 0x000000021090c000
error: memory read failed for 0x21090c000
Target 0: (HelloUPX4-compressed) stopped.
(lldb) exitWhat should have happened?
It should print the same text as before it was compressed.
This is the output when it is not compressed.
% lldb "HelloUPX4"
(lldb) target create "HelloUPX4"
Current executable set to '/Users/evatu/Work/HelloUPX4/build/Debug/HelloUPX4' (x86_64).
(lldb) run
Process 14516 launched: '/Users/evatu/Work/HelloUPX4/build/Debug/HelloUPX4' (x86_64)
Hello UPX4!
Done.
Process 14516 exited with status = 0 (0x00000000)
Do you have an idea for a solution?
Actually no, but It was working on macOS 12.
How can we reproduce the issue?
I made a tiny CMake project with 16 lines of code for this bug report. It is configured to generate an amd64 executable. (Not arm64)
- Clone the project at https://github.com/chchwy/upx-macos13-crash and go to the folder
mkdir buildcd buildcmake .. -G Xcodecmake --build .upx -vk ./Debug/HelloUPX4 -o./Debug/HelloUPX4-compressed./Debug/HelloUPX4-compressed<= crashed here
Please tell us details about your environment.
- UPX version used (
upx --version): 3.96 (I tried the latest devel4 branch and it doesn't work either) - Host Operating System and version: macOS 13.0 Ventura
- Host CPU architecture: arm64
- Target Operating System and version: same
- Target CPU architecture: amd64 (or called x86_64)
I fear we do not have enough info to support macOS 13 at this time - has Apple released any current kernel/libc/ld/... source code yet?
Info from #613 : macOS 13 source code has been released https://github.com/apple-oss-distributions/xnu
Unsurprislingly arm64 macos13 also doesn't work.
@markus-oberhumer The comment ... doesn't work does not help. Please specify what you tried, and what happened: how did it not work? It is much better to say, "I ran steps 1 through 7 of the Description using MacOS 13.0, and got SIGSEGV." And for SIGSEGV it is even better to run under lldb, then at the crash use lldb to report the pc, register contents, and traceback. And of course Apple's vaunted truss or dtrace is totally useless because it demands a complete run-time environment. "Anything acceptable to execve" is too simple.
$ uname -a
Darwin macm1.oberhumer.com 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64
$ wget https://github.com/upx/upx/releases/download/v4.0.0/upx-4.0.0-src.tar.xz
$ tar -xJf upx-4.0.0-src.tar.xz
$ make -C upx-4.0.0-src build/release-clang
$ make -C upx-4.0.0-src/build/release-clang test
@chchwy On Apple M1 hardware, then MacOS 13.0 (Ventura) with Xcode 14.1 (Nov.1, 2022) is not ready for prime time:
## Download from Apple and install Command_Line_Tools_for_Xcode_14.1.dmg
## Then:
% cmake .. -G Xcode
zsh: command not found: cmake
## Download and install HomeBrew:
% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
[[snip]]
% brew --version
Homebrew 3.6.10
Homebrew/homebrew-core (git revision 3e28f99040e; last commit 2022-11-13)
% brew install cmake
% cmake .. -G Xcode
CMake Error:
Xcode 1.5 not supported.
% cmake --version
cmake version 3.24.3
Suggestions?
@markus-oberhumer Your report omitted the observed behavior, so is not reproducible.
Digging deeply, I see
% lldb /Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed
(lldb) target create "/Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed"
Current executable set to '/Users/jreiser/upx-4.0.0-src/build/release-clang/upx.packed' (arm64).
(lldb) process launch -s -- --version
error: Bad executable (or shared library)
So probably the error is ENOEXEC or EBADEXEC. What you see?
Did some research on my intel x86-64 model.
crashed in "call upx_main"
bugs in "amd64-darwin.macho-fold.h":
Two address calls sub_369. the second call crashed in sub_369:
That's all i can do.
Thanks.
Did some research on my intel x86-64 model. crashed in "call upx_main" bugs in "amd64-darwin.macho-fold.h": Two address calls sub_369. the second call crashed in sub_369:
That's all i can do. Thanks.
@chchwy On Apple M1 hardware, then MacOS 13.0 (Ventura) with Xcode 14.1 (Nov.1, 2022) is not ready for prime time:
## Download from Apple and install Command_Line_Tools_for_Xcode_14.1.dmg ## Then: % cmake .. -G Xcode zsh: command not found: cmake ## Download and install HomeBrew: % /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" [[snip]] % brew --version Homebrew 3.6.10 Homebrew/homebrew-core (git revision 3e28f99040e; last commit 2022-11-13) % brew install cmake % cmake .. -G Xcode CMake Error: Xcode 1.5 not supported. % cmake --version cmake version 3.24.3Suggestions?
Run the following command before cmake. Please specify the Xcode.app path to the one you installed.
sudo /usr/bin/xcode-select --switch /Applications/Xcode-14.1.0.app
Probably you also need to open the Xcode app for the license accepting stuff for the first time.
We're seeing the same behavior on Google Closure Compiler: google/closure-compiler-npm#265
Oddly enough, an amd64 binary run on a M1 Mac via Rosetta does not exhibit the issue. Only when the binary is run on a non-M1 Mac does this come up.
Info from #613 : macOS 13 source code has been released https://github.com/apple-oss-distributions/xnu
In addition to my effort using upx/src/stub/tools/macho-snip/macho-snip.c, which is somewhat bottom-up (find a minimal runnable executable by starting with /bin/date, then remove Macho_command from it, one-by-one, trying to figure out why E_NOEXEC, E_BADEXEC, etc.) I am also trying a somewhat top-down approach. Start with the xnu source, compile it, gather all *.o into a *.a archive library, use that library with a user-mode main program that just does execve(path, args, envp), and use lldb to trace execution in user mode of the subroutines from xnu. The hope is to gather more clues about what is needed, and why, for a packed-by-upx file to execute successfully.
But I encounter problems when I try https://github.com/apple-oss-distributions/xnu.git. Starting with commit 5c2921b07a2480ab43ec66f5b9e41cb872bc554f of Tue Oct 18 20:56:54 2022 +0000 whose log says xnu-8792.41.9,
I encounter a missing file
xnu/libkern/os/log.c:33:10: fatal error: 'os/firehose_buffer_private.h' file not found
#include <os/firehose_buffer_private.h>
xnu/libkern/c++/OSKext.cpp:38:10: fatal error: 'os/firehose_buffer_private.h' file not found
#include <os/firehose_buffer_private.h>
Searching the net, I find a plausible substitute https://opensource.apple.com/source/libdispatch/libdispatch-1271.100.5/os/firehose_buffer_private.h which I download as xnu/libkern/os/firehose_buffer_private.h. But firehose_buffer_private.h still cannot be found by log.c or OSKext.cpp, even after starting over via make clean; make. Has anybody actually built xnu?
(The xnu source also draws many hundreds of complaints from -Wunused-but-set-variable.)
Adding here - maybe helpful ;) parts of the segfault report on Ventura 13.2
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------
Incident Identifier: 354490EB-BA0E-4E82-9A4D-A72B17D36B48
CrashReporter Key: D0826634-F6DF-2191-09BF-81297CC113F0
Hardware Model: MacBookPro16,1
Process: autocmt [37949]
Path: /Users/USER/Downloads/*/autocmt
Identifier: autocmt
Version: ???
Code Type: X86-64 (Native)
Role: Unspecified
Parent Process: zsh [37456]
Coalition: com.googlecode.iterm2 [82571]
Responsible Process: iTerm2 [37452]
Date/Time: 2023-03-14 16:42:02.2474 +0200
Launch Time: 2023-03-14 16:42:01.9276 +0200
OS Version: macOS 13.2 (22D49)
Release Type: User
Report Version: 104
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x00000000000000c8
Exception Codes: 0x0000000000000001, 0x00000000000000c8
VM Region Info: 0xc8 is not in any region. Bytes before following region: 140737488043832
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START
--->
shared memory 7ffffffb4000-7ffffffb5000 [ 4K] r-x/r-x SM=SHM
Termination Reason: SIGNAL 11 Segmentation fault: 11
Terminating Process: exc handler [37949]
Highlighted by Thread: 0
Backtrace not available
No thread state (register information) available
Binary Images:
0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???
Error Formulating Crash Report:
_dyld_process_info_create failed with 30
dyld_process_snapshot_get_shared_cache failed
Failed to create CSSymbolicatorRef - corpse still valid ¯\_(ツ)_/¯
thread_get_state(PAGEIN) returned 0x10000003: (ipc/send) invalid destination port
thread_get_state(EXCEPTION) returned 0x10000003: (ipc/send) invalid destination port
thread_get_state(FLAVOR) returned 0x10000003: (ipc/send) invalid destination port
EOF```
Any updates to this? We are experiencing the same issue
same issue
command: upx --best --lzma main
outputs:
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2020
UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
File size Ratio Format Name
-------------------- ------ ----------- -----------
100291424 -> 20709392 20.65% macho/amd64 main
Packed 1 file.
run command: ./main
outputs:
zsh: segmentation fault ./main
OS info:
Darwin B-P59QMD6R-2102.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64
@wenxuwan Please run under the debugger lldb, then report the faulting instruction, faulting address, contents of all general registers, and the process memory map. This information will be much more useful than just "segmentation fault".
@wenxuwan Please run under the debugger lldb, then report the faulting instruction, faulting address, contents of all general registers, and the process memory map. This information will be much more useful than just "segmentation fault".
wenxuwan@B-P59QMD6R-2102 main % lldb ./main
(lldb) target create "./main"
Current executable set to '/Users/wenxuwan/code/mosn/cmd/mosn/main/main' (x86_64).
(lldb)
Current executable set to '/Users/wenxuwan/code/mosn/cmd/mosn/main/main' (x86_64).
(lldb) run
Process 67740 launched: '/Users/wenxuwan/code/mosn/cmd/mosn/main/main' (x86_64)
Process 67740 stopped
- thread 1, stop reason = EXC_BAD_ACCESS (code=1, address=0xc8)
frame #0: 0x00007ff80a8dced4
-> 0x7ff80a8dced4: movl 0xc8(%rbx), %eax
0x7ff80a8dceda: testl %eax, %eax
0x7ff80a8dcedc: je 0x7ff80a8dcee8
0x7ff80a8dcede: movq (%r12), %rcx
Target 1: (main) stopped.
Glad to see I'm not the only one suffering this issue, and yes, as mentioned by others, runs fine on M1/2 via Rosetta 2.
Perplexing.
What can be done from this side ( as a C/C++ developer ) to try help resolve this Ventura induced issue ?
@inflex 0) Locate and give the URL of any documentation or other info available on changes that Ventura made with regard to runtime permissions, safety or security, including "just-in-time" (JIT) code generation.
- Produce and post (upload) the smallest executable main program which suffers the problem. In plain-C this might be something like
#include <stdlib.h>
int const x[5000] = {1,2,3}; /* this is obviously compressible */
int func(char const *arg)
{
return x[atoi(arg)];
}
int main(int argc, char *argv[])
{
return func(argv[1]); /* attempt to defeat super-optimizing compiler */
}
- Run
otool -hlon the uncompressed binary executable program; post the output. - Compress using upx default method (not lzma).
- Run
otool -hlon the compressed prgram; post the output. - Run the compressed program under debugger
lldb:
$ lldb a.out.upx
(lldb) process launch -s # load program into new memory image, but do not execute yet
(lldb) x/8xg $sp # show initial stack contents
(lldb) info reg # show initial register contents
(lldb) continue
When the error appears, then print the registers, print the pc and instruction at the point of error, print several instructions before and after the point of error, print the top 64 bytes of the stack, show the layout of the process address space (run vmstat in another Terminal)
We really need to fix the macOS 13 issue - it's 7 months since Apple released source code. @jreiser Do you want help?
Yes, I want help. Note that Apple's source code 8792.82.1 omits at least one #include file, does not compile (the directions in the README.md produce several hundred errors, including obvious security bugs in the implementation of IPv6 detected by clang++ at compile time). And after I fixed (or worked-around the compile errors), then it does not load: many undefined externals. Furthermore, if a main program uses JIT (Just-In-Time) code (which UPX of a main program does) then the .exe must have some special permissions enabled in some Resources, which are typically specified in the SOURCE of the app. (There's a brief explanation somewhere in a Comment to an Issue here, but I cannot find it right now.) So Apple really wants UPX not to work, because [paraphrasing] "MacOS cannot guarantee the integrity of the code of this app." Furthermore, Apple provides no way for UPX to ask, "OK, I've decompressed everything, please do the MacOS integrity checking now."
See #434 (comment) for the plist and entitlement protocol [or what it once was.]
https://stackoverflow.com/questions/39863112/what-is-required-for-a-mach-o-executable-to-load has some comments (https://stackoverflow.com/posts/42399119/revisions of April 2022) that appear to be relevant.
My strategies include a Comment three months ago (March 11) in this Issue #612 (comment) .
@jreiser I'm not surprised that rebuilding the macos kernel won't work, but I had hoped that we find enough info in the source code to understand what is going on. Did you have a closer look?
The initial complaint (HelloUPX4) of this Issue has been fixed in 8db62b0 (tip of branch devel4); so many compressed main programs that run on MacOS 12 and below, but used to fail on MacOS 13, now run correctly. Artifacts have been posted to https://github.com/upx/upx/actions/runs/5249192624 . Working on the CI complaint now.
(Edit: you must re-compress using the newest UPX on devel4.)
The initial complaint (HelloUPX4) of this Issue has been fixed in 8db62b0 (tip of branch devel4); so many compressed main programs that run on MacOS 12 and below, but used to fail on MacOS 13, now run correctly. Artifacts have been posted to https://github.com/upx/upx/actions/runs/5249192624 . Working on the CI complaint now.
I tried a simple "hello world" application and compiled it with zig cc for arm64/macos. The uncompressed file runs smoothly on a M2 MacMini Pro, the compressed crashes with SIGKILL again. UPX is git-8db62b downloaded from CI artifacts as proposed.
@teras Please give details about the crash. Run under lldb:
$ lldb hello_app
(lldb) process launch -s # load the address space, but suspend execution (-s: do not start)
(lldb) continue # start
SIGKILL
(lldb) x/9i $pc-4*4 # faulting instruction and context
(lleb) x/8xg $sp # top-of-stack contents
Also please confirm the OS version.
Here is the result. It seems it thinks it's a bad executable?
hello_app is the uncompressed app and I run it as a reference. hello_app.upx is the compressed app.
also uname -a produces
Darwin macmini 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:53 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6020 arm64
teras@macmini ~/target> lldb hello_app
(lldb) target create "hello_app"
Current executable set to '/Users/teras/target/hello_app' (arm64).
(lldb) process launch -s
Process 22625 stopped
* thread #1, stop reason = signal SIGSTOP
frame #0: 0x0000000100014a40 dyld`_dyld_start
dyld`:
-> 0x100014a40 <+0>: mov x0, sp
0x100014a44 <+4>: and sp, x0, #0xfffffffffffffff0
0x100014a48 <+8>: mov x29, #0x0
0x100014a4c <+12>: mov x30, #0x0
Target 0: (hello_app) stopped.
Process 22625 launched: '/Users/teras/target/hello_app' (arm64)
(lldb) continue
Process 22625 resuming
Hello
Process 22625 exited with status = 0 (0x00000000)
(lldb) ^D
teras@macmini ~/target>
teras@macmini ~/target>
teras@macmini ~/target> lldb hello_app.upx
(lldb) target create "hello_app.upx"
Current executable set to '/Users/teras/target/hello_app.upx' (arm64).
(lldb) process launch -s
error: Bad executable (or shared library)
(lldb) continue
error: Command requires a current process.
(lldb)
@teras Thank you for testing. Was this on Monterey. Ventura, or Sonoma?
@jreiser Ventura
# sw_vers
ProductName: macOS
ProductVersion: 13.5
BuildVersion: 22G74
I'm also having problem with M1 running macOS 13.5 and upx 4.0.2. I have a universal exe that I want to pass through upx, but it seems like upx would corrupt the arm64 portion of the binary.
file myexe
myexe: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
myexe (for architecture x86_64): Mach-O 64-bit executable x86_64
myexe (for architecture arm64): Mach-O 64-bit executable arm64
❯ upx --color --overlay=skip -vk ./myexe -o compressed
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2023
UPX 4.0.2 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 30th 2023
File size Ratio Format Name
-------------------- ------ ----------- -----------
133366 -> 24592 18.44% macho/fat compressed
Packed 1 file.
❯ file myexe
myexe: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
myexe (for architecture x86_64): Mach-O 64-bit executable x86_64
myexe (for architecture arm64): Mach-O 64-bit executable arm64
❯ file compressed
compressed: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
compressed (for architecture x86_64): Mach-O 64-bit executable x86_64
compressed (for architecture arm64): empty
And running the compressed binary fails with malformed Mach-o file: ./compressed.
should I open a new issue?
@billyzs As you can see by the task list for UPX v4.2.0 #686, MacOS has been erecting barriers that impede execution of UPX-compressed executables. You should open a new issue only if the single arm64 main program, extracted from the "fat" executable by lipo, is rejected for a new reason when attempting to execute.
So the first (huge) blocker is that all code is required to be signed on arm64 Mac. When you create the compressed binary, the code is not signed:
faye@FayesMaxBookPro hello % upx --version
upx 4.1.0-devel.21+git-7636abc1
UCL data compression library 1.03
zlib data compression library 1.3.0.1-motley
LZMA SDK version 4.43
doctest C++ testing framework version 2.4.11
Copyright (C) 1996-2023 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2023 Laszlo Molnar
Copyright (C) 2000-2023 John F. Reiser
Copyright (C) 2002-2023 Jens Medoch
Copyright (C) 1995-2023 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
Copyright (C) 2016-2023 Viktor Kirilov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.
faye@FayesMaxBookPro hello % upx hello -o compressed
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2023
UPX git-7636ab Markus Oberhumer, Laszlo Molnar & John Reiser Aug 8th 2023
File size Ratio Format Name
-------------------- ------ ----------- -----------
33666 -> 16400 48.71% macho/arm64 compressed
Packed 1 file.
WARNING: this is an unstable beta version - use for testing only! Really.
faye@FayesMaxBookPro hello % codesign -dv compressed
compressed: code object is not signed at all
faye@FayesMaxBookPro hello % codesign -s - compressed
faye@FayesMaxBookPro hello % codesign -dv compressed
Executable=/Users/faye/Source/hello/compressed
Identifier=compressed-c88db7b0b9074f51e17b9e700eb173b437f94dc6
Format=Mach-O thin (arm64)
CodeDirectory v=20100 size=324 flags=0x2(adhoc) hashes=5+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
faye@FayesMaxBookPro hello % file hello
hello: Mach-O 64-bit executable arm64
faye@FayesMaxBookPro hello % file compressed
compressed: Mach-O 64-bit executable arm64
So that's one reason why the code doesn't work.
Secondly, it took a dynamic executable and turned it into a static one.
faye@FayesMaxBookPro hello % otool -h hello
hello:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777228 0 0x00 2 17 1056 0x00200085
faye@FayesMaxBookPro hello % otool -h compressed
compressed:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777228 0 0x00 2 6 616 0x00200001
faye@FayesMaxBookPro hello % lldb hello
(lldb) target create "hello"
Current executable set to '/Users/faye/Source/hello/hello' (arm64).
(lldb) run
Process 16891 launched: '/Users/faye/Source/hello/hello' (arm64)
Hello world
Process 16891 exited with status = 0 (0x00000000)
(lldb) quit
faye@FayesMaxBookPro hello % lldb compressed
(lldb) target create "compressed"
Current executable set to '/Users/faye/Source/hello/compressed' (arm64).
(lldb) run
error: Bad executable (or shared library)
does this information give you any extra detail?
When you create the compressed binary, the code is not signed:
This is intended and correct. The developer of the input to compression should (must) sign the output from compression.
it took a dynamic executable and turned it into a static one.
If the compressed static output runs, then you should rejoice for partial success. One step at a time.
Apple is deliberately, on purpose, doing everything it can to prevent UPX from working, under the banner of "secure execution". Apple's goal is "no code runs unless it has been inspected and approved by Apple." For the moment Apple must allow arbitrary executables produced by clang commandline to run [in some environments], and also delegates some portion of its approval power [for execution in some environments] to registered Developers who have certificates. But as soon as it can figure out how, then Apple will ban "bare" code produced by clang, and force every compiled binary executable to be produced only by Xcode or Swift or other Apple-controlled tools.
does this information give you any extra detail?
No. It does not identify any specific item (or collection of items) whose correction (with reasons why, and pointers to how) would enable execution. It's like saying just: "Your code has a bug. You lose; I quit."
@fayep The calculus of MacOS code signing (how code signing interacts with everything else) contains many surprises.
% codesign -d -v /bin/date ## /bin/date is signed by Apple
Executable=/bin/date
Identifier=com.apple.date
Format=Mach-O universal (x86_64 arm64e)
CodeDirectory v=20400 size=583 flags=0x0(none) hashes=13+2 location=embedded
Platform identifier=14
Signature size=4442
Signed Time=Jun 23, 2023 at 5:23:16 PM
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=1 size=64
% cp /bin/date my_date
% ./my_date ## A copy of /bin/date can be executed. Good!
Tue Aug 22 13:50:19 PDT 2023
% codesign --remove-signature my_date
% ./my_date ## Removing the signature disables execution
zsh: killed ./my_date
% codesign -s - ./my_date ## Signing as "nobody" is not enough. Bad!
% ./my_date
zsh: killed ./my_date
% codesign --remove-signature my_date
% codesign -s upx my_date ## A local KeychainAccess code-signing certificate is not enough; Bad!
% ./my_date
zsh: killed ./my_date
%
What more is needed so that my local, freshly-created KeychainAccess code-signing certificate can sign executables so that they can run, so that I can test the output from UPX compression?
Update: MacOS Ventura 13.5 (22G74), Apple M1 CPU (arm64)
@jreiser I just ran into this; when running any executable that was compressed with upx, I just get killed. I'm debugging it now. I'm surprised about the result here:
% codesign -s - ./my_date ## Signing as "nobody" is not enough. Bad!
% ./my_date
I'm compiling a Go binary. This is how the Go compiler signs code: https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/cmd/internal/codesign/codesign.go
Executable=/Users/me/path/to/executable
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=66046 flags=0x20002(adhoc,linker-signed) hashes=2061+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
Notice the ad-hoc signature - that runs fine.
When I run it normally, it's fine. When I strip the signature, then run it, it displays killed. However, if I run:
codesign -s - ./program
Then run codesign -d -v ./program:
Executable=/Users/user/program
Identifier=program-c2699260dfe19866f2d29d191a0c93482f0c76de
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=66156 flags=0x2(adhoc) hashes=2061+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
And it still runs
So, I'm confused as to why my_date won't run. My output is almost the same as yours:
Executable=/Users/liambowen/my_date
Identifier=com.apple.date
Format=Mach-O universal (x86_64 arm64e)
CodeDirectory v=20400 size=583 flags=0x0(none) hashes=13+2 location=embedded
Platform identifier=14
Signature size=4442
Signed Time=Jun 23, 2023 at 8:23:16 PM
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=1 size=64
% codesign --remove-signature my_date
% codesign -d -v my_date
my_date: code object is not signed at all
% codesign -s - my_date # this exact line worked fine with my Go binary
% ./my_date # killed!!! 🧐
I'm not sure why my_date won't run with an ad-hoc. Maybe it's because the same code is already present with a non-ad-hoc signature? That's a guess. What does your output of codesign -s - <whatever executable you compiled>; codesign -d -v <whatever executable you compiled>?
Thank you for the examples of signing by go. Perhaps if I can glean a clue or two.
when running any executable that was compressed with
upx, I just getkilled.
That's my experience, too. I've been running otool -hl and staring at the "tea leaves". My theory is that Darwin (the MacOS kernel) looks for some combination of Mach_command to decide whether to report ENOEXEC, EBADEXEC, etc. For instance, for the entry point (address to begin execution) upx produces LC_UNIXTHREAD whereas clang produces LC_MAIN with LC_LOAD_DYLIB. But the official dynamic linker /usr/lib/dyld has LC_ID_DYLINKER with LC_UNIXTHREAD, so LC_UNIXTHREAD is not banned everywhere; perhaps the output from upx needs to specify "I am a DYLINKER" in order to use LC_UNIXTHREAD.
On the other hand, perhaps upx should just treat an entire main program as a shared library, with main as a static constructor for exit. Just copy the LC_LOAD_DYLIB (/usr/lib/libSystem.B.dylib) and LC_LOAD_DYLINKER (/usr/lib/dyld),
change the first address in the vector of global static constructors (remembering the original) to point to the upx runtime stub, and proceed. As long as libSYstem.B.dylib doesn't mind being initialized twice (or upx can avoid using it, by making "bare" system calls), then things might just work.
See upx/src/stub/tools/macho-snip/ for a tool that I am using to excise various Mach_command from a working one-instruction program (udf.s, which aborts) in the hope of finding a minimal set that works. But I'm stuck because nothing that I codesign (not even hello-world, after removing the clang signature), with either -s - or with -s self-signed-certificate, will run.
A couple years ago Apple introduced the Mach_command LC_NOTE. At first I thought, "Hurrah! Now I can comment-out a Mach_command by changing one byte, thus making my macho-snip superfluous". Except that LC_NOTE has a 16-byte owner field, and thus cannot be applied to a Mach_command that is shorter than 24 bytes, which many are. So close to something really useful for hacking, and yet so far.
% clang -g -o hello hello.c
% ./hello
Hello world!
% codesign -d -v ./hello
Executable=/Users/jreiser/upx-devel4/hello
Identifier=hello
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=382 flags=0x20002(adhoc,linker-signed) hashes=9+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
% codesign --remove-signature ./hello
% ./hello
zsh: killed ./hello
% codesign -d -v ./hello
./hello: code object is not signed at all
% codesign -s - ./hello
% ./hello
Hello world!
% codesign -d -v ./hello
Executable=/Users/jreiser/upx-devel4/hello
Identifier=hello-55554944beb987cb08103357b629ceaf65fb37bc
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=487 flags=0x2(adhoc) hashes=9+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
I'm presuming we may not be able to resolve this one with the way Apple is tightening down everything all around?
On the other hand, perhaps upx should just treat an entire main program as a shared library, with main as a static constructor for exit. (My Comment above, one week ago on Sep.3.) This currently looks like a promising approach.
Shorten __PAGEZERO to length 0xf0000000 (4GB - 256MB). Shorten existing __TEXT segments by compression, putting the compressed data into an additional __TEXT segment at (4GB - 256MB), but leaving the vmaddr of existing __TEXT as-is. The Mach-o headers would remain mostly "the same" except for the additional __TEXT and smaller __PAGEZERO. Those headers tend to be fairly compressible, but they are "well defended" by lack of documentation.
class static1
{
public: int *data;
static1(void) {
data = new int[10];
}
};
static1 instance1;
int main(int argc, char *argv[])
{
return 5;
}
Compiling with clang++ -g and running under lldb; process launch -s; break static1; continue shows a traceback of
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
* frame #0: 0x0000000100003eec static-init`static1::static1(this=0x0000000100003f94) at static-init.cpp:4
frame #1: 0x0000000100003f80 static-init`::__cxx_global_var_init() at static-init.cpp:9:9
frame #2: 0x0000000100003f94 static-init`_GLOBAL__sub_I_static_init.cpp at static-init.cpp:0
frame #3: 0x00000001aa6581d8 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
frame #4: 0x00000001aa699e94 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
frame #5: 0x00000001aa68d1a4 dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528
frame #6: 0x00000001aa6382d8 dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
frame #7: 0x00000001aa68c1cc dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
frame #8: 0x00000001aa699958 dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
frame #9: 0x00000001aa65485c dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 448
frame #10: 0x00000001aa654c10 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
frame #11: 0x00000001aa658264 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
frame #12: 0x00000001aa654d90 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
frame #13: 0x00000001aa678984 dyld`dyld4::APIs::runAllInitializersForMain() + 468
frame #14: 0x00000001aa63d2d0 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3480
frame #15: 0x00000001aa63be18 dyld`start + 1964
Running otool -hl reveals a list of static constructors:
Section
sectname __init_offsets
segname __TEXT
addr 0x0000000100003fa8
size 0x0000000000000004
offset 16296
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000016
reserved1 0
reserved2 0
The strategy is to change the first pointer to be the UPX de-compression sub, expand compressed __TEXT into the empty page "holes" left by compression, "bless" the de-compressed contents as "Just-In-Time" code, then run the original static inits, and proceed.
macOS 14 "Sonoma" source code has been released https://github.com/apple-oss-distributions/xnu
Perhaps the *.cpp, *.c, and *.h are all there, but significant details are missing: which version of the SDK is required (and how to get it), which version of Clang is required, etc.
Compiling fails for me. Following the directions in the top-level README.md on a MacBook Air running macOS Monterey (Version 12.7) with Xcode that does not complain about needing an update (and does not say which version it IS):
% git clone https://github.com/apple-oss-distributions/xnu.git
% cd xnu
% make
xcrun: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -find tightbeamc 2> /dev/null' failed with exit code 17664: (null) (errno=No such file or directory)
xcrun: error: unable to find utility "tightbeamc", not a developer tool or in PATH
/Users/jreiser/xnu/makedefs/MakeInc.kernel:286: *** Could not determine xnu version from SDK or KDK! Set RC_DARWIN_KERNEL_VERSION environment variable.. Stop.
Google, Bing and Yandex have no idea what "tightbeamc" is, and neither does Apple Developer's site - @markus-oberhumer were you able to get this to build? I get the same error as @jreiser . I'm off to a rough start 😅
Same error on MacOS Ventura 13.6, with Xcode Version 14.3.1 (14E300c).
Build fails for me as well on a mac mini M2 with kernel 22.6.0.
Some quick search on the file system:
$ xcodebuild -version
Xcode 14.3.1
Build version 14E300c
$ sudo bash
$ gfind /Applications /Library /System -iname "*tightbeam*"
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/PrivateFrameworks/Tightbeam.framework
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Tightbeam.tbd
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Versions/A/Tightbeam.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/System/Library/PrivateFrameworks/Tightbeam.framework
/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Tightbeam.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Versions/A/Tightbeam.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk/System/Library/PrivateFrameworks/Tightbeam.framework
/Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Tightbeam.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk/System/Library/PrivateFrameworks/Tightbeam.framework/Versions/A/Tightbeam.tbd
gfind: '/Library/Caches/com.apple.aned': Operation not permitted
gfind: '/System/Library/Templates/Data/private/var/db/oah': Operation not permitted
/System/Library/PrivateFrameworks/Tightbeam.framework
/System/Library/PrivateFrameworks/Tightbeam.framework/Tightbeam
gfind: '/System/Volumes/Update/mnt1/System/Library/Templates/Data/private/var/db/oah': Operation not permitted
/System/Volumes/Update/mnt1/System/Library/PrivateFrameworks/Tightbeam.framework
/System/Volumes/Update/mnt1/System/Library/PrivateFrameworks/Tightbeam.framework/Tightbeam
gfind: failed to read file names from file system at or below '/System': No such file or directory
Update Xcode by hand, still get same error.
AppStore > Search Xcode > Update (3.2 GB)
% xcodebuild -version
Xcode 15.0
Build version 15A240d
% makexcrun: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -find tightbeamc 2> /dev/null' failed with exit code 17664: (null) (errno=No such file or directory)
xcrun: error: unable to find utility "tightbeamc", not a developer tool or in PATH
/Users/jreiser/xnu/makedefs/MakeInc.kernel:286: *** Could not determine xnu version from SDK or KDK! Set RC_DARWIN_KERNEL_VERSION environment variable.. Stop.
make[1]: *** [build_setup_bootstrap_DEVELOPMENT^X86_64^NONE] Error 2
make: *** [all] Error 2
%
Just a reminder of why I want to build. After compiling I will build a *.a library of everything that was compiled, and perhaps also a *.so shared library. Then I will make a main program that calls do_execve() (or whatever the right name is for the Darwin kernel subroutine that installs the executable file for a main program into the address space), and run that main program under a debugger. This will let me figure out exactly what is required to make execve work, which will tell me what the upx stub needs to do.
@jreiser Sounds like a reasonable plan
Maybe a StackOverflow question like Has anyone succeeded in building the macOS 14 xnu kernel from source? could help.
EDIT: or maybe asking on Twitter/X
Would Apple's Kernel Debug Kit be useful for debugging this? You can find it in the "More Downloads" section of the Apple developer website.
just fyi, we are deprecating homebrew formula due to the crashing issue listed in here.
@chenrui333 Sounds reasonable. Hopefully we will be able to support macOS 13+ at some future point...
Packing macOS binaries is disabled in Release builds for now. Commit 4870765
For what it's worth, I did some poking around (unrelated to y'all's project) - tightbeamc appears to be a proprietary preprocessor for .tightbeam.c and .tightbeam.h files (possibly comparable to MIG, see below).
I have found references to something called Tightbeam kernel runtime which seems to be related to a new (or upcoming?) Mach feature called Exclaves. There is a related entity called Conclaves, but these might be derivative of Exclaves.
Presently all the syscalls for Exclaves are stubbed out to simply return KERN_NOT_SUPPORTED. What are Exclaves? Frankly I haven't the slightest clue. They seem related to IOKit/DriverKit, and appear to have a lot of calls related to IPC/RPC (mach ports are the underlying structure still) - I have also found references to XRT alongside this.
Whatever Exclaves are, they're internal to Apple, at least for now. I'm not sure if they're active on current versions of *OS or not.
Luckily, they can be disabled. To do so, you need to remove ExclaveKit and ExclaveCore from SUPPORTED_PLATFORMS in makedefs/MakeInc.cmd. Then, either add TIGHTBEAMC=nothx to your make invocation or, from the same file, remove the following lines:
ifeq ($(origin TIGHTBEAMC),undefined)
export TIGHTBEAMC := $(shell $(XCRUN) -sdk $(SDKROOT) -find tightbeamc)
endif
Now everything should behave as normal (however due to other, unrelated, errors I haven't managed to actually build the kernel yet - but installhdrs worked fine).
Could you please provide exact instructions how you are building the xnu kernel?
I'm stuck at *** Could not determine xnu version from SDK or KDK! Set RC_DARWIN_KERNEL_VERSION environment variable.. Stop.
My setup on a Mac Mini M2, Darwin 22.6.0:
$ git describe
xnu-10002.1.13
$ xcodebuild -version
Xcode 14.3.1
Build version 14E300c
Ah my b - I forgot about something else I had to add to my make invocation. Here's the full command I'm running:
make SDKROOT=macosx ARCH_CONFIGS="X86_64 ARM64" KERNEL_CONFIGS=RELEASE TIGHTBEAMC==nothx LOGCOLORS=y RC_DARWIN_KERNEL_VERSION=23.0.0I borked my macOS SDK when I ran the same invocation with the addition of the installhdrs target, so for me, running the above failed due to not being able to build certain modules or find certain headers, etc.
I stopped there and haven't had time to try continuing onwards. The guide I'm following is a mish mash of the *OS internals book's instructions and the ones found at https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html
I had to make some adjustments, attached below are the notes I took about what has changed about the process.
Best of luck, and if you figure out how to compile it, please do share! Regardless, I'll be back when I figure out how to compile it.
Compiling XNU for Apple Silicon
Just adding diffs from https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html for now.
Install Dependencies
DTrace/ctf*
- Download DTrace
git clone https://github.com/apple-oss-distributions/dtrace.git
-
Open
dtrace/dtrace.xcodeproj -
Add codesigning identity to
ctf*subprojects -
Allow terminal.app to modify apps (security & privacy)
-
Corrected
dittocommand:
sudo ditto "$PWD/dst/" "$TOOLCHAIN"
AvailabilityVersions
- Download AvailabilityVersions
git clone https://github.com/apple-oss-distributions/AvailabilityVersions.git
- Install the Ninja build system
brew install ninja
-
Possible: run
makebeforemake install -
make
dst/usr/local/libexec/availability.plexecutable
Install XNU Headers
- Remove
ExclaveKitandExclaveCorefromSUPPORTED_PLATFORMSinmakedefs/MakeInc.cmd - Add
TIGHTBEAMC=nothxto themakearguments
Many thanks for your info!
I'm thinking about creating a GitHub Actions workflow that clones the apple-oss-distributions repositories, applies necessary patches and then builds the XNU kernel - running under GitHub macos-13.
But then I'm sure somebody else must have had this idea. Any hints?
I've created https://github.com/upx/upx-fork-apple-oss-distributions-xnu , please continue XNU build discussions there.
I've never used GitHub Actions before - so unfortunately I can't help you there. I'll add updates to the XNU fork as I figure things out though.
@tomnific Well, GitHub Actions are basically shell scripts that are run in a container (more technically its a YAML file that is parsed by NodeJS which then generates and runs the shell scripts).
Please visit upx/upx-fork-apple-oss-distributions-xnu#1
and have a look at https://github.com/upx/upx-fork-apple-oss-distributions-xnu/actions/runs/7021092042
and https://github.com/upx/upx-fork-apple-oss-distributions-xnu/blob/master/.github/workflows/build-xnu-macos-13.yml
I guess zsh: killed appears because the header of a non-x86_64 architecture executable must contains the MH_DYLDLINK flag and the mach-o must contains LC_LOAD_DYLINKER command (https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L1265, https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L1466).
I wrote a script to verify this
#!/usr/bin/env python3
import argparse
import lief
binary = None
# https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L731
def pie_required(exectype, execsubtype):
if exectype == lief.MachO.CPU_TYPES.x86_64:
return False
elif exectype == lief.MachO.CPU_TYPES.ARM64:
return True
elif exectype == lief.MachO.CPU_TYPES.ARM and execsubtype == 12: # CPU_SUBTYPE_ARM_V7K
return True
else:
return False
def add_dyldlink_flag(binary):
if isinstance(binary, lief.MachO.FatBinary):
for sub_binary in fat_binary:
header = sub_binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and header.has(lief.MachO.HEADER_FLAGS.DYLDLINK) == False:
header.add(lief.MachO.HEADER_FLAGS.DYLDLINK)
else:
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and header.has(lief.MachO.HEADER_FLAGS.DYLDLINK) == False:
header.add(lief.MachO.HEADER_FLAGS.DYLDLINK)
def add_LC_LOAD_DYLINKER(binary):
load_command = lief.MachO.DylinkerCommand("/usr/lib/dyld")
if isinstance(binary, lief.MachO.FatBinary):
for sub_binary in fat_binary:
header = sub_binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and sub_binary.has(lief.MachO.LOAD_COMMAND_TYPES.LOAD_DYLINKER) == False:
sub_binary.add(load_command)
else:
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.LOAD_DYLINKER) == False:
binary.add(load_command)
parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input", help="input file", type=str, required=True)
parser.add_argument("-o", "--output", help="output file", type=str, required=True)
args = parser.parse_args()
input_file = args.input
output_file = args.output
binary = lief.parse(input_file)
add_dyldlink_flag(binary)
add_LC_LOAD_DYLINKER(binary)
binary.write(output_file)
This script add MH_DYLDLINK and LC_LOAD_DYLINKER to the executable, the zsh: killed problem was solved, but there was another problem
zsh: segmentation fault
lldb
Process 57821 stopped
* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x48)
frame #0: 0x0000000183c037a0 dyld`dyld3::MachOAnalyzer::forEachRebase_Relocations(Diagnostics&, dyld3::MachOLoaded::LinkEditInfo const&, dyld3::MachOFile::SegmentInfo const*, void (char const*, dyld3::MachOLoaded::LinkEditInfo const&, dyld3::MachOFile::SegmentInfo const*, bool, unsigned int, unsigned char, unsigned long long, dyld3::MachOAnalyzer::Rebase, bool&) block_pointer) const + 116
dyld`dyld3::MachOAnalyzer::forEachRebase_Relocations:
-> 0x183c037a0 <+116>: ldr w2, [x8, #0x48]
0x183c037a4 <+120>: mov x0, x20
0x183c037a8 <+124>: mov x1, x24
0x183c037ac <+128>: bl 0x183bf75e4 ; dyld3::MachOLoaded::getLinkEditContent(dyld3::MachOLoaded::LayoutInfo const&, unsigned int) const
Target 0: (test_upx) stopped.
@61bcdefg Thank you for the URLs and script! That is progress in understanding the requirements of MacOS.
At first glance, it looks like the site of the SIGSEGV, namely dyld:dyld3::MachOAnalyzer::forEachRebase_Relocations, is trying to rebase a table of relocations, but there is no such table in the specific executable. This is typical of Apple code, which assumes that any executable is a full-blown result of an XCode project with all the bells and whistles: symbol table, relocations, libraries, two-level relocations, linker info, etc. Even the vaunted DTrace is sh*t because it assumes much too much, and as a result cannot process the simplest executable (a two-instruction program that performs a "bare" system call to exit(0); with no symbols, no libraries, no dylinker, etc.)
UPX would rather not construct a compressed executable with all that baggage: it occupies space and could easily interfere with the de-compressed program. What if the dylinker used by UPX is not the same as the one used by the de-compressed program, what if the system library does not understand being "initialized" twice (once by the UPX de-compresion stub, once by the de-compressed program), etc.
Because of the current murkiness involving instantiation and initialization that is performed by execve, now I am considering "just" intercepting the initialization of the executable's static constructors, after all the "system" machinations with dylinker, symbol tables, etc. Perhaps it will be possible to use only {mmap, munmap, fetch and store to memory} to de-compress the LC_SEGMENT_64 appropriately.
dyld:dyld3::MachOAnalyzer::forEachRebase_Relocations, is trying to rebase a table of relocations, but there is no such table in the specific executable.
In dyld:dyld3::MachOAnalyzer::forEachRebase_Relocations the dyld read leInfo.dynSymTab->locreloff, but there were no dynSymTab in executable generated by upx. Besides, If the executable contains LC_DYSYMTAB, it must also contains LC_SYMTAB, and the dyld only support LC_MAIN.
I modified the script to add LC_SYMTAB, LC_DYSYMTAB and replace LC_UNIXTHREAD with LC_MAIN.
#!/usr/bin/env python3
import argparse
import lief
# https://github.com/apple-oss-distributions/xnu/blob/5e3eaea39dcf651e66cb99ba7d70e32cc4a99587/bsd/kern/mach_loader.c#L731
def pie_required(exectype, execsubtype):
if exectype == lief.MachO.CPU_TYPES.x86_64:
return False
elif exectype == lief.MachO.CPU_TYPES.ARM64:
return True
elif exectype == lief.MachO.CPU_TYPES.ARM and execsubtype == 12: # CPU_SUBTYPE_ARM_V7K
return True
else:
return False
def add_dyldlink_flag(binary):
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and header.has(lief.MachO.HEADER_FLAGS.DYLDLINK) == False:
header.add(lief.MachO.HEADER_FLAGS.DYLDLINK)
def add_LC_LOAD_DYLINKER(binary):
load_command = lief.MachO.DylinkerCommand("/usr/lib/dyld")
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.LOAD_DYLINKER) == False:
binary.add(load_command)
def add_LC_SYMTAB(binary, binary2):
load_command = binary2.get(lief.MachO.LOAD_COMMAND_TYPES.SYMTAB)
load_command.numberof_symbols = 0
load_command.strings_offset = 0
load_command.strings_size = 0
load_command.symbol_offset = 0
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.SYMTAB) == False:
binary.add(load_command)
def add_LC_DYSYMTAB(binary, binary2):
load_command = binary2.get(lief.MachO.LOAD_COMMAND_TYPES.DYSYMTAB)
load_command.external_reference_symbol_offset = 0
load_command.external_relocation_offset = 0
load_command.idx_external_define_symbol = 0
load_command.idx_local_symbol = 0
load_command.idx_undefined_symbol = 0
load_command.indirect_symbol_offset = 0
load_command.local_relocation_offset = 0
load_command.module_table_offset = 0
load_command.nb_external_define_symbols = 0
load_command.nb_external_reference_symbols = 0
load_command.nb_external_relocations = 0
load_command.nb_indirect_symbols = 0
load_command.nb_local_relocations = 0
load_command.nb_local_symbols = 0
load_command.nb_module_table = 0
load_command.nb_toc = 0
load_command.nb_undefined_symbols = 0
load_command.toc_offset = 0
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.DYSYMTAB) == False:
binary.add(load_command)
def replace_LC_UNIXTHREAD_with_LC_MAIN(binary):
thread_command = binary.get(lief.MachO.LOAD_COMMAND_TYPES.UNIXTHREAD)
pc = thread_command.pc
load_command = lief.MachO.MainCommand(pc - 0x100000000, 0)
header = binary.header
if pie_required(header.cpu_type, header.cpu_subtype) and binary.has(lief.MachO.LOAD_COMMAND_TYPES.MAIN) == False:
binary.remove(lief.MachO.LOAD_COMMAND_TYPES.UNIXTHREAD)
binary.add(load_command)
parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input", help="input(packed) file", type=str, required=True)
parser.add_argument("-u", "--unpacked", help="unpacked(original) file", type=str, required=True)
parser.add_argument("-o", "--output", help="output file", type=str, required=True)
args = parser.parse_args()
input_file = args.input
output_file = args.output
binary = lief.parse(input_file)
binary2 = lief.parse(args.unpacked)
if isinstance(binary, lief.MachO.FatBinary) or isinstance(binary2, lief.MachO.FatBinary):
print("Fat Binary is unsupported")
exit(-1)
add_dyldlink_flag(binary)
add_LC_LOAD_DYLINKER(binary)
add_LC_SYMTAB(binary, binary2)
add_LC_DYSYMTAB(binary, binary2)
replace_LC_UNIXTHREAD_with_LC_MAIN(binary)
binary.write(output_file)
The executable is still not working properly:
Process 24133 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1000e4180)
frame #0: 0x00000001000e4180 test_upx
-> 0x1000e4180: brk #0
0x1000e4184: mov x28, x30
0x1000e4188: ldr x0, [x2], #0x8
0x1000e418c: cbnz x0, 0x1000e4188
Target 0: (test_upx) stopped.
I don't know exactly how upx works so I don't know how to fix it, hope the information would be helpful to you
Is there a good alternative until this is fixed?
There is no alternative known to us. The Apple documentation known to us is not specific enough to implement against (what are the exact requirements for execve so succeed?) The best we can do is trial-and-error, which is slow and frustrating. The Apple open-source kernel does not build, so cannot be used as a hermeneutic for discovery.
Well, if the only thing that matters is small download size: compress the executable using your favorite file compressor, and make a shell archive (such as shar) containing the result. The shell de-compresses info a temporary file, and exec that file with the original parameters to the script. The drawbacks are: need a place in the filesystem for the file, need a name for the file, need storage space for the contents of the file, the file must have eXecute permission and must be executable (mounting a filesystem with option noexec defeats this, and other MacOS properties and permissions probablly apply), must handle deleting the file eventually, the de-compressor must be available, probably requires more execution time.
The Apple open-source kernel does not build, so cannot be used as a hermeneutic for discovery.
Have you ever try https://github.com/blacktop/darwin-xnu-build