Build N64Recomp to be actually bootable?
Cellenseres opened this issue · comments
Heyhey!
After quite some testing and trying, I finally was able to get a working .elf from a game of my choice that actually works with N64Recomp!
So I finally was able to run "./N64Recomp siliconvalley.toml"!
But... what now?
n64comp@cellenser:/mnt/N64Recomp-Project/SiliconValley# ./N64Recomp siliconvalley.toml Sections Num symbols: 1170 Found entrypoint, original name: func_80125900 Function count: 1030 Working dir: /mnt/N64Recomp-Project/SiliconValley
This is my toml-file:
[input] entrypoint = 0x80125900 elf_path = "siliconvalley.elf" directory = "output" output_func_path = "output/functions"
it created the output which contains four files:
And another file that has been generated by another tool is the siliconvalley.map
Does anyone have an Idea how to finalize it?
There's no Guide or Explaination for that yet. What to do with the Files that have been created?
Hi!
From a quick look at your files I can see that you have something misconfigured. N64Recomp should have created a lots of C files, each file for a each function from the original ROM. Only getting the entrypoint is not enough.
The N64Recomp tool expects functions to be marked as such in the input ELF file.
You can check this by using readelf -s game.elf and check the size and type columns of your functions. So if you see something like this it is wrong:
1580: 80072204 0 NOTYPE GLOBAL DEFAULT 14 func_80072204
size must be greater than zero and type must be FUNC:
1643: 800770e8 136 FUNC GLOBAL DEFAULT 14 func_800770E8
Make sure your functions are properly defined when you build your asm and C files.
Another source of problems is functions marked as ABS on the Ndx column:
8013: 800394a0 684 FUNC GLOBAL DEFAULT ABS func_800394A0
This means the function was defined using a linker script. If that's the case remove those functions from your linker script.
Once you get all the C functions generated properly you'll need to hook up a runtime and build it.
What people other people is currently doing is to reuse files from the Zelda64Recomp project and strip/change anything that is MM specific, like all the patches. You'll require C++ and cmake knowledge to do this.
Wiseguy is working on a minimal example on how to set up a recomp project, personally I suggest you to wait for that instead.
I think I just got a new Script to work for my purpose! At least the functions aren't zero anymore"
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 SECTION LOCAL DEFAULT 1 .header
2: 00000040 0 SECTION LOCAL DEFAULT 2 .boot
3: 80246000 0 SECTION LOCAL DEFAULT 3 .main
4: 000e6430 0 SECTION LOCAL DEFAULT 4 .E6430
5: 802226a8 0 NOTYPE GLOBAL DEFAULT ABS D_802226A8
6: 80273160 952 FUNC GLOBAL DEFAULT 3 func_80273160
7: 8032d5c0 0 NOTYPE GLOBAL DEFAULT ABS D_8032D5C0
8: 803612c0 0 NOTYPE GLOBAL DEFAULT ABS D_803612C0
9: 802e6ed8 328 FUNC GLOBAL DEFAULT 3 func_802E6ED8
10: 802b3828 8 FUNC GLOBAL DEFAULT 3 func_802B3828
11: 8033c378 0 NOTYPE GLOBAL DEFAULT ABS D_8033C378
12: 802fb668 0 FUNC GLOBAL DEFAULT 3 L802FB668
13: 803399a0 0 NOTYPE GLOBAL DEFAULT ABS jtbl_803399A0_main
14: 8028b50c 536 FUNC GLOBAL DEFAULT 3 func_8028B50C
15: 802b459c 80 FUNC GLOBAL DEFAULT 3 func_802B459C
16: 80253488 256 FUNC GLOBAL DEFAULT 3 func_80253488
17: 8028ec2c 44 FUNC GLOBAL DEFAULT 3 func_8028EC2C
18: 80339360 0 NOTYPE GLOBAL DEFAULT ABS jtbl_80339360_main
19: 80336c18 0 NOTYPE GLOBAL DEFAULT ABS D_80336C18
20: 80337c70 0 NOTYPE GLOBAL DEFAULT ABS jtbl_80337C70_main
21: 80338e14 0 NOTYPE GLOBAL DEFAULT ABS D_80338E14
22: 80332860 0 NOTYPE GLOBAL DEFAULT ABS D_80332860
23: 802dbe68 756 FUNC GLOBAL DEFAULT 3 func_802DBE68
24: 80255654 184 FUNC GLOBAL DEFAULT 3 func_80255654
25: 8025db48 0 FUNC GLOBAL DEFAULT 3 L8025DB48
26: 8028af4c 192 FUNC GLOBAL DEFAULT 3 func_8028AF4C
27: 803386f0 0 NOTYPE GLOBAL DEFAULT ABS jtbl_803386F0_main
28: 80332862 0 NOTYPE GLOBAL DEFAULT ABS D_80332862
29: 803327fc 0 NOTYPE GLOBAL DEFAULT ABS D_803327FC
30: 80249c58 0 FUNC GLOBAL DEFAULT 3 L80249C58
31: 802f7924 76 FUNC GLOBAL DEFAULT 3 func_802F7924
32: 8028526c 136 FUNC GLOBAL DEFAULT 3 func_8028526C
33: 802a551c 8 FUNC GLOBAL DEFAULT 3 func_802A551C
34: 802b2ba8 0 FUNC GLOBAL DEFAULT 3 L802B2BA8
35: 80332aa4 0 NOTYPE GLOBAL DEFAULT ABS D_80332AA4
...
Output of my Script:
./makeelf.sh supermario64.z64
Generating YAML config for the ROM...
Writing config to supermario64.yaml
Disassembling the ROM...
splat 0.9.2 (powered by spimdisasm 1.25.1)
Scanning E6430: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.10s/it]
Splitting E6430: 0%| | 0/4 [00:00<?, ?it/s]Segment 1000, function at vram 80246DF8 ends with extra nops, indicating a likely file split.
File split suggestions for this segment will follow in config yaml format:
- [0x1E70, asm]
...
<snip>
...
Converting .bin to .bin.o files...
Compiling assembly files...
Compiling C files...
Linking object files to create ELF...
Converting ELF to binary...
ELF file created at build/supermario64.elf
Binary file created at build/supermario64.bin
Update:
With this method, I got an elf that had functions with not-0 sizes. Sounds good so far!
When using the elf in the .toml file, I get this when running N64Recomp:
./N64Recomp sm64.toml
Sections
Num symbols: 7737
Found entrypoint, original name: func_80246000
Function count: 4828
Working dir: /mnt/N64Recomp-Project
No function found for jal target: 0x8037A9A8
<snip>
No function found for jal target: 0x80383CB4
Unhandled instruction: cache
Error in recompiling func_80325D20, clearing output file
Error recompiling func_80325D20
Any idea?
Another thing I noticed is that when I disassemble the ROM using splat 0.9.5 (which comes with the mischief makers decomp-project), the elf-file I rebuild using
mips-linux-gnu-ld supermario64.ld -Map supermario64.map -T undefined_syms_auto.txt -T undefined_funcs_auto.txt -T undefined_funcs.txt --no-check-sections looks fine (FUNC sizes > 0).
But when I use splat 0.24.5 (most recent version available using python pip install), all funcs in the .elf have a size of zero after rebuilding the elf with the same command.
Using the .elf (disassembled with splat 0.9.5, build with mips-linux-gnu-ld) extracts a few functions but with errors when running through N64Recomp
Using the .elf (disassembled with splat 0.24.5, build with mips-linux-gnu-ld) extracts no functions (.c) when running through N64Recomp
@Mr-Wiseguy I'm really sorry for the ping. But could you explain why it doesn't work and which tools should be used to build the .elf? The rest should be doable.
The Script I'm using for disassembling the rom (splat 0.9.5 from the mischief makers project) + building the elf.
https://pastebin.com/eu5A3vwE
EDIT:
Both .bin files that were created (disassembled with different splat versions and rebuild back to .elf) using
mips-linux-gnu-objcopy -O binary supermario64.elf supermario64.bin are identical and do work in the Emulator
I'm confused. Which game are you trying to recompile? So far you have mentioned sm64, mischief makers and silicon valley.
For sm64, are you trying to create a new splat project for that game? Wouldn't be more simple to just use the existing decomp project that has a few versions completely matching?
But when I use splat 0.24.5 (most recent version available using python pip install), all funcs in the .elf have a size of zero after rebuilding the elf with the same command.
Using the .elf (disassembled with splat 0.9.5, build with mips-linux-gnu-ld) extracts a few functions but with errors when running through N64Recomp
Using the .elf (disassembled with splat 0.24.5, build with mips-linux-gnu-ld) extracts no functions (.c) when running through N64Recomp
The difference in splat behavior you are seeing per version is caused because different versions have different defaults for its settings. For example splat 0.24.5 opted to default to turning off the size directives for IDO projects because it was causing issues for those projects. This no longer is true if other tools are up to date so turning the option on should be fine (splat docs)
When using the elf in the .toml file, I get this when running N64Recomp:
N64Recomp may be complaining about missing jal targets because there are indeed missing functions. Try checking the elf to see if there's any non-ABS function on those addresses.
You may not be getting all the functions from the start because of how N64 games work. Since there's no much space in RAM to load everything the game may want to use it has to swap stuff around, usually loading assets and code in runtime by the boot segment (the first segment). In simple terms a segment is a chunk of code and data that can be loaded to RAM by the game.
Many games have more than one code segment which may get loaded at any time, but current static analysis tools like splat's create_config can only see the first code segment (usually referred to as boot).
To be able to disassemble the rest of the functions you need to reverse engineer the game and figure out where the game is loading new code segments, what are the ROM addresses from those code segments and what is the VRAM address used for each of those segments. We can't provide help with doing this kind of reverse engineering since it is out of the scope of this project.
Even if you were working with a game that only has 1 code segment you can't just setup a new splat project for a game and expect it to work out of the box. This tool expects that all the Nintendo libultra SDK functions used in this game are properly identified and named. There are tools that can help identifying libultra functions, but again explaining this kind of reverse engineering is out of scope.
@AngheloAlf I'm trying to do it with multiple games and see what N64Recomp is giving me. I prepared a Script that takes a random .z64/n64 rom file and generates a .elf file out of it (by using splat for disassembling the rom and mips-linux-gnu to rebuild it to .elf after compiling .s and .bin files to .o).
This Script seems to work very well, even adding "undefined references to functions" to the undefined_funcs.txt when trying to build the elf for the first time:
Linking object files to create ELF...
mips-linux-gnu-ld: build/asm/1050.o: in function `func_8032B260':
(.text+0xe5214): undefined reference to `func_84001068'
mips-linux-gnu-ld: (.text+0xe52d8): undefined reference to `func_8400100C'
mips-linux-gnu-ld: (.text+0xe52e0): undefined reference to `func_84001780'
mips-linux-gnu-ld: (.text+0xe5320): undefined reference to `func_840010D4'
mips-linux-gnu-ld: (.text+0xe532c): undefined reference to `func_840010FC'
Handling undefined references...
by doing this:
echo "Linking object files to create ELF..."
if ! ${LD} ${LDFLAGS} -o "${BUILD_DIR}/${BASENAME}.elf" $(find ${BUILD_DIR} -name '*.o') 2>&1; then
echo "Handling undefined references..."
# Extrahiere undefined references und erstelle eine undefined_funcs.txt
${LD} ${LDFLAGS} -o "/dev/null" $(find ${BUILD_DIR} -name '*.o') 2>&1 | grep "undefined reference to" | sed -e "s/.*undefined reference to \`func_\\([0-9a-fA-F]*\)'/func_\\1 = 0x\\1;/" > "undefined_funcs.txt"
# Versuche erneut zu linken
${LD} ${LDFLAGS} -o "${BUILD_DIR}/${BASENAME}.elf" $(find ${BUILD_DIR} -name '*.o')
fi
The Script now uses splat version 0.24.5 and made sure it adds "asm_emit_size_directive: True" to YAML config automatically.
Yea, I know it will probably not be enough to get a game ported with N64Recomp, but it's a fine start I think.
I still have a lot to learn when it comes to reverse engineering N64 Games, but I'm willing to learn.
I try different Games (like sm64, siliconvalley and mischief makers) to see, if anything else has to be done than just disassembling them, building the .elf and using that in N64Recomp.
And it seems it is NOT enough to just do that and I totally understand.
But still I learned a lot already by doing that :D
About the ABS Functions and the jal errors, you're right.
All non-ABS functions do have a size > 0, while all ABS Functions have a size of 0.
80327d68 0 FUNC GLOBAL DEFAULT ABS func_80327D68
802461dc 0 FUNC GLOBAL DEFAULT ABS func_802461DC
8029d1d8 0 FUNC GLOBAL DEFAULT ABS func_8029D1D8
80327d58 0 FUNC GLOBAL DEFAULT ABS func_80327D58
80327c80 0 FUNC GLOBAL DEFAULT ABS func_80327C80
802550b0 0 FUNC GLOBAL DEFAULT ABS func_802550B0
80327ea8 0 FUNC GLOBAL DEFAULT ABS D_80327EA8
80327eb0 0 FUNC GLOBAL DEFAULT ABS func_80327EB0
80327b98 0 FUNC GLOBAL DEFAULT ABS func_80327B98
80327d10 0 FUNC GLOBAL DEFAULT ABS func_80327D10
802461cc 0 FUNC GLOBAL DEFAULT ABS func_802461CC
8032b330 0 FUNC GLOBAL DEFAULT ABS D_8032B330
80327650 0 FUNC GLOBAL DEFAULT ABS D_80327650
80324c20 0 FUNC GLOBAL DEFAULT ABS func_80324C20
Almost all other Functions have an NDX of "5", one has the NDX of "3" (which is the entrypoint of that game).
If you have any tips on how to proceed further (e.g. getting the funcs of the other code segments using splat or how to handle the ABS funcs), that'd be frickin awesome! :D