Dumper GP4 generator fails on long paths
eleganthack opened this issue · comments
The dumper generates GP4 files which sometimes contains empty file nodes like this:
< file targ_path = "" orig_path = "" />
This seems to be caused by particularly long filenames, like in Until Dawn: Rush of Blood. Here's an example file that triggers this issue:
localcachepink/streams/models.building_blocks.lodge.cellar.set_dressing.mainlodge_garage_fivegallonbucket.textures._lom_mainlodge_garage_fivegallonbucket_b001_c003_mat_0_textureset.corestream
I followed the link to Al Azif's dumper-testing project and checked the gp4.cpp file, which seems to use std::filesystem objects to store paths. Maybe this has a max path buffer length that is too small?
Apologies for posting this here, when it should be on the dumper-testing project. The latter has no Issues tracking that I'm able to find, and I thought it might be helpful to have this documented SOMEwhere, even if it's not the right place, in hopes it makes its way to someone who might have some influence on the code.
Ill look into it
Thanks, LightningMods. I had another batch of games to dump today. The store required an update, which was successful, and it seems to have broken the dumper in a myriad of ways. Some problems I've found:
- In the GP4 file, the "psproject / volume / chunk_info / chunks" section is missing entirely. "scenarios" is still there, but no "chunks". The resulting file fails to open in orbis-pub-gen. This happened for all four titles I tried to dump.
If I just paste in that section from a GP4 file created by a prior version ....
- The dumper somehow managed to create a playgo-chunks.xml file in the root directory of Dark Souls Remastered (CUSA08692) that is 315KB, but can't be read by Windows. Building the pkg in pub-gen fails at 0% with the error "Could not open or read input file. (playgo-chunks.xml )" Scandisk reports no errors, and I don't think this is a hardware or filesystem-level issue. In Explorer, right-click - Properties on this file shows 0 bytes.
I think the problem is with the filename. The GP4 refers to this:
<file targ_path="playgo-chunks.xml " orig_path="playgo-chunks.xml " />
Notice the extra space between xml and the quotes, so I think it may have created a file called "playgo-chunks.xml " that can't be accessed using normal I/O APIs that strip whitespace from the ends of the filename. (I will try to check later by hex-editing the block device under Linux.) Not sure if the disc really has a file with a space at the end, of if that was a parsing bug, but I suspect the latter.
-
Using a hastily patched GP4 file for The Last of Us Part II, pub-gen gets about 2% complete building the package, then complains that it "Could not create system directory. (sce_sys/about)" I'm not totally familiar with how all of this works, but I looked in some of the older GP4 files (from the previous HBStore version) and it does not seem to include sce_sys/about/rights.sprx file, or anything else below sce_sys except trophies. The folders are there in the dumps, just seem to be excluded in the GP4.
-
The status files created in the root of the USB drive were never renamed to "complete" despite the dumper finishing "without errors." They're still all "CUSAxxxxx.dumping".
There may be more problems, but at this point, I'm not sure it's a good idea to keep hacking fixes into the project files lest I get a fundamentally broken dump to build successfully. :-)
can you also tell me what version your store app is (in the corner of the screen)? before continuing
The Store requested another update, which I'm obliged to proceed with, so I'm now running Store Version 2.3-V-2022-04-01T01:52:38. Must've been built in a time zone that is 4 months behind my local time.
This version reversed the changes that broke the dumper (status files are again renamed to .complete, extra sce_sys files and sub-directories are removed from the GP4 file manifest, and the chunks section is back), but, as is probably expected, it hasn't fixed the problem with long names. I tried to re-dump Until Dawn and it still has entries with empty file names.
Also, the problem with Dark Souls Remastered ("playgo-chunks.xml ") seems to be an unrelated and known issue. I researched that one some more and found a GBAtemp post where someone noted that title has an issue with that file, and that it isn't required, so I manually removed it from the GP4 and the package built successfully. Now I have two directories with files I can't delete under Windows. :-) But that's an unrelated bug, or an issue with the mastering of that disc's retail package.
Im going to close this issue since itemzflow is being separated into its own standalone app soon, but after that happens if you still have this issue feel free to resubmit this issue on that repo instead