Cannot download / unpack XML

Question

Cannot download / unpack XML

cinnamonmatexfce opened this issue 3 years ago · comments

Is this a know behaviour with this NPEB02182?
Shouldn't PS3 automatically handle PKG merging by the instructions provided via the XML file?
The real package is not downloaded.

Here's what I found:

/dev_hdd0/vsh/game_pkg/80000001:
EP0101-NPEB02182_00-MGS4MAINGAME0000.pkg [1,94KiB]
ICON_FILE [1,20KiB]
d0.pdb [738B]
d1.pdb [792B]
f0.pdb [0B]

Damián Parrino · Answer 1 · Mon Jan 24 2022 19:59:50 GMT+0800 (China Standard Time)

The app doesn't support XML data files. It can only download direct .pkg links. See #34

cinnamonmatexfce · Answer 2 · Mon Jan 24 2022 20:26:45 GMT+0800 (China Standard Time)

The app doesn't support XML data files. It can only download direct .pkg links. See #34

OK thank you.
How actually TONY handles splitted multipart .pkg games? Is this a known procedure or it still needs to be discovered / reverse-engineered?

Damián Parrino · Answer 3 · Mon Jan 24 2022 21:37:42 GMT+0800 (China Standard Time)

The PS3 background download manager is probably in charge of handling that, but it hasn't been fully reversed so we don't really know.

My assumption is that the PS3 parses the XML, and adds all the parts to the background manager that downloads and merges everything as a single file.

A good test for someone with access to the store, would be to queue one of these big xml downloads, and once it's queued then just pause it. Next, with either HEN or CFW, grab all the tasks files (d0.pdb, d1.pdb, f0.pdb, etc) from the /dev_hdd0/vsh/game_pkg/ folder.
With those files we could probably get a clear view of how it is handled and maybe it can be replicated on pkgi.

cinnamonmatexfce · Answer 4 · Mon Jan 24 2022 22:47:01 GMT+0800 (China Standard Time)

I didn't fully understand all the story behind TONY shutting down PSN for PS3... Is nowadays possible to access PSN and purchase PSN games from it? I would glady try it with my account in my PS3.
Only thing I should care is disabling syscalls+clear history? Or should I take extra precautions?

Damián Parrino · Answer 5 · Mon Jan 24 2022 22:59:23 GMT+0800 (China Standard Time)

They reverted that, so currently on latest firmware (4.88) you can access the store and even buy new content for the PS3. (but not from the store's web site, only from the store app on the PS3)
I think they don't allow adding funds on the PS3 store app, so you need for example to add funds first, and then just do the purchase.

Removing syscalls + clear history should be enough, yes. If you want additional safety and you have patience, you could install a 4.88 dual-boot firmware (so it acts just like OFW), then after the purchase you can go back and re-install CFW (without any patch or jailbreak).

cinnamonmatexfce · Answer 6 · Thu Jan 27 2022 03:55:09 GMT+0800 (China Standard Time)

The PS3 background download manager is probably in charge of handling that, but it hasn't been fully reversed so we don't really know.

My assumption is that the PS3 parses the XML, and adds all the parts to the background manager that downloads and merges everything as a single file.

A good test for someone with access to the store, would be to queue one of these big xml downloads, and once it's queued then just pause it. Next, with either HEN or CFW, grab all the tasks files (d0.pdb, d1.pdb, f0.pdb, etc) from the /dev_hdd0/vsh/game_pkg/ folder.
With those files we could probably get a clear view of how it is handled and maybe it can be replicated on pkgi.

Please note this: while downloading from within the Store, if I choose to put the task in background, then it makes me wait about 4 minutes telling me preparation in progress.
During these four minutes of waiting (note I'm still inside the PlayStation Store APP) there is no network activity (I can tell because leds of the router's ethernet port aren't constantly blinking) but a lot of hard disk activity is happening instead (I see harddisk's led of the PS3 steady blinking)!

So, what should I report to you?

"Background Download" files (copied after the completion of the 4 minutes preparation in progress)?
or
"Normal" files (ungracefully shutting down PS3 while downloading from within the PlayStation Store)?

Damián Parrino · Answer 7 · Thu Jan 27 2022 04:14:56 GMT+0800 (China Standard Time)

So, what should I report to you?

"Background Download" files (copied after the completion of the 4 minutes preparation in progress)?
or

"Normal" files (ungracefully shutting down PS3 while downloading from within the PlayStation Store)?

Let's try first with the "background download" files generated after the 4 minutes preparation process. Once the Background download is ready, just pause the download, grab the pdb files and all the stuff in the task folder /dev_hdd0/vsh/....

Btw, I assume you'll also find a huge empty file that is a placeholder for the download. No need to share that file as I think it has nothing for this research.

We can start with those files as they should be small and easy to review, if that doesn't help we'll look into the other option (ungraceful shutdown, etc)

cinnamonmatexfce · Answer 8 · Thu Jan 27 2022 05:23:18 GMT+0800 (China Standard Time)

So, what should I report to you?

"Background Download" files (copied after the completion of the 4 minutes preparation in progress)?
or

"Normal" files (ungracefully shutting down PS3 while downloading from within the PlayStation Store)?

Let's try first with the "background download" files generated after the 4 minutes preparation process. Once the Background download is ready, just pause the download, grab the pdb files and all the stuff in the task folder /dev_hdd0/vsh/....

Inside the attached dev_hdd0--vsh--task.zip you'll find 00000001 and 00000002 folders from dev_hdd0/vsh/task/.
I don't know why there's also the 00000001 folder.
Instead, the 00000002 folder is the one where PS3 is actually downloading the PSN PKG game.

Btw, I assume you'll also find a huge empty file that is a placeholder for the download. No need to share that file as I think it has nothing for this research.

Yup, I confirm.

Damián Parrino · Answer 9 · Thu Jan 27 2022 21:40:56 GMT+0800 (China Standard Time)

I was reviewing the files and all the links are embedded in the pdb files, so my guess confirms that the background downloader downloads each link in order and stores everything in the big empty "placeholder" file.

When I have a chance, I'll try to replicate and generate such files manually and see if the ps3 actually downloads the parts.

cinnamonmatexfce · Answer 10 · Sat Apr 30 2022 19:25:37 GMT+0800 (China Standard Time)

Hey @bucanero , any news / updates? 😉

Thanks

Damián Parrino · Answer 11 · Thu May 12 2022 00:49:40 GMT+0800 (China Standard Time)

Hey @bucanero , any news / updates? 😉

Thanks

no, I didn't had any time to work on PS3 homebrews or check this download issue. I have been focusing on PS4 stuff lately, but at some point I'll go back to ps3 stuff and take a look.

For the moment these kind of links should be downloaded on a PC, merged into a single file, and then installed as a regular .pkg

cinnamonmatexfce · Answer 12 · Thu May 12 2022 02:20:06 GMT+0800 (China Standard Time)

No problem and thanks for your kind work, pal!! 😎💪

Damián Parrino · Answer 13 · Thu Apr 20 2023 06:23:15 GMT+0800 (China Standard Time)

Leaving the source XML of these task files, for future reference:

<hfs_manifest>
<file_name>EP0101-NPEB02182_00-MGS4MAINGAME0000_bg_1_a625a8e6ac40a4f6228733858b44bca3e8c003e9.pkg</file_name>
<file_size>28598265664</file_size>
<number_of_split_files>7</number_of_split_files>
<pieces file_size="4687500000" hash_value="0a727a8f7c0a1992770f90f0505c23aa816dea79" index="0" url="http://xxx/xxx_00.pkg"/>
<pieces file_size="4687500000" hash_value="cbd20df7687aee236792c4d913d9d67e5df766c6" index="1" url="http://xxx/xxx_01.pkg"/>
<pieces file_size="4687500000" hash_value="ef4c3fcb50ef29f44b882d03a07eab4f71595b31" index="2" url="http://xxx/xxx_02.pkg"/>
<pieces file_size="4687500000" hash_value="9a49afbb21d15a2d65547c43777c7c09e8142d4b" index="3" url="http://xxx/xxx_03.pkg"/>
<pieces file_size="4687500000" hash_value="58f67181ae14e566796d5aa55831c8131b78e40b" index="4" url="http://xxx/xxx_04.pkg"/>
<pieces file_size="4687500000" hash_value="95233845784a1d30fcf0572614ec4d8bbc402d64" index="5" url="http://xxx/xxx_05.pkg"/>
<pieces file_size="473265664" hash_value="8a27fa16663d4c6570b4a9909fdf911f689cceb1" index="6" url="http://xxx/xxx_06.pkg"/>
</hfs_manifest>