Watch Dogs: Legion research & discussion (FAT5, version 13)
LennardF1989 opened this issue · comments
So, since you are already scanning/generating based on the namehashes in the FAT, I reckon your code is already newer than what is last pushed. Regardless, I did some research into the new file format. Perhaps we can document some things here and trigger others to help along. The modding potential of Watch Dogs Legion seems great.
FAT (BugFileV3):
- The new version of FAT is 13, the current code only goes up to 8.
- It looks like the Target.Win32 is now the Win64 target.
- Platform Windows seems to be 8 (wild guess)
- Unknown 70 is still 70 :P I skipped that check to let it continue.
- Before scanning entries, we have to skip 12 bytes.
- Entries are now 20 bytes long, not 16 anymore (I added a V13 serializer).
- First 4 bytes are still the name-hash. The hashing algorithm seems unchanged compared to Watch Dogs (As is evident by your scanning endeavors).
- It looks like the 16-20th byte is actually the offset (or part thereof). In a handful of my tests that range consistently contains only a single entry for 0 (eg. the first file in the DAT).
Will update as I go through it.
Sounds.dat has a few handy entries to figure out the format:
- Entry 0 (offset 0): 0A C7 EB 0B 0B 13 AA AA 06 EA 2C 00 00 00 00 00 18 A8 B3 00
- Entry 1 (offset 2cea06): B1 B2 55 2A B0 65 F4 A5 70 A7 2B 80 81 3A 0B 00 C0 9D AE 00
- Entry 2 (offset 589176): 8C BF CF AC 61 E3 E5 BC 76 04 00 80 5D 24 16 00 D8 11 00 00
- Entry 3 (offset 5895EC): -
I'm going to assume the RIFF-headers are uncompressed, meaning the V8 serializer holds up for the NameHash, CompressionScheme and CompressedSize (except its UncompressedSize in this case?).
Well-defined now.
The big hurdle here will be proper LZ4LW (LZ4 variant) support. I have this mostly understood except for an edge case where there's trailing literal bytes which the length is somehow inferred to by a packed length value before the LZ4 compressed data.
Looking great code-wise so far :) What is your method of figuring out the file-names that go with the hashes? Memory scanning and investigating?
Got an example of the edge-case you are trying to figure out?
commonengine.fat
BBF771F0C529DF2A @40605, 257434 bytes (12756 compressed bytes, scheme LZ4LW)
I believe this is engine\settings\defaultrenderconfig.xml but I may be misremembering.
Has a header value of 0x27 (39) - single byte, followed by LZ4LW compressed data. 18 literal bytes at the end without any LZ4 tag to indicate them. Somehow their decompressor knows without anything obvious in place to stop and copy the remaining bytes.
As for file lists, initial file list was contributed by a friend who's also researching the game. I have a functioning runtime hook for the game that dumps names that get hashed by the game to a file.
Got it - if you want I'm also actively playing the game, so I can run the hook at the same time to gather data.
Also any chance you can push the experimental LZ4LW you are working to a dev-branch or something?
Also, random thought because I've seen this before. Could it be that compression happens in particular block-sizes and that in this case the remaining 18 bytes is simply because they were not large enough to be a block during compression? This could explain why it's always at the end.
Decompressor-wise they probably have some sort of streaming in place that keeps decompressing until there is nothing left to decompress. Then the resulting size - expected size = number of bytes to read and append.
Thanks! Been playing around with it and seems to work for a large number of files. Eg. I've extracted almost all LUA files and they decompile to actual code just fine. Only occasionally one has some trailing bytes. Under DecompressLZ4LW I've made the modification to just read the byte and not do anything with it (not even in the compressed size).
Seems like commonengine.dat has a few situations where decompressing yields a partially bad file, most peculiar would be "gpudatabase.xml" which has corrupted data all over (but consistently). Whereas if you compare that to a faulty extract of say "defaultrealtreeconfig.xml" only the first few bytes are corrupt (causing the commented header to look weird).
As for the defaultrenderconfig.xml you mentioned - the right number of bytes to NOT read is 19. So entry.CompressedSize - 19 in the LZ4LWDecoderStream yields the proper output for the remaining bytes to be append raw from the offset the input-stream is at then. I've tried a few things to come up with a number that makes sense (eg. (first byte / 2) - 1 or (first byte - 1) / 2) then floor came to mind, or if 39 somehow depicts the number of blocks to decompress), but it's an odd one alright. I'm pretty sure it's a flag-byte with a number packed in there to tell the decompiler how to chunk the bytes into equal packages.
EDIT: Epiphany... If you read the byte and shift it by 1, 39 becomes 19, which is the number of bytes to subtract from the CompressedSize.
EDIT EDIT: Nope, this doesn't add up for entry 4 which is 217 in size and also has 27 as a first byte, but only 11 trailing bytes.
I investigated this a little further, extracting a few entries, one of which also has the 0x27 first byte.
My conclusion is that the first byte doesn't say anything about the size of the decompression, as the later test-case uses the full uncompressed size.
Rather the "tailing bytes" are actually supposed to be "decompressed" using a "copy literal" instruction, I presume. This could mean there is simply a bug/oversight in the decoder.
As for the first byte, my first guess is it is some sort of flag that describes the most ideal block size to decompress it or something, given the values are used a lot of times independent of the input/output size.
Yup, pretty sure it's a bug in the decoder, this is a HEX string of the ending part of the entry you mentioned:
F9 02 0D 0A 0D 0A 3C 2F 50 72 6F 66 69 6C 65 3E 0D 0A 0D 0A
Notice the starting F9 02, looks a lot like a token for L4Z, so I processed it manually.
F9
_LiteralLength = 15
_MatchLength = 9 + 4
15 means read next byte and add
15 + 2 = 17
finish with proper CompressedSize will append the final byte.
When I manually set the starting position of the input stream to this location, it does exactly this.
Getting closer, for some mystical reason it goes wrong at the single-last token:
06 09 03 07 F9 02
06 in this case means read two bytes (09 03), then rest at 07 for the next token. This should however be F9 when we look at it with human eyes. The decoder is following protocol as it should, yet the file Watch Dogs has compressed just slapped an extra 07 in there. We would have to find another file that has leading bytes like this to verify if this is something consistent or not.
I reverse engineered the decompression from the game. They do in-place decompression like here:
private static Int32 ReadNum(byte[] input, ref Int32 inputPosition)
{
Int32 num = 0, b;
do
{
b = input[inputPosition++];
num += b;
} while (b == 0xFF);
return num;
}
public static Int32 Decompress(byte[] buffer, int inputStartPosition, int first)
{
Int32 inputEndPosition = buffer.Length;
Int32 inputPosition = inputStartPosition, outputPosition = 0;
while (outputPosition < inputEndPosition)
{
byte token = buffer[inputPosition++];
int proceed = token >> 4, copy = token & 0x0F;
if (proceed == 0x0F)
{
proceed += ReadNum(buffer, ref inputPosition);
}
if (proceed > 0)
{
//for (Int32 i = 0; i < proceed; ++i)
//{
// buffer[outputPosition++] = buffer[inputPosition++];
//}
Buffer.BlockCopy(buffer, inputPosition, buffer, outputPosition, proceed);
outputPosition += proceed;
inputPosition += proceed;
}
if (inputPosition == inputEndPosition || outputPosition >= inputPosition)
{
break;
}
int offset = buffer[inputPosition] | (buffer[inputPosition + 1] << 8);
inputPosition += 2;
if (offset >= 0xE000)
{
int extra = buffer[inputPosition++];
offset += extra * 0x2000;
}
if (offset == 0 || outputPosition < offset)
{
throw new InvalidOperationException("invalid lz4 stream");
}
if (copy == 0x0F)
{
copy += ReadNum(buffer, ref inputPosition);
}
copy += 4;
for (Int32 i = 0; i < copy; ++i)
{
buffer[outputPosition] = buffer[outputPosition - offset];
++outputPosition;
}
}
return outputPosition;
}
}I read header of compressed block and call decompression like this:
private static void DecompressLz4(Entry entry, Stream input, Stream output)
{
Console.WriteLine("... Size: " + entry.UncompressedSize);
int inputPosition = (int) (entry.UncompressedSize - entry.CompressedSize);
var uncompressedBytes = new byte[entry.UncompressedSize];
if (input.Read(uncompressedBytes, inputPosition, (int) entry.CompressedSize) != entry.CompressedSize)
{
throw new EndOfStreamException("could not read all compressed bytes");
}
int first = uncompressedBytes[inputPosition++];
if (first >= 0x80)
{
first = (first - 0x80) + ((int)uncompressedBytes[inputPosition++] << 7);
if (first >= 0x4000)
{
first = (first - 0x4000) + ((int)uncompressedBytes[inputPosition++] << 14);
if (first >= 0x200000)
{
first = (first - 0x200000) + ((int)uncompressedBytes[inputPosition++] << 21);
}
}
}
Int32 actualUncompressedLength = Compression.LZ4.Decompress(uncompressedBytes, inputPosition, first);
if (actualUncompressedLength < 0)
{
throw new InvalidOperationException("LZ4 decompression failed at position " + (- actualUncompressedLength - 1));
}
if (actualUncompressedLength != uncompressedBytes.Length)
{
throw new InvalidOperationException("LZ4 decompression is not complete");
}
output.WriteBytes(uncompressedBytes);
}Code had two branches, one branch had optimizations and used first, but other unoptimized branch didn't use first at all.
@ahmet-celik Have you tested this with commonengine.dat? Specifically the gpudatabase.xml and the defaultrenderconfig.xml.
The first one does very weird decompression, the second one has weird trailing text added because they did something funky with the final bytes in the compressed bytes. The assumption made earlier that there are "leading bytes" that have to be added is incorrect.
EDIT: I was too quick, it kind of works, but doesn't make fully sense yet.
EDIT: @ahmet-celik Can you also try to figure out the code branch that used the first-value? Because we also have/want to repack assets we kind have to understand what first actually means and does to the code.
EDIT: Just tested it again, in the end this does the same as Ricks code (including the first byte), but instead it loads the full file into memory, whereas Ricks version uses streaming. If you have some information on how the first byte somehow related to the situation in defaultrenderconfig.xml that would be really great @ahmet-celik!
EDIT: Investigated even further, the 07 F9 02 is actually valid and copies the second-last part of the XML. This is however where the decompression should end and append the final 18 bytes as-is. So my earlier statements about the 07 are wrong and misinterpreted.
@LennardF1989 I don’t have access to Watch Dogs 2 anymore. I did this research back in the summer of 2019. Optimized branch was similar to optimized lz4 code.
@ahmet-celik Ah okay, we're researching WD: Legion in this thread :) This could mean the answer is probably easier to be found in WD2 @gibbed due to Denuvo etc.
@gibbed Figured it out with the help of Sir Kane. The first byte (the way you read it with PackedS32 is fine by the way) signifies where the "safe decompression loop" starts. If the current offset in the decompressed output is x bytes before the end, the flow of decompression changes like this: https://github.com/lz4/lz4/blob/dev/lib/lz4.c#L1781 and this https://github.com/lz4/lz4/blob/dev/lib/lz4.c#L1913
The trick is in the shortcut where a few things are assumed in how to read them.
So, first is only important for performance.
The game does in place decompression as described here: https://github.com/lz4/lz4/blob/9cf3f106a8dea906ff4550f749112e9e89536678/doc/lz4_manual.html#L395.
@LennardF1989 Going to the safe loop still processes tokens though, and in my sample case, this leads to invalid behavior as far as I could tell (unless I did something wrong, also totally possible).
@gibbed Finally figured it out, and its so stupid I could facepalm myself. Once the decompression gets into the safe decoding phase, you have to check a condition before you attempt to read the next token: If the current outputPosition in the buffer is bigger than the current inputPosition in the buffer, just stop and don't touch the buffer anymore. The final bits are already in the right location then.
@LennardF1989 If you look at my decompress code that’s what I do, and that’s why I copied compressed bytes to the end of decompress buffer.
if (inputPosition == inputEndPosition || outputPosition >= inputPosition) {
break;
}
@ahmet-celik Correct, but you do it right after the literal phase, which is the wrong location from the algorithms perspective. It should be checked right before reading the next token. On top of that, the remaining bytes will then already be at the right location and won't need additional processing/copying. Other than that and some additional sanity checks, your code is feature complete :)
In case of the streaming version, it's going to be a nasty thing to get in. The Stream is going to need a persistent inputPosition and outputPosition variable, so you can keep track of where you are in the whole process (even if reading 2048 bytes per run).
@LennardF1989 There is also check for that in loop condition.