Could you please consider to update your java library to handle secondary compression of delta file sections
RudyBisiaux opened this issue · comments
All is in the title.
I have a project with a file and a patch given by a server. When i try do decode with this library on my Android application i have this error:
"java.io.IOException: Secondary compression of delta file sections is not supported"
i specify that the server don't use this java library to generate the file but still use a VCDiff library.
Thanks for your answers.
The original RFC didn't specify any secondary compression schemes. If there's a followup, could you link me to it?
It could be a proprietary compressor that the code generating it uses. Do you know what server you're using? Adding support for user-registered secondary compression is an option.
It's also possible that you found a bug or the diff is corrupt. If there isn't anything sensitive in it, could you send me the dictionary and the delta?
That error shouldn't happen. You should have seen "Secondary compression is not supported" before it was able to get that far. I'm really curious to see the delta file.
Hi, thanks for reply
I use the xdelta to generate patch on server part (i own it) :
https://github.com/jmacd/xdelta
As you can see on command-line there is on option to disabled secondary compression
-S [djw|fgk] enable/disable secondary compression
but don't change the behavior of the library
i use your library in my android application to apply patch on the original file.
thanks again.
I'm looking into it. Does it work fine if you disable secondary compression?
Hi,
If i don't use secondary compression i have the same error this is why i'm pretty confused with this ^^.
Thanks again.
Couple of things: xdelta3 is pretty non-standard by default. You need to add -S to disable secondary compression (it defaults to lzma), and -A to disable adding an "app header."
After adding those, vcdiff-java and Google's open-vcdiff both failed to decode the patch. The error was something like
The length of the delta encoding does not match the size of the header plus the sizes of the data sections
In the short-term, I have a test case I can add (that will fail) with an xdelta3 diff, and an improvement to an exception message. Longer term, I'll try to poke around the simple diff to figure out if open-vcdiff or xdelta3 is wrong.
Ok, Thanks a lot for you informations and actions.
Sincerely.
Try using -S -A -n when using xdelta3. Without -n, an adler32 chcecksum is added to the window section, but this seems to be non-standard.
If I have time to add in xdelta3 extensions, which would you rather have: adler32 checksums or lzma secondary compression?
Hi,
When i use the -S -A -n options i have a exception like :
Source segment length (91) is larger than target file (0)
my dictionary length is 2287616
and my encoding length is 312551 (if it can help).
i think Izma secondary compression is more valuable to have
Thanks again !
Can you send me the dictionary and the delta?
One other thing to try: in the Java code, make sure you didn't disable target matches, e.g.
VCDiffDecoderBuilder.builder()
.withAllowTargetMatches(false)
.buildStreaming();
allowTargetMatches should probably be set to true (the default) if you're seeing that error. Or try setting it to false for fun.
Also make sure your expected file is <64M. If it's larger, you might need to increase the maximum target file and window sizes.
The reason for fine-grained control like this is corrupt deltas can decode to some pretty large output, and capping memory usage (target matches, target window size) and target size can prevent this.
Hi,
I unfortunately can't give you the dictionary and the delta for now, the data inside are sensitive, but i will looking to give you something, without certainty.
I try to play with the withAllowTargetMatches(true) or false
i still have the same error with true option :
Source segment length (91) is larger than target file (0)
and have this error with false option
Delta file contains VCD_TARGET flag, which is not allowed by current decoder settings
Thanks a lot to be concerned
What about just the delta? There's definitely stuff in it, but if the dictionary is 10x bigger, it's a lot harder to reconstruct in a meaningful way. If you've encoded text, you can see the text just by looking at the delta file.
Ok, i have the permission to give those file. Hope it will help you.
[redacted]
thanks again
Got it. I'll late a look later today.
open-vcdiff produced this error, too (I get the same error when I swap dictionaries):
$ vcdiff decode -dictionary product_20170518 -delta testpatch -target out
ERROR: Source segment length (91) is larger than target file (0)
Error trying to decode data chunk of length 983489
Since vcdiff-java is based on open-vcdiff, this could be a common bug, or it could be an xdelta3 bug. I'll continue investigating.
The delta you sent me has an xdelta3 app header. When I hit this before, adding -A fixed the problem. If you do head -c 6 testpatch | hexdump -C, you get
00000000 d6 c3 c4 00 04 06 |......|
the 04 is the vcdiff Hdr_Indicator. xdelta3 defines them here.
Once I saw that, I tried using xdelta3 3.0.10 to create a diff:
xdelta3 -e -S -n -A -s product_20170518 product_20170529 testpatch
I didn't have any problems applying the patch to product_20170518, and the output matched product_20170529.
Were you able to get your diff to work by adding -A to the xdelta3 command?
Hi, Sorry for the wait.
This work perfectly !
Thank you a lot for your assistance.
Best regards