Discrepancy with reference gpb implementation when enc/dec uint64
vihu opened this issue · comments
I have been trying to understand the reason why there is a discrepancy between this library vs google's reference python implementation when trying to encode/decode 64 bit unsigned integers.
I created two simple repositories to test this out:
python: https://github.com/vihu/gpb-py-test
erlang: https://github.com/vihu/gpb-erl-test
They have the exact same proto definition. The difference is that the python one will clearly error out trying to enc/dec a 72 bit integer (when the proto specified a 64 bit one) whereas the erlang one seems to happily enc/dec it. Please let me know if this was chose by design or if there is an actual issue here. Thanks!
Just oversight. gpb decodes some overly long ints, but has a limit:
3> {ok, test, Code} = gpb_compile:string(test, "syntax='proto3'; message test { uint64 id = 1; }", [maps, binary]).
4> code:load_binary(test, "test.erl", Code).
5> [{NBits, try test:decode_msg(test:encode_msg(#{id => (1 bsl NBits)-1}, test), test) catch _:_ -> no end} || NBits <- lists:seq(63,72)].
[{63,#{id => 9223372036854775807}},
{64,#{id => 18446744073709551615}},
{65,#{id => 36893488147419103231}},
{66,#{id => 73786976294838206463}},
{67,#{id => 147573952589676412927}},
{68,#{id => 295147905179352825855}},
{69,#{id => 590295810358705651711}},
{70,#{id => 1180591620717411303423}},
{71,no},
{72,no}]
Is it important for you that 64 bits is enforced, or are you just curious why?
It means that gpb will construct and parse protobuffs which are not valid in other implementations. This is a problem for us with interoperability, yes.
The larger problem here is that that same code works when you change id to a uint32, which is even more radically out of spec.
Regarding constructing, there is the verify option to let you generate code that verifies types and ranges on encoding erl→protobuf. Here is an example:
18> {ok, test3, Code3} = gpb_compile:string(test3, "syntax='proto3'; message test { uint64 id = 1; }", [maps, binary, {verify,always}]).
19> code:load_binary(test3, "test3.erl", Code3).
20> test3:encode_msg(#{id => (1 bsl 70)-1}, test).
** exception error: {gpb_type_error,{{value_out_of_range,uint64,unsigned,64},
[{value,1180591620717411303423},{path,"test.id"}]}}
in function test3:mk_type_error/3 (test3.erl, line 351)
in call from test3:v_msg_test/3 (test3.erl, line 330)
in call from test3:encode_msg/3 (test3.erl, line 68)
~ ~ ~ ~
Regarding deserialization, the gpb-py-test example does not actually test that. There is an exception setting¹ a value out of range, the code never reaches deserialization of it. I took the liberty to modify the gpb-py-test to really deserialize an overly long uint64, and there is no exception. See vihu/gpb-py-test#1. It seems the value is truncated to 64 bits, whereas in gpb, it is not. Have you found any explicitly specified semantics what should happen deserializing an overly long integer?
¹ Edit: adding that the python code was setting the value by parsing the text format, no by parsing the wire-format, which gpb does.
I pushed tentative fix to a branch, decoding-overly-long-ints. On decoding, it truncates uint32 and uint64 to 32 or 64 bits, to mimic the Python.
Side-note: it will still abort decoding of any ints or uints longer than 70 bits, ie uint71 values or larger will not result in a truncated value, but a decoder error instead. The reason for this is that it would otherwise be possible to construct an (big) value such that decoding it could cause the decoder loop in gpb's generated code (in its current form) to chew up large amounts of cpu and memory. So I thought best to set a limit to avoid denial-of-service like attacks.
Thanks @tomas-abrahamsson for the detailed explanations. Its helpful to learn that verify_option exists, although I thought something like that would be set to always on by default.
That's a good point. It would mean some sort of change of interface, but I'll consider it for when it is time for gpb-5.x.x.
Edit: the text below is wrong, I had mixed things up.
Hi again, I was writing some unit tests for decoding, to verify that it masks correctly for the various integer types, int32, uint32, sint32 and ditto for 64 bits, and noted some odd values from the reference Python protobuf decoder, so I checked what the reference C++ protobuf decoder outputs, and results differ!
So I think the conclusion is that the behaviour is not well defined what should happen on decoding integers with too many bits on the wire. On the decoding-overly-long-ints branch, gpb works like the C++ protobuf reference decoder in this regard, ie it differs from the Python reference decoder.
(mixed results deleted)
Mea culpa, I had mixed things up. Sorry! Upon revisiting, the Python implementation and the C++ implementation actually do decode to the same values for more than 32/64 bits. My last commit was wrong, please disregard it.
The fix to mask the decoded value to 32/64 bits, in case more bits were decoded from the bytes on the wire, was merged and included in the just-released 4.17.1, so I'll close this one. Thanks for notifying. Please feel free to re-open or open a new if there would be anything.