Stream encryption is embarrassingly broken

[Jens Alfke]

It turns out that the stream encryption following the handshake -- Session::encrypt/decrypt and the Cap'n Proto stream adapter that calls them -- is totally broken.

• If the receiver isn't receiving the same size blocks that the writer is writing, it will decrypt garbage. (This should have shown up during my testing ... I think it didn't because the individual writes are small and I was testing on localhost, i.e. the loopback interface, which tends to relay each write to the reader instead of mushing them together into packets.)
• And the encryption is useless anyway because each write ends up being xor'ed with the exact same bitstream.

In my defense, the libSodium stream-cipher API is IMHO really misleading. crypto_stream_xor uses a stream cipher, XSalsa20, and "the ciphertext is the message combined with the output of the stream cipher". What it doesn't say is that the cipher stream doesn't advance as a result of the call. If you call crypto_stream_xor multiple times, the data isn't xor'ed with successive portions of the infinite cipher-stream; instead it resets to the beginning of the key-stream every time.

When real-world encryption started failing I looked closely at the signature of the function and noticed that it doesn't have any mutable state -- since the key is declared as const unsigned char * it's a (conceptually) pure function that will always produce the same output given the same input. libSodium doesn't even seem to have any API to access the real cipher-stream, i.e. a function that pulls new bits from the cipher-stream each time it's called.

In any case, xor'ing with the cipher stream has weaknesses since there's no data validation. An attacker can mess with the cleartext, flipping bits in it by flipping the same bits in the encrypted data. I always knew this wasn't ideal, but apparently it's known to lead to real vulnerabilities.

The solution seems to be to use crypto_secretbox to add a MAC to every write. This then requires prepending the byte-count (in clear) so the receiver knows how large the "box" is. Which happens to be exactly what Scuttlebutt does.

[Jens Alfke]

Fixed with 14b6389. Now using libSodium secretbox API to encrypt each write operation. Adds an overhead of 18 bytes per write, unfortunately. Each write now sends [length][MAC][ciphertext(cleartext)], where the length is a 2-byte bit-endian int.

I have a feeling some may complain that I'm sending each message-length in clear. Tampering with it won't break anything since there are MACs, but an eavesdropper would likely be able to figure out where the message boundaries are. I assume in some cases that could leak some kind of information.

Scuttlebutt's closely-related box-stream protocol gets around this with double encryption: it wraps the fixed-size [length][MAC] part in another secretbox, so you end up with [MAC][ciphertext([length][MAC])][ciphertext(cleartext)]. Which I consider a kludge.