Document the current state of the fuzzer

[luke-goddard]

Hi, I really like this project but as an outsider, I found the project a little confusing. Your README.md says 'Fuzzer and test suite' and your documentation only talks about the test suite. By looking through the code I would assume the fuzzer is only partially implemented. It would save users a bit of time to mention the state of the fuzzer. I'm quite interested in the project (specifically the fuzzer) and might be a future contributor.

[Hubert Kario]

so, the fuzzer part both is and isn't implemented :)

it's implemented in that there are ways for modifying sent messages: https://tlsfuzzer.readthedocs.io/en/latest/modifying-messages.html#modifying-messages fuzzer-like

but it isn't implemented in that the way those modifiers are usually used is through exhaustive iteration; for example, if we have a 2048 bit RSA signature, we flip every bit of the plaintext and see if the message is rejected, then when it's used in the test case, we may run just few random examples of that full list, but we generate the full list:

tlsfuzzer/scripts/test-tls13-certificate-verify.py

Lines 533 to 571 in 5b32de5

 # check that fuzzed signatures are rejected 
 for pos in range(numBytes(private_key.n)): 
     for xor in [0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80]: 
         conversation = Connect(hostname, port) 
         node = conversation 
         ciphers = [CipherSuite.TLS_AES_128_GCM_SHA256, 
                    CipherSuite.TLS_EMPTY_RENEGOTIATION_INFO_SCSV] 
         ext = {} 
         groups = [GroupName.secp256r1] 
         ext[ExtensionType.key_share] = key_share_ext_gen(groups) 
         ext[ExtensionType.supported_versions] = \ 
             SupportedVersionsExtension().create([(3, 4), (3, 3)]) 
         ext[ExtensionType.supported_groups] = \ 
             SupportedGroupsExtension().create(groups) 
         ext[ExtensionType.signature_algorithms] = \ 
             SignatureAlgorithmsExtension().create(sig_algs) 
         ext[ExtensionType.signature_algorithms_cert] = \ 
             SignatureAlgorithmsCertExtension().create(RSA_SIG_ALL) 
         node = node.add_child(ClientHelloGenerator( 
             ciphers, extensions=ext)) 
         node = node.add_child(ExpectServerHello()) 
         node = node.add_child(ExpectChangeCipherSpec()) 
         node = node.add_child(ExpectEncryptedExtensions()) 
         node = node.add_child(ExpectCertificateRequest()) 
         node = node.add_child(ExpectCertificate()) 
         node = node.add_child(ExpectCertificateVerify()) 
         node = node.add_child(ExpectFinished()) 
         node = node.add_child(CertificateGenerator(X509CertChain([cert]))) 
         node = node.add_child(CertificateVerifyGenerator( 
             private_key, padding_xors={pos:xor})) 
         node = node.add_child(FinishedGenerator()) 
         node = node.add_child(ExpectAlert( 
             AlertLevel.fatal, AlertDescription.decrypt_error)) 
         node.add_child(ExpectClose()) 
  
         scheme = SignatureScheme.toRepr(sigalg) 
         conversations_long["check that fuzzed signatures are rejected." + 
                            " Malformed {0} - xor {1} at {2}".format( 
                            certType, hex(xor), pos)] = conversation 

only combinatorial explosion stops us from testing more than 1 flipped bit at a time: if we would generate those modifications on the fly (which is possible, they are pre-generated so that we can deterministically run them or skip them) we could have a script that keeps fuzzing the signature with more and more complex patterns till the end of time :)

to sum it up: if you want to use it as a fuzzer, you totally can, but I limit that kind of use and instead try to create custom tailored test cases, as those are more likely to find issues in a cryptographic protocol like TLS and use fuzzing-like behaviour just to speed up execution

I'm glad you find it interesting enough to consider contributing, feel free to ask questions, I'll gladly discuss solutions or provide explanations for the behaviour; I'm also open to any suggestions to improve documentation, I know it's not in the best shape, but it's a priority for me to fix it

[Hubert Kario]

@luke-goddard do you have any follow-up questions?