Fix fingerprinting logic for mitm if using Chromium
piercefreeman opened this issue · comments
As it stands today all proxies fail the TLS fingerprinting test so a remote server could easily detect if we're using a proxy as an intermediary layer. Modify our proxy TLS ClientHello payload to mirror the browser's logic if the user-agent coming from the browser matches either Chrome or Chromium.
Specifically the TLS fingerprinting (at least for Ja3) utilizes the following payload signatures:
TLSVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats
In mitmproxy, which came out near the top across the performance trials, the handshake is located in this class definition.
Fully Wireshark logs are included below. These are originally captured by the fingerprinting utility poetry run benchmark fingerprint execute --output-directory {directory} and isolated to view the headfull chromium no proxy & proxy tests for comparison.
Chromium client (no proxy)
TLSv1.3 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 512
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 508
Version: TLS 1.2 (0x0303)
Random: 14542521e71dc25762d64ea539106ce5f1110c6a13df85f432462a2bac5f563d
Session ID Length: 32
Session ID: cb74f97a5edf4cc464ca98c267e8bb2750b283e63cbb851db6b543ca996ad578
Cipher Suites Length: 32
Cipher Suites (16 suites)
Cipher Suite: Reserved (GREASE) (0x6a6a)
Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 403
Extension: Reserved (GREASE) (len=0)
Type: Reserved (GREASE) (6682)
Length: 0
Data: <MISSING>
Extension: server_name (len=15)
Type: server_name (0)
Length: 15
Server Name Indication extension
Server Name list length: 13
Server Name Type: host_name (0)
Server Name length: 10
Server Name: freeman.vc
Extension: extended_master_secret (len=0)
Type: extended_master_secret (23)
Length: 0
Extension: renegotiation_info (len=1)
Type: renegotiation_info (65281)
Length: 1
Renegotiation Info extension
Renegotiation info extension length: 0
Extension: supported_groups (len=10)
Type: supported_groups (10)
Length: 10
Supported Groups List Length: 8
Supported Groups (4 groups)
Supported Group: Reserved (GREASE) (0x4a4a)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: session_ticket (len=0)
Type: session_ticket (35)
Length: 0
Data (0 bytes)
Extension: application_layer_protocol_negotiation (len=14)
Type: application_layer_protocol_negotiation (16)
Length: 14
ALPN Extension Length: 12
ALPN Protocol
ALPN string length: 2
ALPN Next Protocol: h2
ALPN string length: 8
ALPN Next Protocol: http/1.1
Extension: status_request (len=5)
Type: status_request (5)
Length: 5
Certificate Status Type: OCSP (1)
Responder ID list Length: 0
Request Extensions Length: 0
Extension: signature_algorithms (len=18)
Type: signature_algorithms (13)
Length: 18
Signature Hash Algorithms Length: 16
Signature Hash Algorithms (8 algorithms)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: SM2 (4)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (5)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (6)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Hash Algorithm Hash: SHA512 (6)
Signature Hash Algorithm Signature: RSA (1)
Extension: signed_certificate_timestamp (len=0)
Type: signed_certificate_timestamp (18)
Length: 0
Extension: key_share (len=43)
Type: key_share (51)
Length: 43
Key Share extension
Client Key Share Length: 41
Key Share Entry: Group: Reserved (GREASE), Key Exchange length: 1
Group: Reserved (GREASE) (19018)
Key Exchange Length: 1
Key Exchange: 00
Key Share Entry: Group: x25519, Key Exchange length: 32
Group: x25519 (29)
Key Exchange Length: 32
Key Exchange: ecf15674b992b224c6502474d70d2b571c53011b07c30360cc32e423e21c085d
Extension: psk_key_exchange_modes (len=2)
Type: psk_key_exchange_modes (45)
Length: 2
PSK Key Exchange Modes Length: 1
PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)
Extension: supported_versions (len=7)
Type: supported_versions (43)
Length: 7
Supported Versions length: 6
Supported Version: Reserved (GREASE) (0xcaca)
Supported Version: TLS 1.3 (0x0304)
Supported Version: TLS 1.2 (0x0303)
Extension: compress_certificate (len=3)
Type: compress_certificate (27)
Length: 3
Algorithms Length: 2
Algorithm: brotli (2)
Extension: application_settings (len=5)
Type: application_settings (17513)
Length: 5
ALPS Extension Length: 3
Supported ALPN List
Supported ALPN Length: 2
Supported ALPN: h2
Extension: Reserved (GREASE) (len=1)
Type: Reserved (GREASE) (27242)
Length: 1
Data: 00
Extension: padding (len=205)
Type: padding (21)
Length: 205
Padding Data: 000000000000000000000000000000000000000000000000000000000000000000000000…
[JA3 Fullstring: 771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-21,29-23-24,0]
[JA3: cd08e31494f9531f560d64c695473da9]
Default mitmproxy
TLSv1.3 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 321
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 317
Version: TLS 1.2 (0x0303)
Random: f830e2468ebb57d6e05ce4ae7d7cc399c79e8e612a68bcb8e10cca93597e138a
Session ID Length: 32
Session ID: 898de1e8f1e27943975be722fbfb722c720acaae82dee12af90c59af4c6205cc
Cipher Suites Length: 58
Cipher Suites (29 suites)
Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
Cipher Suite: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xccaa)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 186
Extension: server_name (len=15)
Type: server_name (0)
Length: 15
Server Name Indication extension
Server Name list length: 13
Server Name Type: host_name (0)
Server Name length: 10
Server Name: freeman.vc
Extension: ec_point_formats (len=4)
Type: ec_point_formats (11)
Length: 4
EC point formats Length: 3
Elliptic curves point formats (3)
EC point format: uncompressed (0)
EC point format: ansiX962_compressed_prime (1)
EC point format: ansiX962_compressed_char2 (2)
Extension: supported_groups (len=22)
Type: supported_groups (10)
Length: 22
Supported Groups List Length: 20
Supported Groups (10 groups)
Supported Group: x25519 (0x001d)
Supported Group: secp256r1 (0x0017)
Supported Group: x448 (0x001e)
Supported Group: secp521r1 (0x0019)
Supported Group: secp384r1 (0x0018)
Supported Group: ffdhe2048 (0x0100)
Supported Group: ffdhe3072 (0x0101)
Supported Group: ffdhe4096 (0x0102)
Supported Group: ffdhe6144 (0x0103)
Supported Group: ffdhe8192 (0x0104)
Extension: session_ticket (len=0)
Type: session_ticket (35)
Length: 0
Data (0 bytes)
Extension: application_layer_protocol_negotiation (len=14)
Type: application_layer_protocol_negotiation (16)
Length: 14
ALPN Extension Length: 12
ALPN Protocol
ALPN string length: 2
ALPN Next Protocol: h2
ALPN string length: 8
ALPN Next Protocol: http/1.1
Extension: encrypt_then_mac (len=0)
Type: encrypt_then_mac (22)
Length: 0
Extension: extended_master_secret (len=0)
Type: extended_master_secret (23)
Length: 0
Extension: signature_algorithms (len=42)
Type: signature_algorithms (13)
Length: 42
Signature Hash Algorithms Length: 40
Signature Hash Algorithms (20 algorithms)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
Signature Hash Algorithm Hash: SHA512 (6)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: ed25519 (0x0807)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (7)
Signature Algorithm: ed448 (0x0808)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (8)
Signature Algorithm: rsa_pss_pss_sha256 (0x0809)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (9)
Signature Algorithm: rsa_pss_pss_sha384 (0x080a)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (10)
Signature Algorithm: rsa_pss_pss_sha512 (0x080b)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (11)
Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: SM2 (4)
Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (5)
Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
Signature Hash Algorithm Hash: Unknown (8)
Signature Hash Algorithm Signature: Unknown (6)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
Signature Hash Algorithm Hash: SHA512 (6)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: SHA224 ECDSA (0x0303)
Signature Hash Algorithm Hash: SHA224 (3)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: SHA224 RSA (0x0301)
Signature Hash Algorithm Hash: SHA224 (3)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: SHA224 DSA (0x0302)
Signature Hash Algorithm Hash: SHA224 (3)
Signature Hash Algorithm Signature: DSA (2)
Signature Algorithm: SHA256 DSA (0x0402)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: DSA (2)
Signature Algorithm: SHA384 DSA (0x0502)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: DSA (2)
Signature Algorithm: SHA512 DSA (0x0602)
Signature Hash Algorithm Hash: SHA512 (6)
Signature Hash Algorithm Signature: DSA (2)
Extension: supported_versions (len=5)
Type: supported_versions (43)
Length: 5
Supported Versions length: 4
Supported Version: TLS 1.3 (0x0304)
Supported Version: TLS 1.2 (0x0303)
Extension: psk_key_exchange_modes (len=2)
Type: psk_key_exchange_modes (45)
Length: 2
PSK Key Exchange Modes Length: 1
PSK Key Exchange Mode: PSK with (EC)DHE key establishment (psk_dhe_ke) (1)
Extension: key_share (len=38)
Type: key_share (51)
Length: 38
Key Share extension
Client Key Share Length: 36
Key Share Entry: Group: x25519, Key Exchange length: 32
Group: x25519 (29)
Key Exchange Length: 32
Key Exchange: 71985c611ab4481d73684720d88d5ee822f3e8c06e1f9b7602d18451ac8aae64
[JA3 Fullstring: 771,4866-4867-4865-49195-49199-49196-49200-52393-52392-158-159-52394-49187-49191-49161-49171-49188-49192-49162-49172-103-107-156-157-60-61-47-53-255,0-11-10-35-16-22-23-13-43-45-51,29-23-30-25-24-256-257-258-259-260,0-1-2]
[JA3: 1fc1c530fdab845a0a2b05d0b5335bcb]
We'd like some automatic way to compare these fingerprint packet captures, especially for the values that are involved in Ja3. The human readable strings in the above dumps are the result of a manual mapping that wireshark defines based on the iana specification.
Start work on a utility that can compare the set of values in each field and flag disagreements.
-
Convert from the raw packet format into format close to wireshark's output - we can use the bundled tshark utility to convert into json:
tshark -r mitmproxy-chrome_headfull-no_proxy.pcap -T json > output.json -
Dumps will contain multiple packet signals.
Filter for the IP of our fingerprint destination server:
[ip][dst] = {Server IP}Method to filter for our hello client packet:
[tls][tls.record][tls.handshake] -
Isolate values relevant to the fingerprint:
TLSVersion: [tls.record][tls.handshake][tls.handshake.version] Ciphers: [tls.record][tls.handshake][tls.handshake.ciphersuites][tls.handshake.ciphersuite*] Extensions: [tls.record][tls.handshake][tls.handshake.extensions_length] EllipticCurves: ["Extension: supported_groups (len=10)"][tls.handshake.extensions_supported_groups][tls.handshake.extensions_supported_group*] EllipticCurvePointFormats: ["Extension: ec_point_formats (len=2)"][tls.handshake.extensions_ec_point_formats][tls.handshake.extensions_ec_point_format]
NOTE: iana has changed the name of the EllipticCurves payload since the original Ja3 paper came out. It is now known as "supported groups." (see here)
Also confirmed this via the Ja3 source, which parses the 0x0a (val:10) extension value prefix byte as the elliptic_curve and the 0x0b (val: 11) extension value as the elliptic_curve_point_format.
Partial raw payload of the hello client packet -
\x00\x9d\x00\x9c\xc0\xa1\xc0\x9d\xc0\xa0\xc0\x9c\x00=\x00<\x005\x00/\x00\xff\x01\x00\x01]\x00\x00\x00\x0f\x00\r\x00\x00\nfreeman.vc\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x00\x0c
\x00\n\x00\x1d\x00\x17\x00\x1e\x00\x19\x00\x18
-> ELLIPTIC
\x00\x10\x00\x0b\x00\t\x08http/1.1\x00\x16\x00\x00\x00\x17\x00\x00\x001\x00\x00\x00\r\x00*\x00(\x04\x03\x05\x03\x06\x03\x08\x07\x08\x08\x08\t\x08\n\x08\x0b\x08\x04\x08\x05\x08\x06\x04\x01\x05\x01\x06\x01\x03\x03\x03\x01\x03\x02
Wireshark Interface -
Note the "Type: supported_groups (10)" row in addition to the content.
The above comparison utility is added in #7
Now turning attention to the mitmproxy hello client handshake implementation.
Some generally relevant files:
- https://github.com/mitmproxy/mitmproxy/blob/6cfde13b02afe60f7317e4b2212ce0ab9d5fd1e0/mitmproxy/contrib/kaitaistruct/tls_client_hello.py
- https://github.com/mitmproxy/mitmproxy/blob/6cfde13b02afe60f7317e4b2212ce0ab9d5fd1e0/mitmproxy/contrib/kaitaistruct/tls_client_hello.ksy
This TLSLayer layer appears to be the primary class responsible for the proxy->server handshake. Specifically the TlsStartClientHook class is important as this hook indicates that the negotiation between the proxy and server has begun:
TlsStartClientHook: https://github.com/mitmproxy/mitmproxy/blob/a6ede600f3023b68c1e9ef0905d906aba4d21df8/mitmproxy/proxy/layers/tls.py#L187
TLSLayer Handshake: https://github.com/mitmproxy/mitmproxy/blob/a6ede600f3023b68c1e9ef0905d906aba4d21df8/mitmproxy/proxy/layers/tls.py#L303
This handshake references the self.tls.do_handshake() implementation, which itself is set here. TlsStartServerHook indicates that an addon is expected to initialize data.ssl_conn, which is the variable that's passed to self.tls.
This addon injection is defined here (there's another implementation in the file for client->proxy but this is the one we care about for server connections):
This in turn routes to the TLS helpers that return an OpenSSL.SSL object.
There's a good technical writeup on changing curl to use boringssl + changing cyphers to mirror chrome's:
https://lwthiker.com/reversing/2022/02/20/impersonating-chrome-too.html
https://github.com/lwthiker/curl-impersonate
https://github.com/lwthiker/curl-impersonate/blob/main/chrome/curl_chrome104
OpenSSL's python implementation for SSL that's used by mittm:
https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/SSL.py
References most binding files from:
https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/_util.py
And then the binding from:
https://github.com/pyca/cryptography/blob/main/src/cryptography/hazmat/bindings/openssl/binding.py
There's some precedent for creating python bindings into boringssl, ie. by compiling a similar cffi as cryptography does during installation time:
https://github.com/pyca/cryptography/blob/34412de1b28eb268be959a79d754cb64a6c8bb79/src/_cffi_src/build_openssl.py#L72
The boringssl mitmproxy addon looks to involves:
- Building a python binding for boringssl similar to cryptography's Binding
- Build a more specific CLI wrapper similar to pyopenssl.SSL
- Mirror Chromium's boringssl integration
Saving this particular task for a rainy day. In the meantime the integration of an existing hello client mimicry library is more straightforward in Go: https://github.com/refraction-networking/utls. Instead going to target gomitmproxy and goproxy for the first implementations of fingerprint mirroring.