Sshuttle not working in combination with WireGuard on macOS Ventura 13.4.1
tbosnjak opened this issue · comments
My setup is as fallows:
#wg.conf
[Interface]
PrivateKey = XXXXXXXXX
Address = 192.168.100.3/32
DNS = 192.168.100.1
[Peer]
PublicKey = XXXXXXXXXX
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = XXXXXXX:13231
sshuttle has no special configuration and it is in version:
# sshuttle --version 17:09:10
1.1.1
I'm using kuberntes pod as bastion and using kuttle
The bastion pod image is alpine based with python version:
# python -V
Python 3.11.4
The command used to start sshuttle is:
sshuttle --dns -r ${sshuttlePo} --exclude 44.205.64.79/32 -e kuttle 44.0.0.0/8 10.250.0.0/16 10.251.0.0/16
Sudoers is set to allow starting of sshutlle without password.
I can confirm that WireGuard works as expected and sshuttle when there is no WireGurad works as expected.
When I connect first to WireGurad and then start sshuttle, dns resolution starts to fail.
I did a quick WireShark check and found out that dns request goes throught the WG interface and dns reply gets back, but the reply doens't get back to lo0 interface.
# sudo pfctl -s all 12:36:17
Password:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat-anchor "com.apple/*" all
rdr-anchor "com.apple/*" all
rdr-anchor "sshuttle6-12300" all
rdr-anchor "sshuttle-12300" all
FILTER RULES:
scrub-anchor "com.apple/*" all fragment reassemble
anchor "com.apple/*" all
anchor "sshuttle6-12300" all
anchor "sshuttle-12300" all
DUMMYNET RULES:
dummynet-anchor "com.apple/*" all
STATES:
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:61931 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64670 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53038 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55589 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:56887 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53058 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51126 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:55009 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55009 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:55198 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55198 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:62635 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:62635 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64880 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:54363 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:54363 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:51397 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51397 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:56766 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:56766 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:52968 MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:58557 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:56887 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:53058 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:51126 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:52968 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:58557 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53229 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:61931 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:64523 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64523 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:52969 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:52969 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:64690 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64690 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:51493 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51493 MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:64880 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:64670 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:53038 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:55589 -> 192.168.100.1:53 SINGLE:NO_TRAFFIC
INFO:
Status: Enabled for 0 days 00:03:48 Debug: Urgent
State Table Total Rate
current entries 41
searches 1060000 4649.1/s
inserts 6790 29.8/s
removals 6749 29.6/s
Counters
match 679939 2982.2/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
dummynet 0 0.0/s
invalid-port 0 0.0/s
TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
grev1.first 120s
grev1.initiating 30s
grev1.estblished 1800s
esp.first 120s
esp.estblished 900s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 6000 states
adaptive.end 12000 states
src.track 0s
LIMITS:
states hard limit 10000
app-states hard limit 10000
src-nodes hard limit 10000
frags hard limit 5000
tables hard limit 1000
table-entries hard limit 200000
TABLES:
OS FINGERPRINTS:
696 fingerprints loaded
#sudo netstat -nr 15s 10:20:03
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default link#18 UCSg utun3
default 172.20.10.1 UGScIg en0
8.8.8.8 link#18 UHW3Ig utun3 11
20.50.46.239 link#18 UHWIig utun3
23.89.10.203 link#18 UHW3Ig utun3 40
23.89.56.117 link#18 UHW3Ig utun3 42
23.89.82.86 link#18 UHW3Ig utun3 40
23.89.83.47 link#18 UHW3Ig utun3 9
69.26.167.48 link#18 UHW3Ig utun3 9
92.123.213.105 link#18 UHW3Ig utun3 100
92.123.213.155 link#18 UHWIig utun3
108.138.36.89 link#18 UHWIig utun3
127 127.0.0.1 UCS lo0
127.0.0.1 127.0.0.1 UH lo0
142.251.5.188 link#18 UHWIig utun3
169.254 link#11 UCS en0 !
170.72.4.197 link#18 UHW3Ig utun3 10
170.72.23.12 link#18 UHW3Ig utun3 10
170.72.41.43 link#18 UHW3Ig utun3 40
170.72.74.201 link#18 UHW3Ig utun3 41
170.72.75.212 link#18 UHW3Ig utun3 41
170.72.134.40 link#18 UHW3Ig utun3 40
170.72.148.152 link#18 UHW3Ig utun3 42
170.72.165.102 link#18 UHW3Ig utun3 42
170.72.166.240 link#18 UHW3Ig utun3 42
170.72.234.3 link#18 UHW3Ig utun3 100
172.20.10/28 link#11 UCS en0 !
172.20.10.1/32 link#11 UCS en0 !
172.20.10.1 16:c8:8b:66:ca:64 UHLWIir en0 363
172.20.10.3/32 link#11 UCS en0 !
172.20.10.3 90:9c:4a:d0:1:76 UHLWI lo0
192.168.100.1 link#18 UHWIig utun3
192.168.100.3 192.168.100.3 UH utun3
224.0.0/4 link#18 UmCS utun3
224.0.0/4 link#11 UmCSI en0 !
239.255.255.250 1:0:5e:7f:ff:fa UHmLWI en0
239.255.255.250 link#18 UHmW3I utun3 129
255.255.255.255/32 link#18 UCS utun3
255.255.255.255/32 link#11 UCSI en0 !
Internet6:
Destination Gateway Flags Netif Expire
default fe80::14c8:8bff:fe66:ca64%en0 UGcIg en0
default fe80::%utun0 UGcIg utun0
default fe80::%utun1 UGcIg utun1
default fe80::%utun2 UGcIg utun2
::1 ::1 UHL lo0
2a01:599:a17:49fb::/64 link#11 UC en0
2a01:599:a17:49fb:46a:fc92:d95d:cfe0 90:9c:4a:d0:1:76 UHL lo0
2a01:599:a17:49fb:95c1:69c7:794d:30e6 16:c8:8b:66:ca:64 UHLWI en0
2a01:599:a17:49fb:d17d:7a7c:2782:6da4 90:9c:4a:d0:1:76 UHL lo0
fe80::%lo0/64 fe80::1%lo0 UcI lo0
fe80::1%lo0 link#1 UHLI lo0
fe80::%en5/64 link#4 UCI en5
fe80::aede:48ff:fe00:1122%en5 ac:de:48:0:11:22 UHLI lo0
fe80::aede:48ff:fe33:4455%en5 ac:de:48:33:44:55 UHLWIi en5
fe80::%en0/64 link#11 UCI en0
fe80::14c8:8bff:fe66:ca64%en0 16:c8:8b:66:ca:64 UHLWIir en0
fe80::18d2:bd00:ffab:3ccb%en0 90:9c:4a:d0:1:76 UHLI lo0
fe80::bcb2:a8ff:fe15:708d%awdl0 be:b2:a8:15:70:8d UHLI lo0
fe80::bcb2:a8ff:fe15:708d%llw0 be:b2:a8:15:70:8d UHLI lo0
fe80::%utun0/64 fe80::99b:f4ad:44dd:eae2%utun0 UcI utun0
fe80::99b:f4ad:44dd:eae2%utun0 link#14 UHLI lo0
fe80::%utun1/64 fe80::e75c:af9d:6831:9791%utun1 UcI utun1
fe80::e75c:af9d:6831:9791%utun1 link#15 UHLI lo0
fe80::%utun2/64 fe80::ce81:b1c:bd2c:69e%utun2 UcI utun2
fe80::ce81:b1c:bd2c:69e%utun2 link#16 UHLI lo0
ff00::/8 ::1 UmCI lo0
ff00::/8 link#4 UmCI en5
ff00::/8 link#11 UmCI en0
ff00::/8 link#12 UmCI awdl0
ff00::/8 link#13 UmCI llw0
ff00::/8 fe80::99b:f4ad:44dd:eae2%utun0 UmCI utun0
ff00::/8 fe80::e75c:af9d:6831:9791%utun1 UmCI utun1
ff00::/8 fe80::ce81:b1c:bd2c:69e%utun2 UmCI utun2
ff01::%lo0/32 ::1 UmCI lo0
ff01::%en5/32 link#4 UmCI en5
ff01::%en0/32 link#11 UmCI en0
ff01::%utun0/32 fe80::99b:f4ad:44dd:eae2%utun0 UmCI utun0
ff01::%utun1/32 fe80::e75c:af9d:6831:9791%utun1 UmCI utun1
ff01::%utun2/32 fe80::ce81:b1c:bd2c:69e%utun2 UmCI utun2
ff02::%lo0/32 ::1 UmCI lo0
ff02::%en5/32 link#4 UmCI en5
ff02::%en0/32 link#11 UmCI en0
ff02::%utun0/32 fe80::99b:f4ad:44dd:eae2%utun0 UmCI utun0
ff02::%utun1/32 fe80::e75c:af9d:6831:9791%utun1 UmCI utun1
ff02::%utun2/32 fe80::ce81:b1c:bd2c:69e%utun2 UmCI utun2
Just to be transparent, I have no clue about MacOS networking stack.
I checked the following issues, without any success:
#706
#563
I tried to apply the patch proposed in: https://github.com/azolotko/sshuttle/pull/1/files, no luck neither.
Any help will be appreciated.