nftables全局代理配置寻求帮助
weber110 opened this issue · comments
目前使用的openwrt用的fw4，基于hev-socks5-tproxy的nftables配置、rule/route配置，可以跑通，但还想对比下ipt2tables。
hst那边文档写得比较齐全，网络基础没那么好，没办法基于ss-tproxy的配置转化成nft配置，希望您能针对小白完善下全局代理的设置方法，万分感谢！
下面是目前的设置方法，启动ipt2tables后一直报错
2024-01-02 17:31:29 ERR: [new_nonblock_sockfd] socket(AF_INET, SOCK_STREAM): No file descriptors available
2024-01-02 17:31:29 ERR: [set_tcp_nodelay] setsockopt(-1, TCP_NODELAY): Bad file descriptor
2024-01-02 17:31:29 ERR: [set_tcp_quickack] setsockopt(-1, TCP_QUICKACK): Bad file descriptor
2024-01-02 17:31:29 ERR: [set_tcp_keepalive] setsockopt(-1, SO_KEEPALIVE): Bad file descriptor
2024-01-02 17:31:29 ERR: [tcp_tproxy_accept_cb] connect to 121.37.247.85#30001: Bad file descriptor
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
[tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
- nft配置：
table inet mangle {
set byp4 {
typeof ip daddr
flags interval
elements = { 0.0.0.0/8, 10.0.0.0/8,
127.0.0.0/8, 169.254.0.0/16,
172.16.0.0/12, 192.0.0.0/24,
192.0.2.0/24, 192.88.99.0/24,
192.168.0.0/16, 198.18.0.0/15,
198.51.100.0/24, 203.0.113.0/24,
224.0.0.0/4, 240.0.0.0/4 }
}
set byp6 {
typeof ip6 daddr
flags interval
elements = { ::,
::1,
::ffff:0:0:0/96,
64:ff9b::/96,
100::/64,
2001::/32,
2001:20::/28,
2001:db8::/32,
2002::/16,
fc00::/7,
fe80::/10,
ff00::/8 }
}
chain prerouting {
type filter hook prerouting priority mangle; policy accept;
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
udp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
udp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
udp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
udp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
udp dport 0-65535 tproxy to :1088 meta mark set 0x00000440 accept
}
chain output {
type route hook output priority mangle; policy accept;
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 meta mark set 0x00000440
udp dport 0-65535 meta mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 meta mark set 0x00000440
udp dport 0-65535 meta mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 meta mark set 0x00000440
udp dport 0-65535 meta mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 meta mark set 0x00000440
udp dport 0-65535 meta mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
tcp dport 0-65535 meta mark set 0x00000440
udp dport 0-65535 meta mark set 0x00000440
}
}-
路由配置
ip rule add fwmark 1088 table 100
ip route add local default dev lo table 100 -
启动脚本
./ipt2socks -s 111.111.111.111 -p 30001 -a uid -k pwd
你说的全局代理是：除了保留地址（比如192.168.x.x）外，都重定向至 ipt2socks，从而转发给后面的 socks5 代理？
ipt2socks 的完整日志发一下。
我描述的全局代理也就是透明代理吧，除了局域网内部通信流量，其它都转到socks5代理包含tcp/udp/dns，目前openwrt内部内置了dnsmasq-full监听53端口了。
2024-01-02 17:30:43 INF: [main] server address: 111.111.111.111#30001
2024-01-02 17:30:43 INF: [main] listen address: 127.0.0.1#1088
2024-01-02 17:30:43 INF: [main] listen address: ::1#1088
2024-01-02 17:30:43 INF: [main] udp cache maximum size: 256
2024-01-02 17:30:43 INF: [main] udp socket idle timeout: 60
2024-01-02 17:30:43 INF: [main] number of worker threads: 1
2024-01-02 17:30:43 INF: [main] enable tcp transparent proxy
2024-01-02 17:30:43 INF: [main] enable udp transparent proxy
2024-01-02 17:31:29 ERR: [new_nonblock_sockfd] socket(AF_INET, SOCK_STREAM): No file descriptors available
2024-01-02 17:31:29 ERR: [set_tcp_nodelay] setsockopt(-1, TCP_NODELAY): Bad file descriptor
2024-01-02 17:31:29 ERR: [set_tcp_quickack] setsockopt(-1, TCP_QUICKACK): Bad file descriptor
2024-01-02 17:31:29 ERR: [set_tcp_keepalive] setsockopt(-1, SO_KEEPALIVE): Bad file descriptor
2024-01-02 17:31:29 ERR: [tcp_tproxy_accept_cb] connect to 121.37.247.85#30001: Bad file descriptor
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
2024-01-02 17:31:29 ERR: [tcp_socks5_recv_authresp_cb] recv from 121.37.247.85#30001: Connection reset by peer
...
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
2024-01-02 17:31:30 ERR: [tcp_tproxy_accept_cb] accept tcp4 socket: No file descriptors available
...
看报错信息，应该是nft规则死循环了。
我待会给一个能用的 nft 规则出来，我先本地搞一下。
你的 socks5 代理运行在哪里？应该没有和 ipt2socks 在同一个主机上吧？
你的 socks5 代理运行在哪里？应该没有和 ipt2socks 在同一个主机上吧？
对的，没有用本地s5 server
socks5 server 在 同一局域网 下的 其他主机 上，对吗
不是，socks5 server是外网的，有自己danted搭的也有别人v2搭的。
按顺序执行以下命令
- 创建 proxy 用户(组)，启动 ipt2socks
# 创建proxy组，用于ipt/nft放行
groupadd proxy # 或 addgroup proxy
# 给可执行文件设置setgid权限位
chgrp proxy /path/to/ipt2socks
chmod g+xs /path/to/ipt2socks
#上面两个操作执行过一次就可以了
# 启动ipt2socks进程，不要用-u选项！
ipt2socks -s 服务器ip -p 服务器port -l 1088- nft脚本（应用之前，先清空相应规则链，避免重复！）
table inet mangle {
set byp4 {
typeof ip daddr
flags interval
elements = { 0.0.0.0/8, 10.0.0.0/8,
127.0.0.0/8, 169.254.0.0/16,
172.16.0.0/12, 192.0.0.0/24,
192.0.2.0/24, 192.88.99.0/24,
192.168.0.0/16, 198.18.0.0/15,
198.51.100.0/24, 203.0.113.0/24,
224.0.0.0/4, 240.0.0.0/4 }
}
set byp6 {
typeof ip6 daddr
flags interval
elements = { ::,
::1,
::ffff:0:0:0/96,
64:ff9b::/96,
100::/64,
2001::/32,
2001:20::/28,
2001:db8::/32,
2002::/16,
fc00::/7,
fe80::/10,
ff00::/8 }
}
chain prerouting {
type filter hook prerouting priority mangle; policy accept;
# 放行发往 local 的
fib daddr type local return
# 放行 reply 方向的
ct direction reply return
# 局域网传出的流量
meta l4proto {tcp,udp} ct state new,related fib saddr type != local jump do_proxy
# 本机和局域网流量 => ipt2socks
meta l4proto {tcp,udp} ct mark 1088 tproxy to :1088 meta mark set 1088
}
chain output {
type route hook output priority mangle; policy accept;
# 放行发往 local 的
fib daddr type local return
# 放行 reply 方向的
ct direction reply return
# 放行本机代理进程
skgid proxy return
# 给 connection 打上 mark
meta l4proto {tcp,udp} ct state new,related jump do_proxy
# 给 packet 打上 mark (ip rule)
ct mark 1088 meta mark set 1088
}
chain do_proxy {
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 1088
}
}- ip rule/route
ip rule add fwmark 1088 table 100
ip route add local default dev lo table 100谢谢您抽出时间来解答。
照您提供的示例操作后，启动./ipt2socks -s $sip -p $sport -l 1088 -a $suid -k $spwd -u proxy提示如下：
2024-01-03 14:44:31 INF: [main] listen address: 127.0.0.1#1088
2024-01-03 14:44:31 INF: [main] listen address: ::1#1088
2024-01-03 14:44:31 INF: [main] udp cache maximum size: 256
2024-01-03 14:44:31 INF: [main] udp socket idle timeout: 60
2024-01-03 14:44:31 INF: [main] number of worker threads: 1
2024-01-03 14:44:31 INF: [main] enable tcp transparent proxy
2024-01-03 14:44:31 INF: [main] enable udp transparent proxy
2024-01-03 14:44:31 ERR: [set_ip_transparent] setsockopt(3, IP_TRANSPARENT): Operation not permitted
2024-01-03 14:44:31 ERR: [set_ip_transparent] setsockopt(4, IPV6_TRANSPARENT): Operation not permitted
2024-01-03 14:44:31 ERR: [set_ip_transparent] setsockopt(5, IP_TRANSPARENT): Operation not permitted
2024-01-03 14:44:31 ERR: [set_ip_transparent] setsockopt(6, IPV6_TRANSPARENT): Operation not permitted
猜想是proxy用户没有cap-net-admin权限的缘故吧，所以修改了
skgid proxy return -> skgid root return
同时启动命令改成下面后，应该是流量都没有转发到1088端口来，日志在'enable udp transparent proxy'后没有任何输出
./ipt2socks -s $sip -p $sport -l 1088 -a $suid -k $spwd -u proxy -u root
下面为完整的nft list ruleset打印内容
table inet mangle {
set byp4 {
typeof ip daddr
flags interval
elements = { 0.0.0.0/8, 10.0.0.0/8,
127.0.0.0/8, 169.254.0.0/16,
172.16.0.0/12, 192.0.0.0/24,
192.0.2.0/24, 192.88.99.0/24,
192.168.0.0/16, 198.18.0.0/15,
198.51.100.0/24, 203.0.113.0/24,
224.0.0.0/4, 240.0.0.0/4 }
}
set byp6 {
typeof ip6 daddr
flags interval
elements = { ::,
::1,
::ffff:0:0:0/96,
64:ff9b::/96,
100::/64,
2001::/32,
2001:20::/28,
2001:db8::/32,
2002::/16,
fc00::/7,
fe80::/10,
ff00::/8 }
}
chain prerouting {
type filter hook prerouting priority mangle; policy accept;
fib daddr type local return
ct direction reply return
meta l4proto { tcp, udp } ct state related,new fib saddr type != local jump do_proxy
meta l4proto { tcp, udp } ct mark 0x00000440 tproxy to :1088 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta l4proto { tcp, udp } ct state related,new fib saddr type != local jump do_proxy
meta l4proto { tcp, udp } ct mark 0x00000440 tproxy to :1088 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta l4proto { tcp, udp } ct state related,new fib saddr type != local jump do_proxy
meta l4proto { tcp, udp } ct mark 0x00000440 tproxy to :1088 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta l4proto { tcp, udp } ct state related,new fib saddr type != local jump do_proxy
meta l4proto { tcp, udp } ct mark 0x00000440 tproxy to :1088 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta l4proto { tcp, udp } ct state related,new fib saddr type != local jump do_proxy
meta l4proto { tcp, udp } ct mark 0x00000440 tproxy to :1088 meta mark set 0x00000440
}
chain output {
type route hook output priority mangle; policy accept;
fib daddr type local return
ct direction reply return
meta skgid 1000 return
meta l4proto { tcp, udp } ct state related,new jump do_proxy
ct mark 0x00000440 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta skgid 1000 return
meta l4proto { tcp, udp } ct state related,new jump do_proxy
ct mark 0x00000440 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta skgid 1000 return
meta l4proto { tcp, udp } ct state related,new jump do_proxy
ct mark 0x00000440 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta skgid 1000 return
meta l4proto { tcp, udp } ct state related,new jump do_proxy
ct mark 0x00000440 meta mark set 0x00000440
fib daddr type local return
ct direction reply return
meta skgid 1000 return
meta l4proto { tcp, udp } ct state related,new jump do_proxy
ct mark 0x00000440 meta mark set 0x00000440
}
chain do_proxy {
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 0x00000440
ip daddr @byp4 return
ip6 daddr @byp6 return
ct mark set 0x00000440
}
}
table inet dnsmasq {
chain prerouting {
type nat hook prerouting priority dstnat - 5; policy accept;
meta nfproto { ipv4, ipv6 } udp dport 53 counter packets 473 bytes 30796 redirect to :53 comment "DNSMASQ HIJACK"
}
}
table inet fw4 {
ct helper amanda {
type "amanda" protocol udp
l3proto inet
}
ct helper ftp {
type "ftp" protocol tcp
l3proto inet
}
ct helper RAS {
type "RAS" protocol udp
l3proto inet
}
ct helper Q.931 {
type "Q.931" protocol tcp
l3proto inet
}
ct helper irc {
type "irc" protocol tcp
l3proto ip
}
ct helper pptp {
type "pptp" protocol tcp
l3proto ip
}
ct helper sip {
type "sip" protocol udp
l3proto inet
}
ct helper snmp {
type "snmp" protocol udp
l3proto ip
}
ct helper tftp {
type "tftp" protocol udp
l3proto inet
}
chain input {
type filter hook input priority filter; policy drop;
iifname "lo" accept comment "!fw4: Accept traffic from loopback"
ct state established,related accept comment "!fw4: Allow inbound established and related flows"
tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
iifname { "br-lan", "phy0-ap0" } jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
iifname "eth1" jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
jump handle_reject
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
iifname { "br-lan", "phy0-ap0" } jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
iifname "eth1" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
jump handle_reject
}
chain output {
type filter hook output priority filter; policy accept;
oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
ct state established,related accept comment "!fw4: Allow outbound established and related flows"
oifname { "br-lan", "phy0-ap0" } jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
oifname "eth1" jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
}
chain prerouting {
type filter hook prerouting priority filter; policy accept;
}
chain handle_reject {
meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
reject comment "!fw4: Reject any other traffic"
}
chain syn_flood {
limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
drop comment "!fw4: Drop excess packets"
}
chain input_lan {
jump accept_from_lan
}
chain output_lan {
jump accept_to_lan
}
chain forward_lan {
jump accept_to_wan comment "!fw4: Accept lan to wan forwarding"
jump accept_to_lan
}
chain accept_from_lan {
iifname { "br-lan", "phy0-ap0" } counter packets 486 bytes 32048 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
}
chain accept_to_lan {
oifname { "br-lan", "phy0-ap0" } counter packets 1 bytes 328 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
}
chain input_wan {
meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
meta nfproto ipv4 meta l4proto igmp counter packets 3 bytes 108 accept comment "!fw4: Allow-IGMP"
meta nfproto ipv6 udp dport 546 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCPv6"
ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 3 bytes 228 accept comment "!fw4: Allow-MLD"
icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply, nd-router-solicit, nd-router-advert } limit rate 1000/second counter packets 3 bytes 192 accept comment "!fw4: Allow-ICMPv6-Input"
icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, nd-neighbor-solicit . no-route, nd-neighbor-advert . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Input"
ct status dnat accept comment "!fw4: Accept port redirections"
jump reject_from_wan
}
chain output_wan {
jump accept_to_wan
}
chain forward_wan {
icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
meta l4proto esp counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-IPSec-ESP"
udp dport 500 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-ISAKMP"
ct status dnat accept comment "!fw4: Accept port forwards"
jump reject_to_wan
}
chain accept_to_wan {
meta nfproto ipv4 oifname "eth1" ct state invalid counter packets 18 bytes 720 drop comment "!fw4: Prevent NAT leakage"
oifname "eth1" counter packets 4452 bytes 285382 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
}
chain reject_from_wan {
iifname "eth1" counter packets 33 bytes 3525 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
}
chain reject_to_wan {
oifname "eth1" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
}
chain dstnat {
type nat hook prerouting priority dstnat; policy accept;
iifname "eth1" jump dstnat_wan comment "!fw4: Handle wan IPv4/IPv6 dstnat traffic"
}
chain srcnat {
type nat hook postrouting priority srcnat; policy accept;
oifname "eth1" jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
}
chain dstnat_wan {
meta nfproto ipv4 fullcone comment "!fw4: Handle wan IPv4 fullcone NAT dstnat traffic"
}
chain srcnat_wan {
meta nfproto ipv4 fullcone comment "!fw4: Handle wan IPv4 fullcone NAT srcnat traffic"
}
chain raw_prerouting {
type filter hook prerouting priority raw; policy accept;
}
chain raw_output {
type filter hook output priority raw; policy accept;
}
chain mangle_prerouting {
type filter hook prerouting priority mangle; policy accept;
}
chain mangle_postrouting {
type filter hook postrouting priority mangle; policy accept;
}
chain mangle_input {
type filter hook input priority mangle; policy accept;
}
chain mangle_output {
type route hook output priority mangle; policy accept;
}
chain mangle_forward {
type filter hook forward priority mangle; policy accept;
iifname "eth1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
oifname "eth1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
}
}
你的 nft 规则已经重复了，存在污染，先重启下系统（或者 nft flush 相关的 mangle 规则链）
猜想是proxy用户没有cap-net-admin权限的缘故吧，所以修改了
skgid proxy return -> skgid root return
这里不能这样改，不然root用户组的所有流量都不会走代理（这就是你后面描述的症状）
我重新编辑了之前回复的内容，这回应该可以了。
你的 nft 规则已经重复了，存在污染，先重启下系统（或者 nft flush 相关的 mangle 规则链）
您指的污染是prerouting/output链内重复出现设置代码吧，这个我也不知道原因，这个nft list ruleset就是重启后打印的。之前用别的s5 client的ruleset-post nft文件，也一样会重复规则，但流量拦截代理还是成功的。
按您最新的关于用户组及权限设置后，和之前直接删除skgid proxy return一样的效果，ipt2socks5启动日志打印后无别的流量日志了。
重启下系统，从干净状态开始