7cai / esp_wifi_repeater

A full functional WiFi Repeater (correctly: a WiFI NAT Router)

esp_wifi_repeater

A full functional WiFi Repeater (correctly: a WiFi NAT Router)

This is a proof of concept implementation of a WiFi NAT router on the esp8266. It can be used as range extender for an existing WiFi network. The esp acts as STA and as soft-AP and transparently forwards any IP traffic through it. As it uses NAT no routing entries are required neither on the network side nor on the connected stations. Stations are configured via DHCP by default in the 192.168.4.0/24 net and receive their DNS responder address from the existing WiFi network.

Measurements show, that it can achieve about 5 Mbps in both directions, so even streaming is possible.

The router also allows for remote monitoring (or packet sniffing), e.g. with Wireshark.

Some details are explained in this video: https://www.youtube.com/watch?v=OM2FqnMFCLw

Building and Flashing

To build this binary you download and install the esp-open-sdk (https://github.com/pfalcon/esp-open-sdk). Make sure, you can compile and download the included "blinky" example. Then download or clone my version of the esp-open-lwip library and its source tree from (https://github.com/martin-ger/esp-open-lwip). Use it to replace the directory "esp-open-lwip" in the esp-open-sdk tree. "make clean" in the esp_open_lwip dir and once again a "make" in the upper esp_open_sdk directory will do the job. This installs a new version of the liblwip_open.a that contains the NAT-features.

Then download this source tree in a separate directory and adjust the BUILD_AREA variable in the Makefile and any desired options in user/user_config.h.

Build the esp_wifi_repeater firmware with "make". "make flash" flashes it onto an esp8266.

If you want to use the precompiled binaries you can flash them with "esptool.py --port /dev/ttyUSB0 write_flash -fs 32m 0x00000 firmware/0x00000.bin 0x10000 firmware/0x10000.bin" (use -fs 8m for an ESP-01)

On Windows you can flash it using the "ESP8266 Download Tool" available at https://espressif.com/en/support/download/other-tools. Download the two files 0x00000.bin and 0x10000.bin from the firmware directory. For a generic ESP12, a NodeMCU or a Wemos D1 use the following settings (for an ESP-01 change FLASH SIZE to "8Mbit"):

For some reasons that I still do not understand, the firmware compiled with the V2.0.0 SDK fails to start on some ESP-01 modules. If you experience these problem, use the files from the directory firmware_sdk_1.5.4 instead (addresses 0x00000 and 0x40000).

Usage

The Firmware starts with the following default configuration:

• ssid: ssid, pasword: password
• ap_ssid: MyAP, ap_password: none, ap_on: 1, ap_open: 1
• network: 192.168.4.0/24

This means it connects to the internet via AP ssid,password and offers an open AP with ap_ssid MyAP. This default can be changed in the file user_config.h. The default can be overwritten and persistenly saved to flash by using a console interface. This console is available either via the serial port at 115200 baud or via tcp port 7777 (e.g. "telnet 192.168.4.1 7777" from a connected STA).

Use the following commands for an initial setup:

• set ssid your_home_router's_SSID
• set password your_home_router's_password
• set ap_ssid ESP's_ssid
• set ap_password ESP's_password
• show (to check the parameters)
• save
• reset

After reboot it will connect to your home router and itself is ready for stations to connect.

The console understands the following commands:

Basic commands (enough to get it working in nearly all environments):

• help: prints a short help message
• set [ssid|password] value: changes the settings for the uplink AP (WiFi config of your home-router)
• set [ap_ssid|ap_password] value: changes the settings for the soft-AP of the ESP (for your stations)
• show [config|stats]: prints the current config or some status information and statistics
• save [dhcp]: saves the current config parameters [+ the current DHCP leases] to flash
• reset [factory]: resets the esp, optionally resets WiFi params to default values
• lock: locks the current config, changes are not allowed
• unlock password: unlocks the config, requires password of the network AP
• quit: terminates a remote session

Advanced commands:

• set network ip-addr: sets the IP address of the internal network, network is always /24, router is always x.x.x.1
• set dns dns-addr: sets a static DNS address that is distributed to clients via DHCP
• set dns dhcp: configures use of the dynamic DNS address from DHCP, default
• set ip ip-addr: sets a static IP address for the ESP in the uplink network
• set ip dhcp: configures dynamic IP address for the ESP in the uplink network, default
• set netmask netmask: sets a static netmask for the uplink network
• set gw gw-addr: sets a static gateway address in the uplink network
• scan: does a scan for APs
• set ap_on [0|1]: selects, wheter the soft-AP is disabled (ap_on=0) or enabled (ap_on=1, default)
• set ap_open [0|1]: selects, wheter the soft-AP uses WPA2 security (ap_open=0, automatic, if an ap_password is set) or open (ap_open=1)
• set speed [80|160]: sets the CPU clock frequency (default 80 Mhz)
• set [upstream_kbps|downstream_kbps] bitrate: sets a maximum upstream/downstream bitrate (0 = no limit)
• set vmin voltage: sets the minimum battery voltage in mV. If Vdd drops below, the ESP goes into deep sleep. If 0, nothing happens
• set vmin_sleep time: sets the time interval in seconds the ESP sleeps on low voltage
• set config_port portno: sets the port number of the console login (default is 7777, 0 disables remote console config)
• portmap add [TCP|UDP] external_port internal_ip internal_port: adds a port forwarding
• portmap remove [TCP|UDP] external_port: deletes a port forwarding
• monitor [on|off|acl] port: starts and stops monitor server on a given port
• acl [from_sta|to_sta] [TCP|UDP|IP] src-ip [src_port] desr-ip [dest_port] [allow|deny|allow_monitor|deny_monitor]: adds a new rule to the ACL
• acl [from_sta|to_sta] clear: clears the whole ACL
• show acl: shows the defined ACLs and some stats
• set acl_debug [0|1]: switches ACL debug output on/off - a denied packets will be logged to the terminal

Status LED

In default config GPIO2 is configured to drive a status LED (connected to GND) with the following indications:

• permanently on: started, but not successfully connected to the AP (no valid external IP)
• flashing (1 per second): working, connected to the AP
• unperiodically flashing: working, traffic in the internal network

In user_config.h an alternative GPIO port can be configured. When configured to GPIO1, it works with the buildin blue LED on the ESP-01 boards. However, as GPIO1 ist also the UART-TX-pin this means, that the serial console is not working. Configuration is then limited to network access.

Port Mapping

In order to allow clients from the external network to connect to server port on the internal network, ports have to be mapped. An external port is mapped to an internal port of a specific internal IP address. Use the "portmap add" command for that. Port mappings can be listed with the "show" command and are saved with the current config.

However, to make sure that the expected device is listening at a certain IP address, it has to be ensured the this devices has the same IP address once it or the ESP is rebooted. To achive this, either fixed IP adresses can be configured in the devices or the ESP has to remember its DHCP leases. This can be achived with the "save dhcp" command. It saves the current state and all DHCP leases, so that they will be restored after reboot. DHCP leases can be listed with the "show stats" command.

Monitoring

From the console a monitor service can be started ("monitor on [portno]"). This service mirrors the traffic of the internal network in pcap format to a TCP stream. E.g. with a "netcat [external_ip_of_the_repeater] [portno] | sudo wireshark -k -S -i -" from an computer in the external network you can now observe the traffic in the internal network in real time. Use this e.g. to observe with which internet sites your internals clients are communicating. Be aware that this at least doubles the load on the esp and the WiFi network. Under heavy load this might result in some packets beeing cut short or even dropped in the monitor session. CAUTION: leaving this port open is a potential security issue. Anybody from the local networks can connect and observe your traffic.

Firewall

The ESP router has a integrated basic firewall. ACLs (Access Control Lists) can be applied to the SoftAP interface. This is a cornerstone in IoT security, when the router is used to bring other IoT devices into the internet. It can be used to prevent e.g. third-party IoT devices from "calling home", being misused as malware bots, and to protect your home network with PCs, tablets and phones from being visible to home automation devices.

The two ACL lists are named "from_sta" and "to_sta" for incoming and outgoing packets. ACLs are defined in "CISCO IOS style". The following example will allow for outgoing local broadcasts (for DHCP), UDP 53 (DNS), and TCP 1883 (MQTT) to a local broker, any other packets will be blocked:

• acl from_sta clear
• acl from_sta IP any 255.255.255.255 allow
• acl from_sta UDP any any any 53 allow
• acl from_sta TCP any any 192.168.0.0/16 1883 allow
• acl from_sta IP any any deny

ACLs for the "to_sta" direction may be defined as well, but this is usually not required, as the reverse direction is quite well protected against unsolicited traffic by the NAT transation.

ACLs consist of filtering rules that are processed for each packet. Each rule consists of a protocol (IP, TCP, or UDP), source address/port, destination address/port, as well as an action "allow" or "deny". In case of plain IP no ports, only addresses are given. IP rules include TCP and UDP packets. Addresses can be given as subnet addresses in the "/" notation, e.g. 192.168.178.0/24. Also "any" can be used as wildcard, it matches on any address or portnumber. A rule is defined by the "acl" command:

• acl [from_sta|to_sta] [TCP|UDP|IP] src-ip [src_port] desr-ip [dest_port] [allow|deny|allow_monitor|deny_monitor]

The rules are processed top-down in the order of their appearance in the list. The first rule that matches a packet is applied and determies whether a packet is allowed (and forwarded) or denied (and dropped). This means, special cases first, general rules at the end. If there are rules in an ACL all packets that don't match any rule are denied by default. Thus, the last rule "from_sta IP any any deny" in the example above is not really needed, as it is the default anyway. If an ACL is empty, all packets are allowed.

Definition of ACL rules works also top-down: a new rule is always added at the end of a list. To change an ACL you first have to clear it completely (acl from_sta clear) and then rebuild it. ACLs are saved with the config. "show acl" will print out the ACLs plus statistics on the number of hits for each rule and the overall number of allowed and denied packets.

With the command "set acl_debug 1" a summary of all denied packets is printed to the console. Also, an MQTT topic can publishe this summary. This can be used for firewall configuration to determine which rules are required to get the connected devices working. It also gives a hint if, if unexpected traffic happens (and is denied).

For deeper analysis the monitoring service can be used (even denied packets are reported to the monitor before they are dropped). When the monitor is started with the "monitor acl port" command, ACLs can be used as online filters. All rules that are defined as "allow_monitor" instead of "allow" and "deny_monitor" instead of "deny" are processed as usual, resulting in allowing of forwarding a packet, but they also send the packet to the monitor. Thus a list of rules that basically "allow" or "allow_monitor" all packets still makes sense, as it can be used to select already during catpure time which packet should be recorded. E.g. a lists:

• acl from_sta clear

• acl from_sta IP 192.168.0.0/16 any allow_monitor

• acl from_sta IP any any allow

• acl to_sta clear

• acl to_sta IP any 192.168.0.0/16 allow_monitor

• acl to_sta IP any any allow

will allow all packets and also select all packets for monitoring that go from a station to the 192.168.0.0/16 (local)subnet and from the 192.168.0.0/16 to a station. Of course such a filter can be applied also after the capture to a full monitoring trace, but if you already know, what you are looking for, these online filters will help to reduce monitoring overhead drastically. It can also be used to debug all deny firewall rules by simply using "deny_monitor" instead of deny.

Bitrate Limits

By setting upstream_kbps and downstream_kbps to a value != 0 (0 is the default), you can limit the maximum bitrate of the ESP's AP. This value is a limit that applies to the traffic of all connected clients. Packets that would exeed the defined bitrate are dropped. The traffic shaper uses the "Token Bucket" algorithm with a bucket size of currently four times the bitrate per seconds, allowing for bursts, when there was no traffic before.

MQTT Support

Since version 1.3 the router has a build-in MQTT client (thanks to Tuan PM for his library https://github.com/tuanpmt/esp_mqtt). This can help to integrate the router/repeater into the IoT. A home automation system can e.g. make decisions based on infos about the currently associated stations, it can switch on and of the repeaters (e.g. based on a time schedule), or it can simply be used to monitor the load. The router can be connected either to a local MQTT broker or to a publically available broker in the cloud. However, currently it does not support TLS encryption.

By default the MQTT client is disabled. It can be enabled by setting the config parameter "mqtt_host" to a hostname different from "none". To configure MQTT you can set the following parameters:

• set mqtt_host IP_or_hostname: IP or hostname of the MQTT broker ("none" disables the MQTT client)
• set mqtt_user username: Username for authentication ("none" if no authentication is required at the broker)
• set mqtt_user password: Password for authentication
• set mqtt_id clientId: Id of the client at the broker (default: "ESPRouter_xxxxxx" derived from the MAC address)
• set mqtt_prefix prefix_path: Prefix for all published topics (default: "/WiFi/ESPRouter_xxxxxx/system", again derived from the MAC address)
• set mqtt_command_topic command_topic: Topic subscribed to receive commands, same as from the console. (default: "/WiFi/ESPRouter_xxxxxx/command", "none" disables commands via MQTT)
• set mqtt_interval secs: Set the interval in which the router publishs status topics (default: 15s, 0 disables status publication)
• set mqtt_mask mask_in_hex: Selects which topics are published (default: "ffff" means all)

The MQTT parameters can be displayed with the "show mqtt" command.

The router can publish the following status topics periodically (every mqtt_interval):

• prefix_path/Uptime: System uptime since last reset in s (mask: 0x0020)
• prefix_path/Vdd: Voltage of the power supply in mV (mask: 0x0040)
• prefix_path/Bpsin: Bytes/s from stations into the AP (mask: 0x0800)
• prefix_path/Bpsout: Bytes/s from the AP to stations (mask: 0x1000)
• prefix_path/Ppsin: Packets/s from stations into the AP (mask: 0x0200)
• prefix_path/Ppsout: Packets/s from the AP to stations (mask: 0x0400)
• prefix_path/Bin: Total bytes from stations into the AP (mask: 0x0100)
• prefix_path/Bout: Total bytes from the AP to stations (mask: 0x0100)
• prefix_path/NoStations: Number of stations currently connected to the AP (mask: 0x2000)

In addition it can publish on an event basis:

• prefix_path/join: MAC address of a station joining the AP (mask: 0x0008)
• prefix_path/leave: MAC address of a station leaving the AP (mask: 0x0010)
• prefix_path/IP: IP address of the router when received via DHCP (mask: 0x0002)
• prefix_path/ScanResult: Separate topic for the results of a "scan" command (one message per found AP) (mask: 0x0004)
• prefix_path/ACLDeny: A packet has been denied by an ACL rule and has been dropped (mask: 0x0080)

The router can be configured using the following topics:

• command_topic: The router subscribes on this topic and interprets all messages as command lines
• prefix_path/response: The router publishes on this topic the command line output (mask: 0x0001)

If you now want the router to publish e.g. only Vdd, its IP, and the command line output, set the mqtt_mask to 0x0001 | 0x0002 | 0x0040 (= "set mqtt_mask 0043").

Power Management

The repeater monitors its current supply voltage (shown in the "show stats" command). If vmin (in mV, default 0) is set to a value > 0 and the supply voltage drops below this value, it will go into deep sleep mode for vmin_sleep seconds. If you have connected GPIO16 to RST (which is hard to solder on an ESP-01) it will reboot after this interval, try to reconnect, and will continue its measurements. If vmin is saved with the config, it will sleep over and over again, until the supply voltage raises above the threshold. These settings are especially (only?) useful if you have powered the ESP with a (lithium) battery whithout undercharge protection. Then a value of 2900mV-3000mV is probably helpful, as it reduces power consumption of the ESP to a minimum and you have much more time to recharge or replace the battery before damage. This only makes sense, if you have the ESP connected directly to the battery. If you have additional logic, this will still drain the battery.

You can send the ESP to sleep manually once by using the "sleep" command.

Caution: If you save a vmin value higher than the max supply voltage to flash, the repeater will immediatly shutdown every time after reboot. Then you have to wipe out the whole config by flashing blank.bin (or any other file) to 0x0c000.

Known Issues

• Due to the limitations of the ESP's SoftAP implementation, there is a maximum of 8 simultaniously connected stations.
• Configuration via TCP (write_flash) requires a good power supply. A large capacitor between Vdd and Gnd can help if you experience problems here.