K8S: Support fsnotify and reload when ConfigMap update
winlinvip opened this issue · comments
K8S uses ConfigMap to store configuration files, for example:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: srs-config
data:
srs.conf: |-
listen 1935;
max_connections 1000;
daemon off;
http_api {
enabled on;
listen 1985;
}
http_server {
enabled on;
listen 8080;
}
vhost __defaultVhost__ {
http_remux {
enabled on;
}
hls {
enabled on;
}
}
EOFConfigMap will be mounted as a volume and become a configuration file:
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: srs-deploy
labels:
app: srs
spec:
replicas: 1
selector:
matchLabels:
app: srs
template:
metadata:
labels:
app: srs
spec:
volumes:
- name: config-volume
configMap:
name: srs-config
containers:
- name: srs
image: ossrs/srs:3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 1935
- containerPort: 1985
- containerPort: 8080
volumeMounts:
- name: config-volume
mountPath: /usr/local/srs/conf
EOFYou can view the running pods:
Mac:trunk chengli.ycl$ pod=`kubectl get po|grep srs-deploy|awk '{print $1}'`
Mac:trunk chengli.ycl$ kubectl exec $pod -- cat conf/srs.conf && echo ""
listen 1935;
max_connections 1000;
daemon off;
http_api {
enabled on;
listen 1985;
}
http_server {
enabled on;
listen 8080;
}
vhost __defaultVhost__ {
http_remux {
enabled on;
}
hls {
enabled on;
}
}When ConfigMap changes, you can use fsnotify to receive notifications of file changes, thereby triggering the reload of SRS.
TRANS_BY_GPT3
The paragraph in Aliyun's SLS log service says link is as follows:
Logtail running in DaemonSet mode will periodically request the configuration server to obtain new or updated configurations and perform hot reloading.
Another solution is to listen for changes in the configuration file. However, there is an issue where the fsnotify signal may not be received due to symbolic links: link
Here is an example of Nginx's reload, which is implemented using fsnotify: link
watcher, watcherErr := fsnotify.NewWatcher()
go func() {
event, ok := <-watcher.Events:
if event.Op&fsnotify.Create != fsnotify.Create {
return;
}
if filepath.Base(event.Name) != "..data" {
return;
}
nginxProcess, nginxProcessErr := os.FindProcess(getMasterNginxPid())
nginxProcess.Signal(syscall.SIGHUP)
}
pathToWatch = "/etc/nginx"
if err := watcher.Add(pathToWatch); err != nil {
stderrLogger.Fatal(err)
}You can see that it also listens for changes in the file directory and then sends the SIGHUP (reload) signal to Nginx.
TRANS_BY_GPT3
It is implemented using inotify under Linux.
INOTIFY_INIT(2) Linux Programmer's Manual INOTIFY_INIT(2)
NAME
inotify_init, inotify_init1 - initialize an inotify instance
SYNOPSIS
#include <sys/inotify.h>
int inotify_init(void);
int inotify_init1(int flags);
DESCRIPTION
inotify_init() initializes a new inotify instance and returns a file descriptor associated with a new inotify event queue.
If flags is 0, then inotify_init1() is the same as inotify_init(). The following values can be bitwise ORed in flags to obtain different results.
ferent behavior:
IN_NONBLOCK Set the O_NONBLOCK file status flag on the new open file description. Using this flag saves extra calls to fcntl(2) to
achieve the same result.
IN_CLOEXEC Set the close-on-exec (FD_CLOEXEC) flag on the new file descriptor. See the description of the O_CLOEXEC flag in
open(2) for reasons why this may be useful.
Since inotify returns an fd and can be initialized as non-blocking mode using inotify_init1, reading it using read is an IO operation, so we should be able to use ST to read this fd.
/* the following are legal, implemented events that user-space can watch for */
#define IN_ACCESS 0x00000001 /* File was accessed */
#define IN_MODIFY 0x00000002 /* File was modified */
#define IN_ATTRIB 0x00000004 /* Metadata changed */
#define IN_CLOSE_WRITE 0x00000008 /* Writtable file was closed */
#define IN_CLOSE_NOWRITE 0x00000010 /* Unwrittable file closed */
#define IN_OPEN 0x00000020 /* File was opened */
#define IN_MOVED_FROM 0x00000040 /* File was moved from X */
#define IN_MOVED_TO 0x00000080 /* File was moved to Y */
#define IN_CREATE 0x00000100 /* Subfile was created */
#define IN_DELETE 0x00000200 /* Subfile was deleted */
#define IN_DELETE_SELF 0x00000400 /* Self was deleted */
#define IN_MOVE_SELF 0x00000800 /* Self was moved */
/* the following are legal events. they are sent as needed to any watch */
#define IN_UNMOUNT 0x00002000 /* Backing fs was unmounted */
#define IN_Q_OVERFLOW 0x00004000 /* Event queued overflowed */
#define IN_IGNORED 0x00008000 /* File was ignored */
/* helper events */
#define IN_CLOSE (IN_CLOSE_WRITE | IN_CLOSE_NOWRITE) /* close */
#define IN_MOVE (IN_MOVED_FROM | IN_MOVED_TO) /* moves */
/* special flags */
#define IN_ONLYDIR 0x01000000 /* only watch the path if it is a directory */
#define IN_DONT_FOLLOW 0x02000000 /* don't follow a sym link */
#define IN_EXCL_UNLINK 0x04000000 /* exclude events on unlinked objects */
#define IN_MASK_ADD 0x20000000 /* add to the mask of an already existing watch */
#define IN_ISDIR 0x40000000 /* event occurred against dir */
#define IN_ONESHOT 0x80000000 /* only send event once */
Note: The event list can be referred to here.
TRANS_BY_GPT3
SRS has added two configurations, automatically enabling auto reload under Docker, and disabling auto reload outside of Docker.
# Whether auto reload by watching the config file by inotify.
# default: off
inotify_auto_reload off;
# Whether enable inotify_auto_reload for docker.
# If on, it will set inotify_auto_reload to on in docker, even it's off.
# default: on
auto_reload_for_docker on;
The added log prints are as follows:
[2020-03-12 10:10:50.020][Warn][2914][636][16] enable auto reload for docker
[2020-03-12 10:10:50.021][Trace][2914][636] auto reload watching console.conf, fd=14, wd=1
[2020-03-12 10:10:53.940][Trace][2914][642] inotify event wd=1, mask=0x2, len=0, name=
[2020-03-12 10:10:53.940][Trace][2914][642] reload config, signo=1
[2020-03-12 10:10:54.033][Trace][2914][636] config parse complete
[2020-03-12 10:10:54.033][Trace][2914][636] srs checking config...
[2020-03-12 10:10:54.033][Warn][2914][636][11] stats network use index=0, ip=172.17.0.2
[2020-03-12 10:10:54.033][Warn][2914][636][11] stats disk not configed, disk iops disabled.
[2020-03-12 10:10:54.033][Trace][2914][636] write log to console
[2020-03-12 10:10:54.033][Trace][2914][636] reload http_api success, nothing changed.
[2020-03-12 10:10:54.033][Trace][2914][636] reload http stream success, nothing changed.
[2020-03-12 10:10:54.033][Trace][2914][636] vhost __defaultVhost__ maybe modified, reload its detail.
[2020-03-12 10:10:54.034][Trace][2914][636] ingest nothing changed for vhost=__defaultVhost__
[2020-03-12 10:10:54.034][Trace][2914][636] reload config success.
You can see that there is an additional fd=10, a_inode(inotify):
[root@05181e679dc0 trunk]# lsof -p `ps aux|grep srs|grep objs|awk {'print $2}'`
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
srs 2753 root 7u IPv4 27833 0t0 TCP *:macromedia-fcs (LISTEN)
srs 2753 root 8u IPv4 27836 0t0 TCP *:hsrp (LISTEN)
srs 2753 root 9u IPv4 27839 0t0 TCP *:webcache (LISTEN)
srs 2753 root 10r a_inode 0,13 0 12074 inotify
TRANS_BY_GPT3
If the file is deleted or moved, and then created again (as part of the ConfigMap update logic).
When a file is deleted, including the target of symbolic links being moved or deleted:
[2020-03-12 11:31:16.387][Trace][981][674] inotify watch 2.conf fd=10, watch=1, mask=0xfff, gone=0
[2020-03-12 11:31:20.405][Trace][981][674] inotify event wd=1, mask=0x4, len=0, name=, reload=0, moved=0
[2020-03-12 11:31:20.406][Trace][981][674] inotify event wd=1, mask=0x400, len=0, name=, reload=0, moved=1
[2020-03-12 11:31:20.406][Trace][981][674] inotify event wd=1, mask=0x8000, len=0, name=, reload=0, moved=1
[2020-03-12 11:31:20.406][Trace][981][674] inotify remove fd=10, watch=1, r0=-1
When a file is created, including the creation of the target of symbolic links:
[2020-03-12 11:31:26.412][Trace][981][674] inotify watch 2.conf fd=10, watch=2, mask=0xfff, gone=1
[2020-03-12 11:31:26.414][Trace][981][674] reload config, signo=1
[2020-03-12 11:31:26.437][Trace][981][668] config parse complete
[2020-03-12 11:31:26.437][Trace][981][668] srs checking config...
[2020-03-12 11:31:26.437][Warn][981][668][11] stats network use index=0, ip=172.17.0.2
[2020-03-12 11:31:26.437][Warn][981][668][11] stats disk not configed, disk iops disabled.
[2020-03-12 11:31:26.438][Trace][981][668] write log to console
[2020-03-12 11:31:26.438][Trace][981][668] reload http_api success, nothing changed.
[2020-03-12 11:31:26.438][Trace][981][668] reload http stream success, nothing changed.
[2020-03-12 11:31:26.438][Trace][981][668] vhost __defaultVhost__ maybe modified, reload its detail.
[2020-03-12 11:31:26.438][Trace][981][668] ingest nothing changed for vhost=__defaultVhost__
[2020-03-12 11:31:26.438][Trace][981][668] reload config success.
TRANS_BY_GPT3
K8S will have a ..data subdirectory:
[root@srs-deploy-698ff4c4b9-jh7rc conf]# ls -al
total 16
drwxrwxrwx 3 root root 4096 Mar 12 11:49 .
drwxr-xr-x 1 root root 4096 Mar 12 11:42 ..
drwxr-xr-x 2 root root 4096 Mar 12 11:49 ..2020_03_12_11_49_56.260663928
lrwxrwxrwx 1 root root 31 Mar 12 11:49 ..data -> ..2020_03_12_11_49_56.260663928
lrwxrwxrwx 1 root root 15 Mar 12 11:48 srs.conf -> ..data/srs.conf
All events of K8S are as follows:
// Startup phase
[2020-03-12 14:13:54.482][Trace][1][652] auto reload watching fd=9, watch=1, file=conf
[2020-03-12 14:13:54.482][Trace][1][652] auto reload watching fd=9, watch=2, file=conf/..data
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=2, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:54.990][Trace][1][656] inotify event wd=2, mask=0x40000010, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_13_51.188104428, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x40000010, len=0, name=, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:13:57.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x20, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x1, len=16, name=srs.conf, reload=0
[2020-03-12 14:14:00.990][Trace][1][656] inotify event wd=2, mask=0x10, len=16, name=srs.conf, reload=0
// Modify ConfigMap
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000100, len=32, name=..2020_03_12_14_19_03.505526463, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000004, len=32, name=..2020_03_12_14_19_03.505526463, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x100, len=16, name=..data_tmp, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40, len=16, name=..data_tmp, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x80, len=16, name=..data, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=0, name=, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000020, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000200, len=32, name=..2020_03_12_14_17_54.925365984, reload=0
[2020-03-12 14:19:03.551][Trace][1][656] inotify event wd=1, mask=0x40000010, len=0, name=13_51.188104428, reload=0
Note: The configuration was updated at
18:23and applied to the Pod at19:03, taking a total of about 40 seconds.
The key events are as follows, it appears that K8S executed a command similar to ln -sf ..2020_03_12_14_17_54.925365984 ..data_tmp && mv ..data_tmp ..data:
- 0x100 ..data_tmp IN_CREATE - File creation.
- 0x40 ..data_tmp IN_MOVED_FROM - File movement.
- 0x80 ..data IN_MOVED_TO - File movement.
inotify should listen to the conf directory.
- The creation and modification of the
confdirectory,IN_MODIFY | IN_CREATE | IN_MOVED_TO, so that the creation of the..datasubdirectory can also be detected.
After receiving the event, it is necessary to determine the file name. Only if it is srs.conf or ..data, it is considered a reload, and others are ignored.
TRANS_BY_GPT3