A watcher for this process already exists. when start a another proc(go server) using pmgo start

[binary111]

Objective:
I wants to spawn up another proc(server) before stoping the current running server.

Issue:
Starting a single proc(server) is working fine but when I start another proc(server), it get defunct after trying to restart the processor for some times. it may be because of race condition or memory leak. though it is working fine on my local server.

Expected:
Older proc should be in running state and second one should be continuously retrying to connect

Please find below the logs

ps -ef | grep app
root 817 29201 0 11:14 ? 00:00:06 app_alpha
root 9908 29201 0 11:16 ? 00:00:00 [app_beta]
root 11437 29201 0 10:01 ? 00:00:00 [app]
root 14878 29201 0 12:41 ? 00:00:00 [app_gamma]
root 29201 1 0 10:00 ? 00:00:06 pmgo start src/app/ app

cat ~/.pmgo/main.log
time="2018-05-09T12:41:50Z" level=info msg="Starting watcher on proc app_gamma"
time="2018-05-09T12:41:50Z" level=info msg="Proc app_gamma is dead, advising master..."
time="2018-05-09T12:41:50Z" level=info msg="Restarting proc app_gamma."
time="2018-05-09T12:41:50Z" level=info msg="Starting watcher on proc app_gamma"
time="2018-05-09T12:41:50Z" level=info msg="Proc app_gamma is dead, advising master..."
...
time="2018-05-09T12:41:52Z" level=info msg="Restarting proc app_gamma."
time="2018-05-09T12:41:52Z" level=info msg="Starting watcher on proc app_gamma"
time="2018-05-09T12:41:52Z" level=info msg="Proc app_gamma is dead, advising master..."
time="2018-05-09T12:41:52Z" level=info msg="State is exit status 1"
time="2018-05-09T12:41:52Z" level=info msg="Restarting proc app_gamma."
time="2018-05-09T12:41:52Z" level=warning msg="A watcher for this process already exists."

tail -f ~/.pmgo/app_gamma/app_gamma.err
2018/05/09 18:21:49 ws server started on 127.0.0.1:8010
2018/05/09 18:21:49 ListenAndServe: listen tcp 127.0.0.1:8010: bind: address already in use

[david zh]

@binary111
I know what you want. like Graceful restart ?
If i start new proc before stop current proc, this will cause some problems, for example, the same listening port etc.
look up this project: https://github.com/facebookgo/grace.

[binary111]

@struCoder thanks for the response
The issue is new proc get defunc after trying for some time and ideally watcher should keep trying to restart infinitely.
I have looked into using facebookgo/grace but It doesn't kill the old proc till all the active connection were not served. The thing is I am using websocket, which keeps connection alive so I have to kill that proc manually.

I think in AddProcWatcher, before restarting the proc it should delete the process from watcher list
//watcher.watchProcs[proc.Identifier()] = procWatcher
//delete(watcher.watchProcs, procWatcher.proc.Identifier())

// AddProcWatcher will add a watcher on proc.
func (watcher *Watcher) AddProcWatcher(proc process.ProcContainer) {
watcher.Lock()
defer watcher.Unlock()
if _, ok := watcher.watchProcs[proc.Identifier()]; ok {
log.Warnf("A watcher for this process already exists.")
return
}
procWatcher := &ProcWatcher{
procStatus: make(chan *ProcStatus, 1),
proc: proc,
stopWatcher: make(chan bool, 1),
}
watcher.watchProcs[proc.Identifier()] = procWatcher
go func() {
log.Infof("Starting watcher on proc %s", proc.Identifier())
state, err := proc.Watch()
procWatcher.procStatus <- &ProcStatus{
state: state,
err: err,
}
}()
go func() {
defer delete(watcher.watchProcs, procWatcher.proc.Identifier())
select {
case procStatus := <-procWatcher.procStatus:
log.Infof("Proc %s is dead, advising master...", procWatcher.proc.Identifier())
log.Infof("State is %s", procStatus.state.String())
watcher.restartProc <- procWatcher.proc
break
case <-procWatcher.stopWatcher:
break
}
}()
}