SRS crash when forward and http hooks
snavruzov opened this issue · comments
Hi, the SRS is crashed for some reason that I couldn't research further but getting crash log(attached below).
It's happened yesterday, though it was working more than 2 months perfectly.
SRS version:XCORE-SRS/2.0.221(ZhouGuowen)
OS type:Ubuntu 14.04. deployed on AWS EC2 t2.medium instance\
SRS config file:
# main config for srs.
# @see full.conf for detail config.
listen 1935;
max_connections 100;
srs_log_tank file;
srs_log_file ./objs/srs.log;
http_api {
enabled on;
listen 1985;
}
http_server {
enabled on;
listen 8080;
dir ./objs/nginx/html;
}
stats {
network 0;
disk sda sdb xvda xvdb;
}
vhost __defaultVhost__ {
transcode liveff {
enabled on;
ffmpeg ./objs/ffmpeg/bin/ffmpeg;
engine ff {
# whether the engine is enabled
# default: off.
enabled on;
iformat flv;
vcodec libx264;
vfps 20;
vthreads 2;
vprofile baseline;
vpreset ultrafast;
# other x264 or ffmpeg video params
vparams {
# # ffmpeg options, @see: http://ffmpeg.org/ffmpeg.html
# #t 100;
level 30;
crf 28;
# # 264 params, @see: http://ffmpeg.org/ffmpeg-codecs.html#libx264
coder 1;
b_strategy 2;
bf 0;
refs 6;
g 40;
}
acodec copy;
oformat flv;
output rtmp://127.0.0.1:[port]/livelow/[stream];
}
}
gop_cache on;
queue_length 5;
}
Last two connection's log:
[2017-01-17 10:56:46.058][trace][7651][17691] RTMP client ip=x.x.x.x
[2017-01-17 10:56:46.109][trace][7651][17691] complex handshake success
[2017-01-17 10:56:46.153][trace][7651][17691] connect app, tcUrl=rtmp://x.x.x.x:1935/liveff, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:56:46.153][trace][7651][17691] out chunk size to 60000
[2017-01-17 10:56:46.358][trace][7651][17691] input chunk size to 60000
[2017-01-17 10:56:46.358][trace][7651][17691] client identified, type=fmle-publish, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:56:46.358][trace][7651][17691] source url=/liveff/rtmpLibs, ip=x.x.x.x, cache=1, is_edge=0, source_id=-1[-1]
[2017-01-17 10:56:46.568][trace][7651][17691] encoder thread cid=17692, current_cid=17691
[2017-01-17 10:56:46.568][trace][7651][17692] start ffmpeg, log: ./objs/ffmpeg-encoder-__defaultVhost__-liveff-rtmpLibs.log, params: ./objs/ffmpeg/bin/ffmpeg -f flv -i rtmp://127.0.0.1:1935/liveff?vhost=__defaultVhost__/rtmpLibs -vcodec libx264 -r 20.00 -threads 2 -profile:v baseline -preset ultrafast -level 30 -crf 28 -coder 1 -b_strategy 2 -bf 0 -refs 6 -g 40 -acodec copy -f flv -y rtmp://127.0.0.1:1935/livelow/rtmpLibs
[2017-01-17 10:56:46.569][trace][7651][17692] vfored ffmpeg encoder engine, pid=1488
[2017-01-17 10:56:46.569][trace][7651][17692] -> ENC time=54000, encoders=1, input=__defaultVhost__/liveff/rtmpLibs
[2017-01-17 10:56:46.578][trace][7651][17691] start publish mr=0/350, p1stpt=20000, pnt=20000, tcp_nodelay=0, rtcid=17693
[2017-01-17 10:56:46.596][trace][7651][17695] RTMP client ip=127.0.0.1
[2017-01-17 10:56:46.597][trace][7651][17695] complex handshake success
[2017-01-17 10:56:46.637][trace][7651][17695] connect app, tcUrl=rtmp://127.0.0.1:1935/liveff?vhost=__defaultVhost__, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:56:46.637][trace][7651][17695] out chunk size to 60000
[2017-01-17 10:56:46.717][trace][7651][17695] ignore AMF0/AMF3 command message.
[2017-01-17 10:56:46.757][trace][7651][17695] client identified, type=Play, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:56:46.757][trace][7651][17695] source url=/liveff/rtmpLibs, ip=127.0.0.1, cache=1, is_edge=0, source_id=17691[17691]
[2017-01-17 10:56:46.757][trace][7651][17695] dispatch cached gop success. count=0, duration=-1
[2017-01-17 10:56:46.757][trace][7651][17695] create consumer, queue_size=5.00, jitter=1
[2017-01-17 10:56:46.768][trace][7651][17695] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 2626560=>218750, realtime=0
[2017-01-17 10:56:46.768][trace][7651][17695] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-17 10:56:46.768][trace][7651][17696] ignored. set buffer length to 3000
[2017-01-17 10:56:47.234][trace][7651][17692] send SIGTERM to pid=1488
[2017-01-17 10:56:48.246][trace][7651][17692] SIGKILL stop process pid=1488 ok.
[2017-01-17 10:56:48.246][trace][7651][17691] cleanup when unpublish
[2017-01-17 10:56:48.247][warn][7651][17691][4] client disconnect peer. ret=1004
[2017-01-17 10:56:48.247][warn][7651][17695][104] client disconnect peer. ret=1004
[2017-01-17 10:57:18.480][trace][7651][17698] RTMP client ip=**x.x.x.x**
[2017-01-17 10:57:18.533][trace][7651][17698] complex handshake success
[2017-01-17 10:57:18.580][trace][7651][17698] connect app, tcUrl=rtmp://x.x.x.x:1935/liveff, pageUrl=, swfUrl=, schema=rtmp, vhost=__defaultVhost__, port=1935, app=liveff, args=null
[2017-01-17 10:57:18.580][trace][7651][17698] out chunk size to 60000
[2017-01-17 10:57:18.782][trace][7651][17698] input chunk size to 60000
[2017-01-17 10:57:18.783][trace][7651][17698] client identified, type=fmle-publish, stream_name=rtmpLibs, duration=-1.00
[2017-01-17 10:57:18.783][trace][7651][17698] source url=/liveff/rtmpLibs, ip=x.x.x.x, cache=1, is_edge=0, source_id=-1[-1]
[2017-01-17 10:57:18.977][trace][7651][100] cleanup die source, total=2
I can't get the coredump and call stack information. Maybe it's caused by source cleanup. So please show me the callstack or upgrade to latest 2.0.
Another crash log passing a couple of hours after upgrading to XCORE-SRS/2.0.231(ZhouGuowen)\
Last connection logs from srs.log:
[2017-01-19 14:03:31.405][trace][18219][106] RTMP client ip=x.x.x.x
[2017-01-19 14:03:31.405][trace][18219][106] srand initialized the random.
[2017-01-19 14:03:31.704][trace][18219][106] complex handshake success
[2017-01-19 14:03:32.009][trace][18219][106] connect app, tcUrl=rtmp://x.x.x.x:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=stream, args=null
[2017-01-19 14:03:32.828][trace][18219][106] client identified, type=fmle-publish, stream_name=testLibs, duration=-1.00
[2017-01-19 14:03:32.849][trace][18219][106] source url=x.x.x.x/stream/testLibs, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:03:33.760][trace][18219][106] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=109
[2017-01-19 14:03:34.389][trace][18219][106] got metadata, width=1280, height=960, vcodec=7, acodec=10
[2017-01-19 14:03:34.389][trace][18219][106] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:03:34.389][trace][18219][106] 33B video sh, codec(7, profile=Baseline, level=3.2, 1280x960, 0kbps, 0fps, 0s)
[2017-01-19 14:03:34.389][trace][18219][106] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:03:38.878][trace][18219][106] cleanup when unpublish
[2017-01-19 14:03:38.878][warn][18219][106][104] client disconnect peer. ret=1004
[2017-01-19 14:03:40.111][trace][18219][111] RTMP client ip=x.x.x.x
[2017-01-19 14:03:40.414][trace][18219][111] complex handshake success
[2017-01-19 14:03:40.717][trace][18219][111] connect app, tcUrl=rtmp://x.x.x.x:1935/dgtz, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=dgtz, args=null
[2017-01-19 14:03:41.532][trace][18219][111] client identified, type=fmle-publish, stream_name=live_id117_55279, duration=-1.00
[2017-01-19 14:03:41.552][trace][18219][111] source url=x.x.x.x/dgtz/live_id117_55279, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:03:42.477][trace][18219][115] api get peer ip success. ip=127.0.0.1
[2017-01-19 14:03:42.477][trace][18219][115] HTTP API GET http://localhost:1985/api/v1/clients/111, content-length=-1, chunked=0/0
[2017-01-19 14:03:42.479][trace][18219][115] HTTP API GET http://localhost:1985/api/v1/streams/18221, content-length=-1, chunked=0/0
[2017-01-19 14:03:42.557][trace][18219][111] ignore private address: x.x.x.x
[2017-01-19 14:03:42.557][warn][18219][111][11] use private address as ip: x.x.x.x
[2017-01-19 14:03:42.557][trace][18219][111] http hook on_publish success. client_id=111, url=http://127.0.0.1:8443/live/start, request={"action":"on_publish","client_id":111,"ip":"x.x.x.x","vhost":"x.x.x.x","app":"dgtz","tcUrl":"rtmp://x.x.x.x:1935/dgtz","stream":"live_id117_55279"}, response=0, ret=0
[2017-01-19 14:03:42.557][trace][18219][111] start forward rtmp://x.x.x.x:1935?vhost=x.x.x.x to rtmp://127.0.0.1:1940?vhost=x.x.x.x, tcUrl=rtmp://x.x.x.x:1940/dgtz, stream=live_id117_55279
[2017-01-19 14:03:42.567][trace][18219][111] forward thread cid=116, current_cid=111
[2017-01-19 14:03:42.568][trace][18219][116] forward connected, stream=live_id117_55279, tcUrl=rtmp://x.x.x.x:1935/dgtz to server=127.0.0.1, port=1940
[2017-01-19 14:03:42.568][trace][18219][116] complex handshake success.
[2017-01-19 14:03:42.577][trace][18219][111] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=117
[2017-01-19 14:03:42.647][trace][18219][116] connected, version=, ip=, pid=0, id=0, dsu=1
[2017-01-19 14:03:43.527][trace][18219][111] got metadata, width=640, height=360, vcodec=7, acodec=10
[2017-01-19 14:03:43.527][trace][18219][111] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:03:43.527][trace][18219][111] 34B video sh, codec(7, profile=Baseline, level=3, 640x368, 0kbps, 0fps, 0s)
[2017-01-19 14:03:43.527][trace][18219][111] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:03:43.687][trace][18219][116] protocol in.buffer=0, in.ack=5000000, out.ack=2500000, in.chunk=4096, out.chunk=60000
[2017-01-19 14:03:53.188][trace][18219][116] -> FWR time=10000, msgs=32, okbps=397,0,0, ikbps=2,0,0
[2017-01-19 14:04:03.188][trace][18219][116] -> FWR time=20000, msgs=35, okbps=476,0,0, ikbps=1,0,0
[2017-01-19 14:04:07.577][trace][18219][111] <- CPB time=21000, okbps=1,0,0, ikbps=458,0,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:13.179][warn][18219][115][104] client disconnect peer. ret=1004
[2017-01-19 14:04:13.188][trace][18219][116] -> FWR time=30000, msgs=26, okbps=521,521,0, ikbps=0,0,0
[2017-01-19 14:04:17.577][trace][18219][111] <- CPB time=31000, okbps=0,1,0, ikbps=520,455,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:23.189][trace][18219][116] -> FWR time=40000, msgs=34, okbps=577,521,0, ikbps=0,0,0
[2017-01-19 14:04:27.578][trace][18219][111] <- CPB time=41000, okbps=0,1,0, ikbps=564,455,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:28.445][trace][18219][119] RTMP client ip=x.x.x.x
[2017-01-19 14:04:28.661][trace][18219][119] complex handshake success
[2017-01-19 14:04:28.863][trace][18219][119] connect app, tcUrl=rtmp://x.x.x.x:1935/stream, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=stream, args=null
[2017-01-19 14:04:29.549][trace][18219][119] client identified, type=fmle-publish, stream_name=testLibs, duration=-1.00
[2017-01-19 14:04:29.549][trace][18219][119] source url=x.x.x.x/stream/testLibs, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:04:30.290][trace][18219][119] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=120
[2017-01-19 14:04:30.794][trace][18219][119] got metadata, width=1280, height=960, vcodec=7, acodec=10
[2017-01-19 14:04:30.794][trace][18219][119] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:04:30.794][trace][18219][119] 33B video sh, codec(7, profile=Baseline, level=3.2, 1280x960, 0kbps, 0fps, 0s)
[2017-01-19 14:04:30.794][trace][18219][119] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:04:32.937][trace][18219][119] cleanup when unpublish
[2017-01-19 14:04:32.937][warn][18219][119][104] client disconnect peer. ret=1004
[2017-01-19 14:04:33.189][trace][18219][116] -> FWR time=50000, msgs=32, okbps=613,521,0, ikbps=0,0,0
[2017-01-19 14:04:33.625][trace][18219][122] RTMP client ip=x.x.x.x
[2017-01-19 14:04:33.778][trace][18219][124] RTMP client ip=x.x.x.x
[2017-01-19 14:04:33.837][trace][18219][122] complex handshake success
[2017-01-19 14:04:33.983][trace][18219][124] complex handshake success
[2017-01-19 14:04:34.041][trace][18219][122] connect app, tcUrl=rtmp://x.x.x.x:1935/dgtz, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=dgtz, args=null
[2017-01-19 14:04:34.273][trace][18219][124] connect app, tcUrl=rtmp://x.x.x.x:1935/debate, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=debate, args=null
[2017-01-19 14:04:34.782][trace][18219][122] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:35.021][trace][18219][124] client identified, type=fmle-publish, stream_name=live_id153_55280, duration=-1.00
[2017-01-19 14:04:35.041][trace][18219][124] source url=x.x.x.x/debate/live_id153_55280, ip=x.x.x.x, cache=0, is_edge=0, source_id=-1[-1]
[2017-01-19 14:04:35.300][trace][18219][122] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:35.300][trace][18219][122] client identified, type=Play, stream_name=live_id117_55279, duration=-1.00
[2017-01-19 14:04:35.300][trace][18219][122] source url=x.x.x.x/dgtz/live_id117_55279, ip=x.x.x.x, cache=0, is_edge=0, source_id=111[111]
[2017-01-19 14:04:35.300][trace][18219][122] dispatch cached gop success. count=0, duration=0
[2017-01-19 14:04:35.300][trace][18219][122] create consumer, queue_size=30.00, jitter=1
[2017-01-19 14:04:35.310][trace][18219][122] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 46080=>218750, realtime=0
[2017-01-19 14:04:35.310][trace][18219][122] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-19 14:04:35.310][trace][18219][122] protocol in.buffer=3000, in.ack=2500000, out.ack=2500000, in.chunk=128, out.chunk=60000
[2017-01-19 14:04:35.877][trace][18219][124] http hook on_publish success. client_id=124, url=http://127.0.0.1:8443/live/debate/start, request={"action":"on_publish","client_id":124,"ip":"x.x.x.x","vhost":"x.x.x.x","app":"debate","tcUrl":"rtmp://x.x.x.x:1935/debate","stream":"live_id153_55280"}, response=0, ret=0
[2017-01-19 14:04:35.877][trace][18219][124] start forward rtmp://x.x.x.x:1935?vhost=x.x.x.x to rtmp://127.0.0.1:1940?vhost=x.x.x.x, tcUrl=rtmp://x.x.x.x:1940/debate, stream=live_id153_55280
[2017-01-19 14:04:35.888][trace][18219][124] forward thread cid=128, current_cid=124
[2017-01-19 14:04:35.888][trace][18219][128] forward connected, stream=live_id153_55280, tcUrl=rtmp://x.x.x.x:1935/debate to server=127.0.0.1, port=1940
[2017-01-19 14:04:35.888][trace][18219][128] complex handshake success.
[2017-01-19 14:04:35.898][trace][18219][124] start publish mr=0/350, p1stpt=20000, pnt=5000, tcp_nodelay=0, rtcid=129
[2017-01-19 14:04:35.967][trace][18219][128] connected, version=, ip=, pid=0, id=0, dsu=1
[2017-01-19 14:04:36.708][trace][18219][124] got metadata, width=640, height=360, vcodec=7, acodec=10
[2017-01-19 14:04:36.708][trace][18219][124] protocol in.buffer=0, in.ack=0, out.ack=2500000, in.chunk=60000, out.chunk=60000
[2017-01-19 14:04:36.708][trace][18219][124] 34B video sh, codec(7, profile=Baseline, level=3, 640x368, 0kbps, 0fps, 0s)
[2017-01-19 14:04:36.708][trace][18219][124] 4B audio sh, codec(10, profile=LC, 1channels, 0kbps, 44100HZ), flv(16bits, 2channels, 44100HZ)
[2017-01-19 14:04:37.007][trace][18219][128] protocol in.buffer=0, in.ack=5000000, out.ack=2500000, in.chunk=4096, out.chunk=60000
[2017-01-19 14:04:37.199][trace][18219][131] RTMP client ip=x.x.x.x
[2017-01-19 14:04:37.439][trace][18219][131] complex handshake success
[2017-01-19 14:04:37.642][trace][18219][131] connect app, tcUrl=rtmp://x.x.x.x:1935/debate, pageUrl=, swfUrl=, schema=rtmp, vhost=x.x.x.x, port=1935, app=debate, args=null
[2017-01-19 14:04:38.324][trace][18219][131] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:38.772][trace][18219][131] ignore AMF0/AMF3 command message.
[2017-01-19 14:04:38.772][trace][18219][131] client identified, type=Play, stream_name=live_id153_55280, duration=-1.00
[2017-01-19 14:04:38.772][trace][18219][131] source url=x.x.x.x/debate/live_id153_55280, ip=x.x.x.x, cache=0, is_edge=0, source_id=124[124]
[2017-01-19 14:04:38.772][trace][18219][131] dispatch cached gop success. count=0, duration=0
[2017-01-19 14:04:38.772][trace][18219][131] create consumer, queue_size=30.00, jitter=1
[2017-01-19 14:04:38.782][trace][18219][131] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 46080=>218750, realtime=0
[2017-01-19 14:04:38.782][trace][18219][131] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0
[2017-01-19 14:04:38.782][trace][18219][131] protocol in.buffer=3000, in.ack=2500000, out.ack=2500000, in.chunk=128, out.chunk=60000
[2017-01-19 14:04:45.008][trace][18219][128] -> FWR time=8000, msgs=28, okbps=1223,0,0, ikbps=3,0,0
[2017-01-19 14:04:46.994][trace][18219][131] -> PLA time=8000, msgs=22, okbps=1113,0,0, ikbps=2,0,0, mw=350
[2017-01-19 14:04:47.578][trace][18219][111] <- CPB time=61000, okbps=0,0,0, ikbps=646,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:55.008][trace][18219][128] -> FWR time=18000, msgs=28, okbps=1312,0,0, ikbps=1,0,0
[2017-01-19 14:04:57.578][trace][18219][111] <- CPB time=71000, okbps=0,0,0, ikbps=684,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:04:57.719][trace][18219][122] -> PLA time=23000, msgs=63, okbps=802,0,0, ikbps=1,0,0, mw=350
[2017-01-19 14:05:04.690][trace][18219][116] -> FWR time=82000, msgs=0, okbps=734,860,0, ikbps=0,0,0
[2017-01-19 14:05:07.578][trace][18219][111] <- CPB time=81000, okbps=0,0,0, ikbps=727,792,0, mr=0/350, p1stpt=20000, pnt=5000
[2017-01-19 14:05:07.921][trace][18219][131] -> PLA time=29000, msgs=0, okbps=1316,1300,0, ikbps=0,0,0, mw=350
[2017-01-19 14:05:10.826][error][18219][128][4] recv server control message failed. ret=1007(Interrupted system call)
I'm not sure whether I run correctly to trace the crash file, I did as shown below and got some results:
apport-retrace -g _opt_srs_trunk_objs_srs.0.crash
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /opt/srs/trunk/objs/srs...done.
[New LWP 18219]
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
Core was generated by `./objs/srs -c conf/srs.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f42ee07acc9 in __GI___libc_sigaction (sig=18219, act=0x472b, oact=0x6) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:53
53 ../sysdeps/unix/sysv/linux/x86_64/sigaction.c: No such file or directory.
I think I caught a consequence of the crash.
When I use http callback options: on_publish, the server is getting crashed.
In fact it's crashing, when I stop RTMP broadcasting, with messages:
pure virtual method called
terminate called without an active exception
Without http callback function the server is working well
The same crashes with the exact output's been noticed in the releases started from v2.0-b3.
In the v2.0-b2(XCORE-SRS/2.0.221(ZhouGuowen)) there is no crashes so far.
The crash signal SIGABRT is weird. I had never got this signal before.
Why there is no call stack of apport-retrace -g _opt_srs_trunk_objs_srs.0.crash? Like the command bt of gdb?
And can you show me the replay step? Only config the http hooks on_publish then will crash? I will try it.
我试了2.0.231加上了on_publish，没有crash。你能否给出配置，重现步骤，或者崩溃时的call stack？
There is call stack version srs-2.0-b4 I got:
root@ip-172-31-52-206:/var/crash# apport-retrace -g _opt_srs-2.0-b4_trunk_objs_srs.0.crash
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /opt/srs-2.0-b4/trunk/objs/srs...done.
[New LWP 22270]
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
Core was generated by `./objs/srs -c conf/srs.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f0f1f57cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f0f1f57cc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f0f1f580028 in __GI_abort () at abort.c:89
#2 0x00007f0f1fe87535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007f0f1fe856d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f0f1fe85703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007f0f1fe861bf in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x0000000000469f2c in SrsKbps::set_io (this=0x139fcc0, in=0x0, out=0x0) at src/protocol/srs_protocol_kbps.cpp:123
#7 0x000000000049c531 in SrsForwarder::on_unpublish (this=0x13a27b0) at src/app/srs_app_forward.cpp:155
#8 0x000000000048cfa8 in SrsSource::destroy_forwarders (this=0x139c360) at src/app/srs_app_source.cpp:2379
#9 0x000000000048c0ee in SrsSource::on_unpublish (this=0x139c360) at src/app/srs_app_source.cpp:2193
#10 0x000000000047f6ff in SrsRtmpConn::release_publish (this=0x139b6c0, source=0x139c360, is_edge=false) at src/app/srs_app_rtmp_conn.cpp:974
#11 0x000000000047e959 in SrsRtmpConn::publishing (this=0x139b6c0, source=0x139c360) at src/app/srs_app_rtmp_conn.cpp:846
#12 0x000000000047cdc3 in SrsRtmpConn::stream_service_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:537
#13 0x000000000047c1d1 in SrsRtmpConn::service_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:415
#14 0x000000000047b145 in SrsRtmpConn::do_cycle (this=0x139b6c0) at src/app/srs_app_rtmp_conn.cpp:210
#15 0x0000000000479467 in SrsConnection::cycle (this=0x139b748) at src/app/srs_app_conn.cpp:89
#16 0x00000000004aa51f in SrsOneCycleThread::cycle (this=0x139b370) at src/app/srs_app_thread.cpp:372
#17 0x00000000004a9cbb in internal::SrsThread::thread_cycle (this=0x1396be0) at src/app/srs_app_thread.cpp:207
#18 0x00000000004a9ec5 in internal::SrsThread::thread_fun (arg=0x1396be0) at src/app/srs_app_thread.cpp:245
#19 0x0000000000530f7b in _st_thread_main () at sched.c:327
#20 0x00000000005316eb in st_thread_create (start=0x535e4f <_st_epoll_pollset_add+746>, arg=0x2710, joinable=0, stk_size=5441783) at sched.c:591
#21 0x00007f0f2047ae30 in ?? ()
#22 0x00007f0f2052bbf0 in ?? ()
#23 0x0000000000530779 in st_poll (pds=<error reading variable: Cannot access memory at address 0x8ffffffa9>,
npds=<error reading variable: Cannot access memory at address 0x8ffffffa5>, timeout=<error reading variable: Cannot access memory at address 0x8ffffff99>) at sched.c:86
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
srs.conf config file where I call my http server:
vhost stream.*****.com {
http_hooks {
enabled on;
on_publish http://127.0.0.1:8443/live/start;
on_unpublish http://127.0.0.1:8443/live/stop;
}
forward 127.0.0.1:1940;
gop_cache on;
queue_length 10;
}
Yes, without http_hooks section the server acts perfectly without any crashes.
OK, following by a recent tests, I tried the same steps on Ubuntu 16.04, no crashes.
It seems the forwarder crash when unpublish, maybe caused by http hooks. I will finger it out.
void SrsForwarder::on_unpublish()
{
pthread->stop();
close_underlayer_socket();
srs_freep(client);
srs_freep(io); // The io is freed here.
kbps->set_io(NULL, NULL); // In this function, the previous io, that is the freed io up there, will be used and it sometimes cause the crash.
}
The io must be released by set_io(NULL) before free it:
kbps->set_io(NULL, NULL);
srs_freep(io);
This bug also exists in:
SrsEdgeIngester::stop()
SrsEdgeForwarder::stop()
SrsEdgeIngester::connect_server()
SrsForwarder::connect_server()
SrsForwarder::on_unpublish()
Fixed in 2.0.232