Ices0 crashes when specifying folders instead of files in playlists
AlphaJack opened this issue · comments
First of all, thank you for having updated Ices0 with a ton of useful features.
I installed it on Arch ARM from the AUR: I modified the PKGBUILD to point the source to this repo and by changing its checksums.
On my small server I've got 5 streams of 320kbps mp3 (original bitrate of the files, no re-encoding when streaming) pointing to my icecast2 server. The CPU usage is almost non-existent, it doesn't even reach 4% with evertything running.
Unfortunately I see that periodically one or two of the streams crashes. Some survives indefinitely, the configuration files are almost the same, except for the playlist files, the mountpoints and the descriptions. I get notified by systemd with the following (the stream has been online for a few minutes in this case, but the last time I checked it went offline just 2 seconds after being launched):
Sep 11 20:59:34 Jack-P2 audit[26445]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=26445 comm="ices0" exe="/usr/bin/ices0" sig=11 res=1
Sep 11 20:59:34 Jack-P2 kernel: audit: type=1701 audit(1568228374.855:429): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=26445 comm="ices0" exe="/usr/bin/ices0" sig=11 res=1
Sep 11 20:59:35 Jack-P2 systemd[1]: Started Process Core Dump (PID 29051/UID 0).
-- Subject: A start job for unit systemd-coredump@5-29051-0.service has finished successfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit systemd-coredump@5-29051-0.service has finished successfully.
--
-- The job identifier is 11833.
Sep 11 20:59:35 Jack-P2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@5-29051-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 11 20:59:35 Jack-P2 kernel: audit: type=1130 audit(1568228375.215:430): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@5-29051-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 11 20:59:35 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.ed399ee4cf084132a928a4e059cd749f.258.1567975660000000.lz4.
Sep 11 20:59:35 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.ed399ee4cf084132a928a4e059cd749f.11924.1567992468000000.lz4.
Sep 11 20:59:35 Jack-P2 systemd-coredump[29055]: Removed old coredump core.systemd-journal.0.ed399ee4cf084132a928a4e059cd749f.162.1568013171000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.ed399ee4cf084132a928a4e059cd749f.16297.1568017802000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.6f1890eab56b4e32a0cfa31166b2b923.235.1568071953000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.6f1890eab56b4e32a0cfa31166b2b923.14796.1568133915000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.6f1890eab56b4e32a0cfa31166b2b923.17222.1568142003000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.6f1890eab56b4e32a0cfa31166b2b923.17772.1568146503000000000000.lz4.
Sep 11 20:59:36 Jack-P2 systemd-coredump[29055]: Removed old coredump core.ices0.0.6f1890eab56b4e32a0cfa31166b2b923.19952.1568165405000000000000.lz4.
Sep 11 20:59:43 Jack-P2 systemd-coredump[29055]: Process 26445 (ices0) of user 0 dumped core.
Stack trace of thread 26445:
#0 0x000000007688f938 memcpy (libc.so.6)
#1 0x0000000000475204 n/a (ices0)
-- Subject: Process 26445 (ices0) dumped core
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: man:core(5)
--
-- Process 26445 (ices0) crashed and dumped core.
--
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.
Sep 11 20:59:43 Jack-P2 systemd[1]: systemd-coredump@5-29051-0.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit systemd-coredump@5-29051-0.service has successfully entered the 'dead' state.
Sep 11 20:59:43 Jack-P2 kernel: audit: type=1131 audit(1568228383.745:431): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@5-29051-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 11 20:59:43 Jack-P2 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@5-29051-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
What other logs can I provide to find the culprit?
I've also noticed a few other bugs:
- Ices0 sometimes plays songs slower than the normal playback speed. I solve this issue by refreshing the page or the stream from the listening client
- Ices0 often skips to the next track before the song ends. I have checked the files and I can't understand why it doesn't play the whole song until the end
We have multiple things to check here, I guess.
I never tried it on ARM, only on x86_64, and have been firing endless playlists of mixed FLAC, MP3 and Ogg Vorbis files at it for months now without a single fault (256 kbit/s MP3 stream on a private Icecast 2.4.4). Mine of course does re-encoding all the time and also uses crossfading.
Let us try to get at it systematically, more or less:
-
What OS on what platform do you use, exactly?
-
Can you reproduce it on a x86_64 system like Ubuntu, Linux Mint or Manjaro?
-
Could you post (or mail me) your ices.conf? (Remember to edit out the passwords and maybe stream locations.)
-
Please try the same with debugging enabled (
<Verbose>1</Verbose>in ices.conf’s<Execution>…</Execution>section) and attach theices.logfile. -
Try enabling re-encoding in the
<Stream>…</Stream>section(s) using<Reencode>1</Reencode>to see if that changes the "too slow/too fast" playback. It doesn’t actually add that much overhead. -
Can you find any correlation between the playlists/the files played and the crashes? ices might crash on broken files, and you wouldn’t believe how many MP3s are broken, really. You might want to use "MP3 Diags" to check for missing frames, obscure tags, or broken headers.
You might also want to set
<Randomize>0</Randomize>so we can better check if it always happens with the same audio file(s). -
Do you use the "script", "perl", or "python" interface or just "builtin" with playlists?
-
If using playlists:
- Are those "native", i.e. just lines with file paths in them, or M3U or M3U8 playlists? (PLS are not supported.)
- If M3U/M3U8: Using the extended or even Squeezebox/Logitech Media Server format?
-
Are the audio files on a "slow" medium, i.e. network NAS or USB2 external drive or maybe an external NTFS-formatted disc? If so, try to have them on a fast local disc and see if this betters anything.
-
Is your Icecast local or remote?
-
Are you using at least version 2.4.1 or better 2.4.4 and have you set
<charset>UTF8</charset>as a mount param in youricecast.xmlfile for the MP3 mount? -
You might also want to "watch" the ices.log while the program is running, use something like
tail -f ices.login a terminal. -
If all else fails, try to start with one stream only and see if the problem occurs, then use two and so forth.
-
If we can’t get at it fast, we might need the compiler/linker output (simply redirect to a file). You’d need to
make cleanandmake distclean, then compile again manually for this. -
Re "skipping to next track": Could you send me some of the files in question, so I can try here or diagnose them? You could use some cloud service and mail the download link to moonbase@quantentunnel.de.
I hope we can narrow this down a little to find the actual problem. Personally, I’ve only ever had problems like skips or slow/fast streaming with damaged audio files (all MP3, never happened with FLAC/Ogg) or flakey connections (but even then, ices usually just gave up after 10 tries or so).
I converted in bulk my files to mp3@320kbps, but rechecking them now I've noticed that something went wrong. Some of them are fine, others are cropped to weight 8.0 or 4.0 MB, others are just empty files. This would explain the "skipping to next track" problem. The culprit of the segmentation fault instead seems to be related to the presence of some folders in the playlists.
Answering in order:
- Raspberry Pi 2, Arch ARM 4.19.71-1
- I could, but it doesn't seems necessary anymore
- e-mail sent
- I'm attaching the relevant parts, I've created a playlist of three elements:
chopped file.mp3
empty file.mp3
non existent file
folder/
This is the log that appears
# chopped files end normally:
[...]
DEBUG: Updated metadata on /test to: Dast - Let's You Go
DEBUG: Done sending
# empty files get skipped:
DEBUG: Builtin playlist handler serving: /usr/share/music/Techno/Slam - Vapour.mp3
DEBUG: Filename cleaned up from [/usr/share/music/Techno/Slam - Vapour.mp3] to [Slam - Vapour]
Error opening /usr/share/music/Techno/Slam - Vapour.mp3: Error reading header: /usr/share/music/Techno/Slam - Vapour.mp3
# non existent files get skipped:
DEBUG: Builtin playlist handler serving: /usr/share/music/Techno/Dast - Le
DEBUG: Filename cleaned up from [/usr/share/music/Techno/Dast - Le] to [Dast - Le]
Error opening /usr/share/music/Techno/Dast - Le: Error opening: No such file or directory
# folders cause a segmentation fault
DEBUG: Builtin playlist handler serving: /usr/share/music/
DEBUG: Filename cleaned up from [/usr/share/music/] to []
[1] 17403 segmentation fault ices0 -c /usr/share/music/_icecast/test.xml
If I create a playlist of only chopped files the program runs as expected.
If I create a playlist of only empty or non existent files the program exits after 10 retries.
If I create a playlist of only folders the program crashes at the first try.
- The slow playback speed problem seems to appear rarely, so I can't confirm any improvement with reecoding, although I believe it does solve this issue
- This, I suppose, was the culprit
- Builtin
- Playlist are just text files containg the absolute path of the files
- Files are at the moment on the same SD card of the OS
- Local
- icecast 2.4.3-3
- Done
- Done, multiple streams are ok
- I could, but it doesn't seems necessary anymore
- e-mail sent
Having said that, I would like to ask you if implementing these feature is feasible:
A. In a playlist, check if the line contains a file or a folder. If it's a folder, skip it
B. Option to select a folder and play recursively the files it contains. Is randomization possible in this case?
C. Do not print password in log. When launching the programs, the first lines say:
Logfile opened
DEBUG: Sending following information to libshout:
DEBUG: Stream: 0
DEBUG: Host: localhost:8000 (protocol: http)
DEBUG: Mount: /test, Password: XXXXXXXXX
D. AIFF reencoding support. Unlike FLAC those files are not compressed so I suppose the reecoding process require less power. Unlike WAV files, ID3v2 is largely used on AIFF ones.
E. The possibility to stream more tags (album artist, year, album artwork...), if Icecast doesn't prevent it
F. I use a script to caputure the PID when launching the files as daemons, is it necessary or Ices0 already creates its PIDfile?
Thanks for helping and debugging!
If I understand correctly, some of your playlist entries are just folders, not actual music files?
This shouldn’t happen (at least it’s illegal for M3U/M3U8 playlists), but of course ices shouldn’t segfault in this case—I’ll look into that.
I don’t know how you create the playlists, if you’re using find you might want to add the -type f argument, i.e. find . -type f -iname "*.mp3".
Responding to your questions:
A. I’ll look into that, simply skipping seems feasible.
B. Thats’s what playlists are for and can be handled much more elegantly by some external code, especially since playlists can be modified while ices is running (it will re-read the playlist in this case). We also have great scripting bindings (shell, Python, Perl) for letting ices know what to play.
Introducing folder traversing within ices is currently not feasible, since
- it breaks compatibility for other software that uses ices
- it would require lots of code changes, i.e. the traversing logic, looking for "legal" filetypes, checking accessibility/readability of folders and files and much more.
- it might introduce unwanted effects, like ices deciding to play 145,000 files before the next entry in the playlist comes up, just because your playlist entry accidentally pointed to the base of my music folder …
C. This is a good point for discussion. I have very long and obscure passwords for my online (remote) Icecast and Shoucast servers, so I was always happy to be able to check these in the logs. It’s only shown when enabling the verbose log (debug) which shouldn’t be used in a production environment anyway (since it’s very verbose and quickly fills up disc space).
I can see your point, though, from a security standpoint. Let me think about it.
D. I’ll probably not have the resources to implement support for more file types into ices. There are so many things to be considered, I just found out when doing this for loudgain. It could probably be done using a more "unified" file reader (like maybe libsndfile or the FFmpeg libraries) but this would require a major rewrite of ices.
E. I know that for instance Shoutcast 2 servers allow some more tag fields to be streamed, but the majority of broadcasting stations still use just one text string, containing only artist and title. This would also mean some kind of elaborate templating logic (i.e. how the user would wish the streaming title to built). It’s somehow a nice idea for an enhancement, but I won’t have the resources in the near future.
F. Ices will create its log, cue and pid file in the folder specified in <BaseDirectory>…</BaseDirectory> (within the <Execution>…</Execution> section of your ices.conf). Be sure to use different folders here in case you run more than one instance of ices.
P.S.: For "extra" questions like B to F, it’s better to create extra issues, so they won’t go under. :-)
Also, I somehow like that somebody actually uses ices on a RasPi, with Arch even! :-)
I took the liberty to rename the issue since this seems to be the main culprit here.
Try version 0.4.8. This should fix trying to play non-regular files like devices or directories and skip them with an error message instead of segfaulting. The fix is independent of the module used (builtin, script, python or perl).
DEBUG: Builtin playlist handler serving: ../Today/xxx/
DEBUG: Filename cleaned up from [../Today/xxx/] to []
Error opening ../Today/xxx/: ERROR: Is directory!
DEBUG: Builtin playlist handler serving: ../Today/xxx
DEBUG: Filename cleaned up from [../Today/xxx] to [xxx]
Error opening ../Today/xxx: ERROR: Is directory!
DEBUG: Builtin playlist handler serving: ../Today/dummy/
DEBUG: Filename cleaned up from [../Today/dummy/] to []
Error opening ../Today/dummy/: Error opening: No such file or directory
DEBUG: Builtin playlist handler serving: ../Today/dummy
DEBUG: Filename cleaned up from [../Today/dummy] to [dummy]
Error opening ../Today/dummy: Error opening: No such file or directory