Searchd memory usage
atomical opened this issue · comments
Hi Pat,
Is there any way to limit memory usage? Currently using 6 gigs with a big index. A cursory search reveals it's not possible?
Hi Adam - I don't think this is possible, as the Sphinx daemon loads all attribute data into memory. Hence: the more records (and more attributes) you have, the larger the memory.
I guess there's the option of splitting up the data into separate daemons/machines, but that's going to get complex quickly - using a single machine with more memory is almost certainly going to be the wiser path.
Yes, we picked single machine as our approach and that was a good idea. Ram is cheap -- except if it's cloud Ram. 😄
I'm actually wondering if running out of ram is the reason the searchd instance is crashing on every deploy. Then we have to remove the binlog and start the process again. We can do this without rebuilding so there isn't much downtime. But still it's not great.
[Tue Dec 22 12:18:48.912 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc_core, indextid=0, logtid=7, pos=81668)
[Tue Dec 22 12:18:48.912 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc2_core, indextid=0, logtid=20, pos=81935)
[Tue Dec 22 12:18:48.913 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc2_core, indextid=0, logtid=116, pos=127173)
[Tue Dec 22 12:18:48.914 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc3_module_core, indextid=0, logtid=67, pos=164857)
[Tue Dec 22 12:18:48.918 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc4_core, indextid=0, logtid=39, pos=329940)
[Tue Dec 22 12:18:48.918 2020] [14080] FATAL: binlog: update: descending tid (index=abc5_core, lasttid=116, logtid=1, pos=330114)
We're using Sphinx 2.3.2-id64-beta.
At a guess I don't think that's a RAM issue… I wonder if the daemon's being forcibly killed rather than shutting itself down properly (and thus the binlog files aren't quite right). Is there anything in the daemon log that suggests a rushed shutdown?
That's probably likely. Although I think it could be forcibly killed because of a ram issue? I haven't seen anything in the log but I'll see what I can dig up from the system.
Yeah, I've seen daemons killed before if the machine's RAM is being maxed out. Not quite sure why I didn't think of that in my last message - my previous assertion about it not being a RAM issue is quite possibly wrong! 🤦🏻♂️