total_pages incorrect when using Manticore as backend

[Sandro Kalbermatter]

Hi Pat

It appears that total_pages is incorrect with Manticore 5.0.2 348514c86@220530 dev as backend. I know it's Manticore because the problem did not occur with Sphinx.

Here's how to reproduce it:

results = Document.search # Document is a model with a search index
# Sphinx Query (35.5ms)  SELECT * FROM `document_core` WHERE `sphinx_deleted` = 0 LIMIT 0, 20
# Sphinx  Found 260 results 
# [Rails then loads 20 documents]
results.total_entries
=> 260
results.per_page
=> 20
results.total_pages
=> 1 # This is wrong (260/20 = 13)

Here's another example, this time not using a model class but ThinkingSphinx directly:

results = ::ThinkingSphinx.search
# [...]
# Sphinx  Found 536 results
results.total_entries
=> 536
results.per_page
=> 20
results.total_pages
=> 15 # This is wrong (536/20 = 26.8)

Maybe this is of interest to you. Luckily the total page number can easily be calculated from the other numbers, so there is an obvious workaround while the problem exists.

Best,
Kalsan

[Pat Allan]

Thanks for logging this issue @kalsan :) It has also been covered by #1213 - but then I reverted that change because it no longer worked with Sphinx, and I was hoping that it might get fixed in Manticore. That said, I never logged an issue with them!

I'll try to find some time to verify what should be the right behaviour for Manticore, and then we can decide whether the TS code needs changes or if a new release of Manticore will fix it.

Granted, it'd be nice if TS just worked properly with both engines… further PRs are definitely welcome if anyone has time before I get to it (this month is already looking pretty hectic).

[Sandro Kalbermatter]

Sounds great! Thank you for your support and looking forward to hearing from you :-)

[Xinyue Lu]

As a temp workaround I'm injecting total_pages into the mask class instance.

p.send(:mask_stack).first.instance_variable_set '@total_pages', (p.total_entries / p.per_page.to_f).ceil

[Sandro Kalbermatter]

Hi there! The problem is much worse than thought. Manticore 5.0.2 randomly forgets objects until ts:index is run again.

Here's an example:

Mymodel.count # 14
Mymodel.search.count # 3 (wrong!)

# Is it TS or Manticore's fault? Let's try to get closer to Manticore...
ThinkingSphinx::Connection.take{|c| c.execute 'show tables;'}.to_a
# [..., {"Index"=>"mymodel", "Type"=>"distributed"},
# {"Index"=>"mymodel_core", "Type"=>"rt"}, ]
ThinkingSphinx::Connection.take{|c| c.execute 'select id from mymodel;'}.to_a # returns 3 IDs, wrong!
ThinkingSphinx::Connection.take{|c| c.execute 'select id from mymodel_core;'}.to_a # returns 3 IDs, wrong!

This leads to the suspicion that Manticore 5.0.2 is broken on itself. Would you agree @pat ? I will likely downgrade Manticore and test with an older version. Which exact version number would you recommend me to try?

Note that this bug only occurs after several days in a fairly busy production environment holding hundreds of thousands of data records. Some indices are affected, others are not. Restarting Manticore does not help, re-indexing helps. The affected indices typically lack between 1 and 20 records, so the amount of lost or missing objects is not proportional to the size of the index (I'm missing 9 of 14 objects in the example above, another index is missing only 4 of over 100'000 objects.

Best,
Kalsan

[Pat Allan]

You could try the freshly released Manticore 5.0.3? But otherwise, maybe jump back to the latest 4.x release?

But also: are these real-time indices, or SQL-backed indices, and in either case, how are you tracking changes (e.g. do you have callbacks in place)? Are you doing anything custom here - in particular, anything that may prompt a deletion of data from Manticore while it remains in your database?

[Sandro Kalbermatter]

Hi Pat

Just wanted to let you know that I've been working on downgrading the version and pushed a 4.x manticore to the server in question recently. Now observing and crossing fingers.

Best,
Kalsan

[Sandro Kalbermatter]

Hi Pat

After running it for 2 weeks, the problem occured again with Manticore 4.2.0_211223.15e927b28-1. I don't think there is anything you can do as this is most likely an upstream bug. Would you like to close this issue?

Best,
Kalsan

[Pat Allan]

@kalsan it's frustrating to hear that the problem with Manticore continues - but yes, sounds like the problem is on their side. Given it's intermittent, it's a hard thing to create a reproducible bug report for their team, but may be good to report nonetheless?

I figured out the cause of the problem with total_pages - since v5.0.0, Manticore has changed the default behaviour of query stats to just consider the current page of results, not how many results there may be overall. I've changed the default behaviour in Thinking Sphinx to ensure total_pages is now set correctly (#1239), and released v5.5.1 of Thinking Sphinx with this fix. Which I guess is a bonus justification for closing this issue 😅 - sorry it took so long to get to the bottom of the cause!

But also, looking back at my previous suggestions: it seems I was mistaken about there being a proper v5.0.3 release of Manticore (at least, thus far). They have a tag in their repo, but no built binaries.

[Sandro Kalbermatter]

Hey Pat!

Great to hear you managed to solve the page issue, thanks a lot.

Yeah, I opened manticoresoftware/manticoresearch#912 but the bug is currently not manifesting. It's a slippery fish. I agree with your assumption that the problem is likely on their side.

Have a great day!