UV images and thumbnails not loading in production
eporter23 opened this issue · comments
As of about 9 AM this morning, I'm seeing no thumbnails in search results except for our Login required placeholders.
IIIF manifests also aren't loading images - I spot checked across various collections including works that I've visited in the last couple of days.
Collection banner images are loading successfully.
UPDATE:
At the same time this was occurring Curate prod was not responding, which probably caused IIIF requests to not respond. Curate came back on its own, but it was inaccessible from about 9:10 AM - 9:25 AM 8/5.
Only thing I'm seeing so far is that the number of httpd processes is spiking around the time of the non-responsiveness. (13:20 UTC is 9:20 EST)
Current plan is to bring down the PassengerMaxQueSize variable, currently set to 1000, down to 200-300, this should cause the server to respond with a 503/Server too busy message instead of appearing to be down. Hopefully this will increase clarity as to what's going on with the server.
Another thread is to change from prefork to worker MPM, this should allow for better performance. Worker MPM will need to be tested out to ensure compatibility.
@SolomonHD some updates from testing in arch around 12pm today: I was able to trigger the same behavior (to a lesser degree) by:
- Opening up and editing 5 or 6 large book objects in Curate in quick succession, keeping many tabs open and refreshing them. The editing step will cause the manifest to regenerate.
- I occasionally got a blank screen in Curate in some tabs with the browser reload bar moving very slowly
- I then went into digital-arch and saw that thumbnails were not loading, and UV images were not loading. After a minute or so (as the Curate edits died down), images came back.
- An additional edit on a very large book then caused images to stop loading Lux, and no pages in Curate are loading or able to be refreshed.
I didn't see any error status messages, in Curate, however, just a blank screen.
Update: I saw a queue is full error at about 12:25:
@libdgg to review with @SolomonHD to get status update.