How do I monitor my instances (RAM/Swap usage)
jelz opened this issue · comments
Description
I want to set up monitoring and alerting for our Typesense instances. We use Typesense Cloud but I think it would be the same for self-hosted clusters.
According to the Cloud dashaboard for writes to be accepted the following must be true: RAM used + Swap used < RAM total. I wanted to page myself when RAM used + Swap used is at 80% of RAM total. While dashboard provides information on both RAM and Swap usage, metrics fetched using the SDK miss the Swap usage.
Steps to reproduce
await client.metrics.retrieve()
{
system_cpu1_active_percentage: '0.00',
system_cpu2_active_percentage: '0.00',
system_cpu_active_percentage: '0.00',
system_disk_total_bytes: '8141348864',
system_disk_used_bytes: '4917432320',
system_memory_total_bytes: '892403712',
system_memory_used_bytes: '537108480',
system_network_received_bytes: '1549922823',
system_network_sent_bytes: '3249466397',
typesense_memory_active_bytes: '348004352',
typesense_memory_allocated_bytes: '307501632',
typesense_memory_fragmentation_ratio: '0.12',
typesense_memory_mapped_bytes: '377749504',
typesense_memory_metadata_bytes: '15846208',
typesense_memory_resident_bytes: '348004352',
typesense_memory_retained_bytes: '258736128'
}
Expected Behavior
Given the metrics I can reliably determine if writes will be accepted so I can alert myself.
Actual Behavior
No way to understand Swap usage.
Note that we run HA Cluster with Typesense Cloud but we only get one metrics object back. Is it for the node we got load balanced to?
Metadata
Typesense Version: Typesense v0.25.2 running on Typesense Cloud
OS: AWS Lambda using typesense-js v1.7.2
0.26 RC builds already have a system_memory_used_swap_bytes value in the metrics end-point.