xueyujie / go-oom-guard

It will set "threshold notify" mechanism for memory cgroup to receive notifications from the kernel. This reduces the chance of Postgres killing by OOM.

go-oom-guard

It creates eventfd and register a notifier by given threshold in CGroup. Then it sends SIGINT or SIGTERM to top memory used process.

Problem

Postgres is unable to control memory usage by backends.

CGroup OOM Killer triggered by some query causes whole database restart.

How to mitigate it?

Possible solutions

1. Just monitor memory usage, make it less impact, recommendations by CrunchyData
2. LD_PRELOAD custom library, solution by TimescaleDB Cloud
3. Register cgroup memory event before OOM triggers. A lot of solutions, for example Tencent Cloud

This repo is example of #3

Prerequisites

• Docker
• python3, psycopg2
• psql
• Go

Demo

terminal #1:

$ make build
$ make docker
$ make docker-run

terminal #2:

$ docker exec -i postgres_oom_guarded ./go-oom-guard

terminal #3:

make test-oom-execute
make test-oom-parse

and the try it without running go-oom-guard, stop terminal #2, and run commands in terminal #3 again.

Cautions

It's only POC, don't use it on production AS IS!

TODOs

• how to get exactly postgres backend's pid ?

Links

• https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt - see 9. Memory thresholds
• https://man7.org/linux/man-pages/man2/eventfd.2.html