Can not spawn new subprocess after long time running
silencej opened this issue · comments
As in marcinhlybin/prometheus-barman-exporter#15 a barman_exporter uses sh module to scrape server status. But after long time running, the process is stuck.
The debug log is as follows. It halts at "starting process".
2021-06-29 11:12:45,681 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json show-backup myServer2 20210628T150023', pid 8379>: process completed
2021-06-29 11:12:45,682 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>: starting process
2021-06-29 11:12:45,694 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: started process
2021-06-29 11:12:45,694 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdin: parsed stdin as a None
2021-06-29 11:12:45,713 debug:645 61:140021602597664 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: <sh.StreamWriter object at 0x7f5951e7eeb0> ready for more input
2021-06-29 11:12:45,713 debug:645 61:140021602597664 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdin: done reading
2021-06-29 11:12:45,713 debug:645 61:140021602597664 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdin: closing, but flushing first
2021-06-29 11:12:45,713 debug:645 61:140021602597664 acquiring buffering lock for flushing buffer
2021-06-29 11:12:45,713 debug:645 61:140021602597664 got buffering lock for flushing buffer
2021-06-29 11:12:45,714 debug:645 61:140021602597664 released buffering lock for flushing buffer
2021-06-29 11:12:45,714 debug:645 61:140021602597664 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdin: got chunk size 0 to flush: b''
2021-06-29 11:12:45,726 info:642 61:140021604444960 <Command '/usr/bin/barman -f json status myServer', pid 8384>: process started
2021-06-29 11:12:45,726 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: acquiring wait lock to wait for completion
2021-06-29 11:12:45,726 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: got wait lock
2021-06-29 11:12:45,726 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: exit code not set, waiting on pid
2021-06-29 11:12:46,198 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: <sh.StreamReader object at 0x7f5951e7e580> ready to be read from
2021-06-29 11:12:46,198 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdout: got chunk size 1024: b'{"myServer": {"acti'
2021-06-29 11:12:46,198 debug:645 61:140021599111968 acquiring buffering lock to process chunk (buffering: 1)
2021-06-29 11:12:46,198 debug:645 61:140021599111968 got buffering lock to process chunk (buffering: 1)
2021-06-29 11:12:46,198 debug:645 61:140021599111968 released buffering lock for processing chunk (buffering: 1)
2021-06-29 11:12:46,199 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: <sh.StreamReader object at 0x7f5951e7e580> ready to be read from
2021-06-29 11:12:46,199 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdout: got chunk size 290: b'iption": "PostgreSQL version",'
2021-06-29 11:12:46,199 debug:645 61:140021599111968 acquiring buffering lock to process chunk (buffering: 1)
2021-06-29 11:12:46,199 debug:645 61:140021599111968 got buffering lock to process chunk (buffering: 1)
2021-06-29 11:12:46,199 debug:645 61:140021599111968 released buffering lock for processing chunk (buffering: 1)
2021-06-29 11:12:46,228 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: <sh.StreamReader object at 0x7f5951e7e580> ready to be read from
2021-06-29 11:12:46,229 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdout: got errno 5, done reading
2021-06-29 11:12:46,229 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>: <sh.StreamReader object at 0x7f5951e7eca0> ready to be read from
2021-06-29 11:12:46,229 debug:645 61:140021599111968 got no chunk, done reading
2021-06-29 11:12:46,230 debug:645 61:140021599111968 acquiring buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 got buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 released buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdout: got chunk size 1314 to flush: b'{"myServer": {"acti'
2021-06-29 11:12:46,230 debug:645 61:140021599111968 <Command '/usr/bin/barman -f json status myServer'>.<Process 8384 [b'/usr/bin/barman', b'-f', b'json', b'status', b'myServer']>.stdout: putting chunk onto pipe: b'{"myServer": {"acti'
2021-06-29 11:12:46,230 debug:645 61:140021599111968 acquiring buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 got buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 released buffering lock for flushing buffer
2021-06-29 11:12:46,230 debug:645 61:140021599111968 got chunk size 0 to flush: b''
2021-06-29 11:12:46,231 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json status myServer', pid 8384>: process completed
2021-06-29 11:12:46,231 debug:645 61:140021604444960 <Command '/usr/bin/barman -f json check myServer'>: starting process
It is most likely an issue with barman, otherwise there would be many similar reports from other programs. Please keep this thread updated with the results of the barman developer's investigations.
It is most likely an issue with barman, otherwise there would be many similar reports from other programs. Please keep this thread updated with the results of the barman developer's investigations.
I set up a timeout but it's not working either:
from sh import barman as barman_cli
output = barman_cli('-f', 'json', _timeout=10, *args, **kwargs)I suppose if the barman command is broken sometime, the timeout should save it.
Thanks for the additional info. I will be able to make further progress once the barman developers conclude their investigation.
Closing, as this seems to be an issue with barman.