Giters

tqdm / tqdm

:zap: A Fast, Extensible Progress Bar for Python and CLI

Home Page:https://tqdm.github.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Hanging pipes in python2

jmarkow opened this issue · comments

commented

Hopefully I'm using this as intended, but I'm trying to add a simple progress bar to a bash script.

#!/bin/bash

while true; do
        echo -ne "Testing\n"
        sleep 1
done

The output looks as intended (testing printed on a new line every 1 second). However, tqdm never progresses if I pipe in the output from this program as follows (tried with and without totals, different delimiters, all no luck):

./test.sh | tqdm --total "5"

The output is stuck here (checked up to 10 minutes).

0it [00:00, ?it/s]

I've used tqdm in plenty of Python scripts but I'm not entirely sure I grok its usage in the command line.

commented

interesting. If you remove the sleep or make it sleep for a short amount of time it works. The python equivalent works without modification as expected. This is odd, I have no explanation (yet) - will investigate

commented

It only works if I comment out sleep entirely. I tried intervals down to 1 ms, sleep .001, with no luck.

commented

@jmarkow I'm using tqdm 4.11.2 and your bash script is working fine for me.

commented

@msampathkumar updated to 4.11.2 and still no dice on my end.

commented

Sorry haven't had much free time. It would be awesome @jmarkow if you could see if there was an older version (v4.4.0 onwards?) which works for you. That would help to determine what's wrong.

commented

@casperdcl Sounds good, I'll give that a shot and keep you posted.

commented

From playing around, I can only reproduce this issue when using sleep on debian arm64. Short sleeps work on ubuntu. Not sure if this is python's timing getting confused with piped programs? Could you confirm if there's any scenario where tqdm fails to function without sleep?

commented

minimal example (no tqdm, just pure python):

$ cat seq.py
"""usage: seq.py <sleep time> <num iterations>"""
import sys, time
t = float(sys.argv[2])
for i in range(int(sys.argv[1])):
    print(i)
    time.sleep(t)
$ cat pipe.py
from __future__ import print_function
import sys
from time import time
for l in sys.stdin:
    print(time(), end=', ')
    sys.stdout.write(l)
$ python seq.py 9 0.1 | python pipe.py
1496053138.42, 0
1496053138.42, 1
1496053138.42, 2
1496053138.42, 3
1496053138.42, 4
1496053138.42, 5
1496053138.42, 6
1496053138.42, 7
1496053138.42, 8

expected/desired behavior: output times increase by 0.1s

obtained behavior: output times constant, corresponding to end of stdin pipe.

commented

This is an python2 issue, and is fixed in python3.5. Consider using python3 -m tqdm (or install and use tqdm in a python3 environment) in your pipes instead of a python2 version.

for i in `seq 10`; do echo -ne "Testing\n"; sleep 0.1; done | python2 -m tqdm  # problem
for i in `seq 10`; do echo -ne "Testing\n"; sleep 0.1; done | python3 -m tqdm  # fix
                                                                    ^

Refs: