cgarciae / pypeln

Concurrent data pipelines in Python >>>

Home Page:https://cgarciae.github.io/pypeln

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Question] Could you help me with using tqdm

iqzer0 opened this issue · comments

When i try to use tqdm with pypeln it's not working for me, could you please help ?

from aiohttp import ClientSession, TCPConnector
import asyncio
import pypeln as pl
from tqdm.asyncio import trange, tqdm

limit = 1
users = list(range(1,10))

async def fetch(users, session):
    None

async def main():
	async for i in trange(len(users)):
	   pl.task.each(
	      fetch,
	      users,
	      workers=limit,
	      on_start=lambda: dict(session=ClientSession(connector=TCPConnector(limit=None,ssl=False))),
	      on_done=lambda session: session.close(),
	    run=True,
	   )

asyncio.run(main())

When i run the code the progress bar is not progressing. could you tell me why it happens ?

no worries.. i managed to find a simpler way.. thanks for the awesome library ;)

from aiohttp import ClientSession, TCPConnector
import asyncio
import pypeln as pl
from tqdm.asyncio import trange, tqdm
import time

limit = 1
users = list(range(1,10))

async def fetch(users, session):
    time.sleep(1)
    pbar.update(1)

with tqdm(total=len(users)) as pbar:
    pl.task.each(
        fetch,
        users,
        workers=limit,
        on_start=lambda: dict(session=ClientSession(connector=TCPConnector(limit=None,ssl=False))),
        on_done=lambda session: session.close(),
        run=True,
    )