cgarciae / pypeln

Concurrent data pipelines in Python >>>

Home Page:https://cgarciae.github.io/pypeln

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to pass multiple arguments ?

amitbcp opened this issue · comments

I have taken the sample code from README.md and changed the function. But I am not able too pass the data :

Code :

import pypeln as pl
import time
from random import random

def slow_add1(x,y):
#     time.sleep(random()) # <= some slow computation
    return x + y

data = [{"x":x,"y":x*2} for x in range(10)]
stage = pl.thread.map(slow_add1, data , workers=2, maxsize=4)
list(stage)

I get the Error :

TypeError                                 Traceback (most recent call last)
<ipython-input-28-30e5fac190e8> in <module>
----> 1 list(stage)

~/anaconda3/lib/python3.8/site-packages/pypeln/thread/stage.py in to_iterable(self, maxsize, return_index)
     80 
     81         with supervisor:
---> 82             for elem in main_queue:
     83                 if return_index:
     84                     yield elem

~/anaconda3/lib/python3.8/site-packages/pypeln/thread/queue.py in __iter__(self)
     89 
     90             try:
---> 91                 x = self.get(timeout=pypeln_utils.TIMEOUT)
     92             except Empty:
     93                 continue

~/anaconda3/lib/python3.8/site-packages/pypeln/thread/queue.py in get(self, block, timeout)
     38                     exception = Exception(f"\n\nOriginal: {exception}\n\n{trace}")
     39 
---> 40                 raise exception
     41 
     42             # get is implemented like this for thread but not for process

TypeError: 

("slow_add1() missing 1 required positional argument: 'y'",)

Traceback (most recent call last):
  File "/Users/aamita/anaconda3/lib/python3.8/site-packages/pypeln/thread/worker.py", line 91, in __call__
    self.process_fn(
  File "/Users/aamita/anaconda3/lib/python3.8/site-packages/pypeln/thread/worker.py", line 180, in __call__
    self.apply(worker, elem, **kwargs)
  File "/Users/aamita/anaconda3/lib/python3.8/site-packages/pypeln/thread/api/map.py", line 27, in apply
    y = self.f(elem.value, **kwargs)
TypeError: slow_add1() missing 1 required positional argument: 'y'

I tried data as a list of supple, list of list but always the same error

Hey! We don't expand tuples or dicts as arguments, in your example you have to do something like:

def slow_add1(elem):
    x, y = elem["x"], elem["y"]
    return x + y