I cant use .shift() on columns that hold lists as values.

Question

I cant use .shift() on columns that hold lists as values.

amorimds opened this issue 3 years ago · comments

kdf = ks.DataFrame({'A': [1, 1, 2, 2], 'B': [[1, 1, 2, 2], [1, 1, 2, 2], [1, 1, 2, 2], [1, 1, 2, 2]]}, columns=['A', 'B'])
kdf.groupby('A')['B'].shift(1)

Danilo Amorim · Answer 1 · Thu Oct 28 2021 22:51:54 GMT+0800 (China Standard Time)

I could go around this problem with:

kdf = ks.DataFrame({'A': [1, 1, 2, 2], 'B': [[1, 1, 2, 2], [1, 1, 2, 2], [1, 1, 2, 2], [1, 1, 2, 2]]}, columns=['A', 'B'])
kdf['B'] = kdf['B'].astype(str)
kdf.groupby('A')['B'].shift(1).apply(lambda x: eval(str(x)))

Haejoon Lee · Answer 2 · Thu Dec 09 2021 09:47:59 GMT+0800 (China Standard Time)

Thanks for the report, @amorimds .

And currently the Koalas project is only in maintaining mode, so the response could be quite delayed.

The Koalas project is currently being managed more actively in PySpark under the name of "pandas API on Spark" (you can simply re-use the existing Koalas code by importing import pyspark.pandas as ks)

So if you're going to continue using Koalas, I recommend using PySpark! (You can get a quicker response if you report the issue to the Apache Spark JIRA)