Is it possible to perform Pandas profiling on Weld

[SHASHWAT TIWARI]

Hi,

I would like to perform pandas profiling on particular data frame. Whether it can be achieved through weld.

Thanks

[Shoumik Palkar]

Hmm, do you mean performance profiling? You can measure how long evaluating a lazy computation takes (e.g., by surrounding a call to evaluate() with time.time()), but unfortunately there isn't any easy way to see how long each operation in a graph takes (the operations will likely be fused into a single loop, so its hard to tease apart where time should be attributed).

…

On Fri, Jul 12, 2019 at 12:10 AM SHASHWAT TIWARI ***@***.***> wrote: Hi, I would like to perform pandas profiling on particular data frame. Whether it can be achieved through weld. Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#460?email_source=notifications&email_token=AAKMEYZ7GL5LXKY5LWTJ23LP7AU5TA5CNFSM4ICDKW52YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6ZZCBA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKMEY753A4CODHB66WPFALP7AU5TANCNFSM4ICDKW5Q> .

-- Shoumik

[SHASHWAT TIWARI]

No, I'm asking about how to perform data profiling (https://pypi.org/project/pandas-profiling/) on top of DataFrameWeld under grizzly or some other classes.

eg (done on top of pandas DataFrame)
p = pandas_profiling.ProfileReport(df,check_correlation=False)
p.to_file("profiling01.html")

Here, pandas profiling takes only dataframe as input. Is it possible to give Weld Data frame as input. If yes please let me know how to do.

I had also tried with weld_df.df. basically it will convert weld object to pandas data frame and then work. But time taken is not reduced, it take some what higher than normal profiling.

Thanks

[Shoumik Palkar]

Aha, yes, right now we don't support that feature unfortunately (looking at what it produces, Grizzly and Baloo <https://github.com/weld-project/baloo> have functions that produce some of these statistics, but not through that interface).

…

On Mon, Jul 15, 2019 at 3:51 AM SHASHWAT TIWARI ***@***.***> wrote: No, I'm asking about how to perform *data profiling* ( https://pypi.org/project/pandas-profiling/) on top of DataFrameWeld under grizzly or some other classes. eg (done on top of pandas *DataFrame*) *p = pandas_profiling.ProfileReport(df,check_correlation=False) p.to_file("profiling01.html")* Here, pandas profiling takes only *dataframe* as input. Is it possible to give *Weld Data frame* as input. If yes please let me know how to do. I had also tried with *weld_df.df*. basically it will convert weld object to pandas data frame and then work. But time taken is not reduced, it take some what higher than normal profiling. [image: profiling] <https://user-images.githubusercontent.com/22785727/61211175-8099d980-a71c-11e9-998d-fe8ae8c514c8.PNG> Thanks — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#460?email_source=notifications&email_token=AAKMEY5XYH64TI5QYN3KJ3TP7RJB5A5CNFSM4ICDKW52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ5KVTI#issuecomment-511355597>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKMEY765YNEHHSYCOKB3MDP7RJB5ANCNFSM4ICDKW5Q> .

-- Shoumik

[SHASHWAT TIWARI]

Thanks cool!