databricks / koalas

Koalas: pandas API on Apache Spark

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Whether the `apply` function is implemented using the pandas_udf function?

RainFung opened this issue · comments

commented

Whether the apply function is implemented using the pandas_udf function?

Yeah, for some case we use mapInPandas which leveraging the pandas_udf internally.

if should_use_map_in_pandas:
output_func = GroupBy._make_pandas_df_builder_func(
self_applied, apply_func, return_schema, retain_index=True
)
sdf = self_applied._internal.to_internal_spark_frame.mapInPandas(
lambda iterator: map(output_func, iterator), schema=return_schema
)