sparklyr / sparklyr

R interface for Apache Spark

Home Page:https://spark.rstudio.com/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

as.raw type conversion in mutate doesn't seem to translate to spark sql properly.

ahcyip opened this issue · comments

I'm trying to convert a column into "ByteType" with sparklyr.

mutate(a = as.raw(b)) doesn't seem to translate into spark sql properly.
as.character works as expected to convert to StringType.

error message:
org.apache.spark.sql.AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function as.raw on search path [system.builtin, system.session, spark_catalog.default].; line 24 pos 2

R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] tidyr_1.3.0 dplyr_1.1.4 stringr_1.5.1 sparklyr_1.8.4 SparkR_3.4.0

loaded via a namespace (and not attached):
[1] rstudioapi_0.15.0 magrittr_2.0.3 uuid_1.2-0 tidyselect_1.2.0
[5] R6_2.5.1 rlang_1.1.3 fansi_1.0.6 blob_1.2.4
[9] httr_1.4.7 tools_4.1.2 parallel_4.1.2 config_0.3.2
[13] utf8_1.2.4 cli_3.6.2 DBI_1.2.1 withr_3.0.0
[17] dbplyr_2.4.0 askpass_1.2.0 yaml_2.3.8 openssl_2.1.1
[21] tibble_3.2.1 lifecycle_1.0.4 purrr_1.0.2 vctrs_0.6.5
[25] glue_1.7.0 stringi_1.8.3 compiler_4.1.2 pillar_1.9.0
[29] generics_0.1.3 jsonlite_1.8.8 pkgconfig_2.0.3