Giters

SymbolixAU / mapdeck

R interface to Deck.gl and Mapbox

Home Page:https://symbolixau.github.io/mapdeck/articles/mapdeck.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

add_polygon (and similar) crash using Rstudio when fill_colour is not blank on Windows

deanm0000 opened this issue · comments

commented

Edit by dcooley:

Until a fix is found, the current working solution to this appears to be downgrading to at the latests R 4.1.3

Describe the bug
Using the fill_colour parameter in shapes crashes Rstudio in windows.

If I remove the fill_colour parameter then it produces the map just fine (although without colors). I put the source code in the editor and the specific place that it crashes is on the line

  shape <- mapdeck:::rcpp_polygon_geojson(data, l, geometry_column, 
                                digits)

Oddly enough, if I run the same example in RGui then it doesn't crash but it does hang for at least 5 minutes before it returns results whereas Rstudio announces the fatal error session crash within 5 seconds.

To Reproduce
If I just use the example from the Colours page of the guide, ie.

pp <- spatialwidget::widget_melbourne

mapdeck() %>%
  add_polygon(
    data = pp , fill_colour = "SA2_NAME"
  )

that will produce a crash (or if in Rgui will take 5+ minutes to complete)

Expected behaviour
Not crashing ;)

Screenshots
If applicable, add screenshots to help explain your problem.

Versions

sessionInfo()
R version 4.2.1 (2022-06-23 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] sf_1.0-8          mapdeck_0.3.40004 RPostgres_1.4.4  

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.2    purrr_0.3.4         jsonify_1.2.1       vctrs_0.4.1         generics_0.1.3      htmltools_0.5.3    
 [7] utf8_1.2.2          blob_1.2.3          rlang_1.0.4         e1071_1.7-11        pillar_1.8.0        glue_1.6.2         
[13] DBI_1.1.3           bit64_4.0.5         lifecycle_1.0.1     htmlwidgets_1.5.4   fastmap_1.1.0       class_7.3-20       
[19] fansi_1.0.3         Rcpp_1.0.9          KernSmooth_2.23-20  classInt_0.4-7      bit_4.0.4           hms_1.1.1          
[25] digest_0.6.29       dplyr_1.0.9         grid_4.2.1          cli_3.3.0           tools_4.2.1         magrittr_2.0.3     
[31] proxy_0.4-27        tibble_3.1.8        spatialwidget_0.2.4 pkgconfig_2.0.3     ellipsis_0.3.2      rstudioapi_0.13    
[37] R6_2.5.1            units_0.8-0         compiler_4.2.1
commented

That's very odd. Do you know anyone else with a Windows machine who can test this as well?

commented

I am on Windows and i can confirm the behaviour.

R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=German_Austria.utf8  LC_CTYPE=German_Austria.utf8    LC_MONETARY=German_Austria.utf8 LC_NUMERIC=C                   
[5] LC_TIME=German_Austria.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mapdeck_0.3.4

loaded via a namespace (and not attached):
 [1] stringfish_0.15.7      Rcpp_1.0.9             rstudioapi_0.13.0-9000 magrittr_2.0.3         qs_0.25.3              tidyselect_1.1.2      
 [7] RApiSerialize_0.1.0    lattice_0.20-45        R6_2.5.1               rlang_1.0.4            fastmap_1.1.0          fansi_1.0.3           
[13] dplyr_1.0.9            tools_4.2.0            grid_4.2.0             config_0.3.1           utf8_1.2.2             DBI_1.1.3             
[19] cli_3.3.0              htmltools_0.5.3        ellipsis_0.3.2         digest_0.6.29          assertthat_0.2.1       yaml_2.3.5            
[25] RcppParallel_5.1.5     tibble_3.1.7           lifecycle_1.0.1        purrr_0.3.4            htmlwidgets_1.5.4      vctrs_0.4.1           
[31] glue_1.6.2             sp_1.5-0               compiler_4.2.0         pillar_1.8.0           generics_0.1.2         spatialwidget_0.2.3   
[37] pkgconfig_2.0.3
commented

does it crash if you use a hex string literal as the fill colour?

mapdeck() %>%
  add_polygon(
    data = spatialwidget::widget_melbourne
    , fill_colour = "#FF0000"
  )
commented

yes it is still crashing

commented

GIven it's working on the CRAN checks, and this is not consistently crashing for other windows users, I really don't know how to solve this one...

Are either of you using any obscure C++ compilers?

commented

I tried to dig into your package, but I have some problems installing it.
First it is not working because of the LinkingTo spatialwidget (>= 0.2.1005)

I have spatialwidget 0.2.3 installed, which is not recognized as a newer package.
When I remove the (>= 0.2.1005) dependency, I get this error:

fatal error: geometries/utils/utils.hpp: No such file or directory
    6 | #include "geometries/utils/utils.hpp"

I get the same error when I try remotes::install_version("spatialwidget", version = "0.2.2")

and I have sfheaders ‘0.4.0’ installed.

I dont think I have a special C++ compiler. Is there a way to check that?

commented

Could I ask you to update to all the latest dependencies as well, so in a brand new session:

remotes::install_github("dcooley/geometries")
remotes::install_github("dcooley/sfheaders")
remotes::install_github("SymbolixAU/geojsonsf")
remotes::install_github("SymbolixAU/colourvalues")
remotes::install_github("SymbolixAU/spatialwidget")
remotes::install_github("SymbolixAU/mapdeck")

and then try again?

commented

I installed geojsonsf from SymbolixAU, because the dcooley fork is still at version 1.3.2.

Anyway, Rstudio is still crashing..

And I also can't build mapdeck due to this error:

fatal error: geometries/geometries.hpp: No such file or directory
   13 | #include "geometries/geometries.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [C:/PROGRA~1/R/R-42~1.0/etc/x64/Makeconf:259: arc.o] Error 1
ERROR: compilation failed for package 'mapdeck'
commented

if {geometries} installed correctly you should be able to run this code:

# remotes::install_github("dcooley/geometries")

library(Rcpp)

cppFunction(
  includes = '
    #include "geometries/geometries.hpp"
  ',
  code = '
    SEXP a_point( SEXP x ) {
      return geometries::matrix::to_geometry_matrix( x );
    }
  '
  , depends = "geometries"
)

a_point(1:3)
commented

Yeah i can run that.

> a_point(1:3)
     [,1] [,2] [,3]
[1,]    1    2    3
commented

and does this line run?

sfheaders:::rcpp_sfg_point(1:2, NULL, "XY")
commented

Yes, also runs.

sfheaders:::rcpp_sfg_point(1:2, NULL, "XY")
     [,1] [,2]
[1,]    1    2
attr(,"class")
[1] "XY"    "POINT" "sfg"
commented

and this one

js <- spatialwidget::widget_point( data = spatialwidget::widget_capitals, fill_colour = "country", legend = TRUE)

substr( js$data, 1, 200 )
commented

and then this one

l <- widget_polygon( widget_melbourne[1:2, ], fill_colour = "AREASQKM16", legend = F)
substr( l$data, 1, 200 )
commented

no, this line crashes RStudio:

js <- spatialwidget::widget_point( data = spatialwidget::widget_capitals, fill_colour = "country", legend = TRUE)

commented

ok I've just pushed an update to spatialwidget to force the correct down-stream versions align. Could you restart you ression, then

remotes::install_github("dcooley/geometries")
remotes::install_github("dcooley/sfheaders")
remotes::install_github("SymbolixAU/spatialwidget")

js <- spatialwidget::widget_point( data = spatialwidget::widget_capitals, fill_colour = "country", legend = TRUE)
commented

It still crashes.

Not sure if it helps, but in the Rstudio session log I am getting this Error:

ERROR CLIENT EXCEPTION (rsession-user): (TypeError) : Cannot read property 'O' of null;|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditor.java#4509::setScrollSpeed|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#46::monitor|||org/rstudio/studio/client/workbench/views/source/editors/text/AceEditorMonitor.java#70::execute|||com/google/gwt/core/client/impl/SchedulerImpl.java#140::execute|||com/google/gwt/core/client/impl/Impl.java#306::apply|||com/google/gwt/core/client/impl/Impl.java#345::entry0|||rstudio-0.js#-1::eval|||com/google/gwt/cell/client/AbstractEditableCell.java#41::viewDataMap|||Client-ID: 33e600bb-c1b1-46bf-b562-ab5cba070b0e|||User-Agent: Mozilla/5.0 (Windows NT 10.0  Win64  x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36
commented

this?

  n <- 5
  params <- list(fill_colour = "col1")
  df <- data.frame(col1 = 1:n)
  lst_params <- spatialwidget:::rcpp_construct_params(df, params)
  data <- df
  total_colours <- nrow( df )
  repeats <- 1L
  lst_defaults <- list(fill_colour = rep(1,5))

  colour_name <- "fill_colour"
  opacity_name <- "fill_colour"
  lst_legend <- list("stroke_colour" = TRUE)
  include_legend <- FALSE

  colour_format = "hex"

  res <- spatialwidget:::rcpp_resolve_colour(
    lst_params,
    params,
    data,
    lst_defaults,
    colour_name,
    opacity_name,
    lst_legend,
    include_legend,
    repeats,
    total_colours,
    colour_format
  )
commented

This runs and returns:

res
$defaults
$defaults$fill_colour
[1] "#44015400" "#3B528B3F" "#21908C7F" "#5DC963BF" "#FDE725FF"


$legend
$legend$stroke_colour
[1] TRUE
commented

ok this is starting to narrow it down. I'll see if I can carry on debugging tomorrow.

Thanks for your help.

commented

actually one last one

df <- spatialwidget::widget_capitals
l <- list(fill_colour = "country")
res <- spatialwidget:::rcpp_construct_params(df, l)

should get

 $parameter
[1] "fill_colour"

$parameter_type
[1] 16

$data_column_index
[1] 0
commented

yes that works

res
$parameter
[1] "fill_colour"

$parameter_type
[1] 16

$data_column_index
[1] 0
commented

sorry for not keeping current. I just went through the latest troubleshooting steps and got the same results as trafficonese

commented

I'm having the same problem. Both stroke_colour and fill_colour hang up for me but it only happens when I assign a hex value. If I assign a column, at least fill_colour works as expected.

commented

There's been a few updates to the imported libraries' C++ code in the last few weeks. Could you try updating those pacakges, and try this again?

## restart your R session
install.packages(c("geometries","sfheaders","colourvalues"))
install.packages("mapdeck")
commented

I've recently tried to use fill_colour() and regardless of what function of mapdeck I try to use it in it crashes my session. I've tried to run this on two computers both running windows and its the same issue.

commented

Reverting to R 4.1.2 seems to have worked

commented

Can someone try these code chunks and let me know if either crash:

  df <- spatialwidget::widget_capitals
  l <- list(fill_colour = "country", stroke_colour = "capital", legend = T)
  data_rows <- nrow( df )
  lst_defaults <- list(
    fill_colour = rep(1.0, data_rows)
    , stroke_colour = rep(1.0, data_rows)
  )

  layer_legend <- c("fill_colour","stroke_colour")
  parameter_exclusions <- c("legend","legend_options","palette","na_colour")

  res <- spatialwidget:::rcpp_params_to_data(
    df, l, lst_defaults, layer_legend, data_rows, parameter_exclusions,
    factors_as_string = TRUE
  )
  n <- 5
  lst_params <- list(parameter = c("stroke_colour", "stroke_width"), parameter_type = c(), data_column_index = c() )
  params <- list(stroke_colour = "col1", stroke_width = 3)
  param_names <- lst_params[[ "parameter" ]]
  df <- data.frame(col1 = 1:n)
  data_names <- names( df )
  data <- df
  data_rows <- nrow( df )
  lst_defaults <- list(stroke_opacity = rep(100, n))

  spatialwidget:::rcpp_construct_data( param_names, params, data_names, lst_defaults, data, data_rows )

# stroke_opacity stroke_colour stroke_width
# 1            100             1            3
# 2            100             2            3
# 3            100             3            3
# 4            100             4            3
# 5            100             5            3

df <- spatialwidget::widget_capitals
  l <- list(fill_colour = "country", stroke_colour = "capital", legend = T)
  p <- spatialwidget:::rcpp_construct_params( data = df, params = l)
  pn <- names( l )
  ll <- c("fill_colour","stroke_colour")

  spatialwidget:::rcpp_construct_legend_list(
    lst_params = p
    , params = l
    , param_names = pn
    , legend_types = ll
    )

# $fill_colour
# [1] TRUE
# 
# $stroke_colour
# [1] TRUE
commented

Hi, the first one is crashing (the last expression, on R 4.3.1, not on R 4.1.3 where it is working fine)
Second and third are OK and not crashing.

commented

ok one more:

  lst <- list(let = letters, vals = 1:5)
  new_lst <- spatialwidget:::rcpp_remove_list_elements(lst, "vals")
commented

That one works perfectly

$let
 [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s" "t" "u" "v" "w" "x" "y" "z"
commented

ok we have to go old-school because I have no idea what's causing this, and I can't reproduce it:

could you / someone

  1. clone the spatialwidget library git clone https://github.com/symbolixau/spatialwidget
  2. checkout the stopping branch - git checkout stopping
  3. build the pacakge and run this code
  df <- spatialwidget::widget_capitals
  l <- list(fill_colour = "country", stroke_colour = "capital", legend = T)
  data_rows <- nrow( df )
  lst_defaults <- list(
    fill_colour = rep(1.0, data_rows)
    , stroke_colour = rep(1.0, data_rows)
  )

  layer_legend <- c("fill_colour","stroke_colour")
  parameter_exclusions <- c("legend","legend_options","palette","na_colour")

  res <- spatialwidget:::rcpp_params_to_data(
    df, l, lst_defaults, layer_legend, data_rows, parameter_exclusions,
    factors_as_string = TRUE
  )
  1. hopefully you should get the error
Error: 1

  1. Then go to the inst/include/spatialwidget/parameters/parameters.hpp file and comment out line 87: https://github.com/SymbolixAU/spatialwidget/blob/stopping/inst/include/spatialwidget/parameters/parameters.hpp#L87

  2. Run teh code again. You should see Error: 2

  3. comment out the next Rcpp::stop() line and run again. Repeat this until you get the crash, and let me know which Rcp::stop() occurs before and which occurs after the crash?

commented

1 to 4 : ok
Running the code generate Error: 1
5 ok and 6 Error: 2
Crashed instead of Error: 5 (so I saw Error: 4, commented out line 120 and not line 148

commented

Any new updates on this issue?

commented

does it crash if you include the fill_opacity argument?

pp <- spatialwidget::widget_melbourne

mapdeck() %>%
  add_polygon(
    data = pp, 
    fill_colour = "SA2_NAME",
    fill_opacity = 1.0
  )
commented

Yeah, the above code crashed for me. However, it doesn't crash if I leave out fill_colour.

mapdeck() %>%
  add_polygon(
    data = pp, 
    #fill_colour = "SA2_NAME",
    fill_opacity = 0.2
  )
commented

could your run each of the colour tests and palette tests?

Colours

library(testthat)
library(spatialwidget)

n <- 5
  params <- list(fill_colour = "col1")
  df <- data.frame(col1 = 1:n)
  lst_params <- spatialwidget:::rcpp_construct_params(df, params)
  data <- df
  total_colours <- nrow( df )
  repeats <- 1L
  lst_defaults <- list(fill_colour = rep(1,5))

  colour_name <- "fill_colour"
  opacity_name <- "fill_colour"
  lst_legend <- list("stroke_colour" = TRUE)
  include_legend <- FALSE

  colour_format = "hex"

  res <- spatialwidget:::rcpp_resolve_colour(
    lst_params,
    params,
    data,
    lst_defaults,
    colour_name,
    opacity_name,
    lst_legend,
    include_legend,
    repeats,
    total_colours,
    colour_format
  )

expect_equal(
    res$defaults$fill_colour, colourvalues::colour_values(1:5, alpha = 1:5)
  )

n <- 5
  params <- list(fill_colour = "col1")
  df <- data.frame(col1 = 1:n)
  lst_params <- spatialwidget:::rcpp_construct_params(df, params)
  data <- df
  total_colours <- nrow( df )
  repeats <- 1L
  lst_defaults <- list(fill_colour = rep(1,5))

  colour_name <- "fill_colour"
  opacity_name <- "not_a_column_will_use_default"
  lst_legend <- list("stroke_colour" = TRUE)
  include_legend <- FALSE

  colour_format = "hex"

  res <- spatialwidget:::rcpp_resolve_colour(
    lst_params,
    params,
    data,
    lst_defaults,
    colour_name,
    opacity_name,
    lst_legend,
    include_legend,
    repeats,
    total_colours,
    colour_format
  )

expect_equal(
    res$defaults$fill_colour, colourvalues::colour_values(1:5)
  )

n <- 5
  params <- list(fill_colour = "col1")
  df <- data.frame(col1 = 1:n)
  lst_params <- spatialwidget:::rcpp_construct_params(df, params)
  data <- df
  total_colours <- nrow( df )
  repeats <- 1L
  lst_defaults <- list(fill_colour = rep(1,5))

  colour_name <- "fill_colour"
  opacity_name <- "fill_colour"
  lst_legend <- list("stroke_colour" = TRUE)
  include_legend <- FALSE

  colour_format = "rgb"

  res <- spatialwidget:::rcpp_resolve_colour(
    lst_params,
    params,
    data,
    lst_defaults,
    colour_name,
    opacity_name,
    lst_legend,
    include_legend,
    repeats,
    total_colours,
    colour_format
  )

expect_equal(
    res$defaults$fill_colour, colourvalues::colour_values_rgb(1:5, alpha = 1:5)
  )

Palette

 lst_params <- list(parameter = c("stroke_colour", "palette", "stroke_width"), parameter_type = c(), data_column_index = c() )
  params <- list(stroke_colour = "col1", palette = "mypalette", stroke_width = 3)
  expect_equal( spatialwidget:::rcpp_resolve_palette( lst_params, params ), "mypalette" )

  lst_params <- list(parameter = c("stroke_colour", "stroke_width"), parameter_type = c(), data_column_index = c() )
  params <- list(stroke_colour = "col1", stroke_width = 3)
  spatialwidget:::rcpp_resolve_palette( lst_params, params )
  expect_equal( spatialwidget:::rcpp_resolve_palette( lst_params, params ), "viridis" )


  res <- spatialwidget:::rcpp_colour_str_with_palette("viridis", letters, 255.0, "#808080FF", TRUE, "fill_colour" )
  col <- colourvalues::color_values(letters, summary = T)

  expect_equal( res$colours, col$colours )
  expect_equal( res$summary_colours, col$summary_colours )
  expect_equal( res$summary_values, col$summary_values )

  res <- spatialwidget:::rcpp_colour_num_with_palette("viridis", 1:26, 255.0, "#808080FF", TRUE, "fill_colour" )
  col <- colourvalues::color_values(1:26, n_summaries = 5, format = T)

  expect_equal( res$colours, col$colours )
  expect_equal( res$summary_colours, col$summary_colours )
  expect_equal( res$summary_values, col$summary_values )

  res <- spatialwidget:::rcpp_colour_num_with_palette("viridis", 1:26, 255.0, "#808080FF", TRUE, "fill_colour", 5)
  col <- colourvalues::color_values(1:26, n_summaries = 5, format = T, digits = 5)

  expect_equal( res$colours, col$colours )
  expect_equal( res$summary_colours, col$summary_colours )
  expect_equal( res$summary_values, col$summary_values )
commented

Hi, I've had the same behavior since last year.
@dcooley I assume you ask to run the tests with the dev version of spatialwidget, otherwise spatialwidget:::rcpp_resolve_colour will throw an error of unused arguments.

I get successful results for all colour and palette tests except the first colour test, which throws:

Error: res$defaults$fill_colour not equal to colourvalues::colour_values(1:5).
4/5 mismatches
x[1]: "#44015400"
y[1]: "#440154FF"

x[2]: "#3B528B3F"
y[2]: "#3B528BFF"

x[3]: "#21908C7F"
y[3]: "#21908CFF"

x[4]: "#5DC963BF"
y[4]: "#5DC963FF"
commented

I just updated all the relevant packages and don't get any errors for the tests, seems like I had an old version of colourvalues. Mapdeck still doesn't work if using fill_colour. It doesn't crash for me, it just hangs forever and I have to restart the R session.

commented

I manually added stops to see where the crash happens and I think it leads to here: https://github.com/SymbolixAU/spatialwidget/blob/stopping/inst/include/spatialwidget/colour/colour.hpp#L60

commented

ok we can test this directly:

does this work?

library(Rcpp)

cppFunction(
  
  depends = c( "spatialwidget")
  , includes = c('#include "spatialwidget/utils/colour/colour.hpp"')
  , code = 'void testCString() {
  
    Rcpp::StringVector colour_vec = {"#FFFFFF"};
    Rcpp::String first_colour = colour_vec[0];
    if ( spatialwidget::utils::colour::is_hex( first_colour.get_cstring() ) ) {
      Rcpp::Rcout << "true" << std::endl;
    } else {
      Rcpp::Rcout << "false" << std::endl;
    }  
  }'
)

testCString()
# true
commented

That code hangs forever for me and makes the R session unresponsive, same as when using fill_colour in mapdeck.

commented

Ok, I did some extra digging and found out that the problem comes from the C++ regex library in Windows. More specifically, the code hangs at this line: https://github.com/SymbolixAU/spatialwidget/blob/4679fc3695665323407114151a9e2630677e7231/inst/include/spatialwidget/utils/colour/colour.hpp#L12

I found that others have solved this by avoiding the use of regex entirely.

So I cloned spatialwidget, and edited the is_hex() function in spatialwidget/inst/include/spatialwidget/utils/colour/colour.hpp as follows:

inline bool isHexDigit(char c) {
  return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
}

inline bool is_hex( std::string str ) {
  int len = str.length();
  if (len != 7 && len != 9 && len != 4 && len != 5) {
    return false; // Invalid length for a hex color
  }
  
  if (str[0] != '#') {
    return false; // Missing "#" at the beginning
  }
  
  for (int i = 1; i < len; i++) {
    if (!isHexDigit(str[i])) {
      return false; // Non-hexadecimal character found
    }
  }
  
  return true; // Passed all checks, valid hex color
}

Then I installed this modified version of spatialwidget, reinstalled mapdeck from source using this version of spatialwidget, and voilá! Mapdeck seems to be finally working with fill_colour!
Not sure if this is the best implementation for is_hex() but seems like this is the culprit.

commented

good work - I'll update the is_hex function with your code, add some tests, then push to CRAN.

commented

Great! Thanks so much @dcooley!