Issues uncovered by CRAN upon submission on 2022-01-18
astamm opened this issue · comments
The new version has the following issues:
-
tools/cmake_call.sh still attempts to use all cores,
-
for UCRT, please use the nlopt library from Rtools42, and
modify Description accordingly".
See https://developer.r-project.org/Blog/public/2021/12/07/upcoming-changes-in-r-4.2-on-windows/index.html
which clearly saysRtools42 includes libraries for almost all CRAN packages, which can and should
be used, instead. -
In the Description, you have
Building
from included sources requires CMake. On Linux and macOS, if
a system build of NLopt (2.7.0 or later) is found, it is
used; otherwise, it is build from included sources via
CMake. On Windows, NLopt is obtained through rwinlib.
@kalibera: I think you were cced by last CRAN email about that. I read some comments of yours about that in the repo. Could you please give me some hints as to how to do that for UCRT?
Related to there first comment, what is a good practice for using multiple cores? I indeed use there parallel::detectCores(logical = FALSE) to use all cores but it might indeed not be ideal. Should I use just one and be done with it? Or is there a more subtle way to deal with this that would allow me to use multiple cores?
Related to there first comment, what is a good practice for using multiple cores?
See Writing R Extensions. Never more than two. In package tiledb we want aggressive performance by default so in everything that CRAN may run (i.e. examples, test scripts, ...) I call a helper to set cores to variable if given and default to two if not -- ensures CRAN only sees two.
So you probably want to limit NCORES to two as well.
I was cc'd because of the Windows stuff. Re the cores - this is just for the building, right? I would not specify a core count at all, but let the external default be used, which I assume will be single core. I am not an expert on cmake, though. You can definitely respond to that email and ask for help if you didn't understand precisely what you should do (and you tried to find out first the answer in the documentation, which I assume is the case).
Re Windows, please just add a Makevars.ucrt file that has PKG_LIBS = -lnlopt and nothing else. This is what my patch that is now again automatically applied does: https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/r_packages/patches/CRAN/nloptr.diff (of course feel free to delete/update the comments, they are just copied from an earlier version). The Makevars.ucrt file will be used by R-devel and R 4.2, but ignored by R 4.1. So, that way, your package will be built using Makevars.win on R 4.1 (and earlier) and using Makevars.ucrt on R 4.2 (and current R-devel).
Thanks for your reply. We do ship a Makevars.ucrt that looks like:
CRT=-ucrt
include Makevars.win
where Makevars.win uses @jeroen winlibs.R script to grab NLopt from rwinlib.
Are you implying I should overwrite the Makevars.ucrt file and only fill it with PKG_LIBS = -lnlopt?
Yes.
Actually I need to set up something more in the Makevars.ucrt because I need a copy step for copying the headers in inst/include. How can I set up this copy step? Where should I copy the headers from?
Yes, in principle that is possible, look e.g. at rgdal package, you can copy it in the Makevars.ucrt. But why do you need to copy the headers? If another source package needs the headers, it will have them in the toolchain as well as yours. The copying is needed when an "installed" (so also binary) version of the package needs some data, such as "share/gdal" or "share/proj" in the rgdal case, but why would an installed version of the package need the headers?
Well, I thought that if I want to be able to use the C API of the nloptr package in another package by e.g. add nloptr to the LinkingTo field, headers are looked for in the include folder of the installed package, which requires headers to be copied in inst/include in the source code. But maybe I missed something?
nloptr, like other packages, makes headers available. That is fairly standard. Because there was demand, I even showed in an entire example package also on CRAN how that can be done. It's a feature that is IMHO too little used or understood with the CRAN world. We can do this as symbols gets loaded.
So I gather from rgdal that headers can be found here: $(R_TOOLS_SOFT)/share/nlopt/include/, am I right?
Well, yes, but really why would you be copying those? They are available automatically when anyone is building their package.
Because src/ != inst/ and only stuff in inst/ gets copied. The package exports an API when installed. And for that they have to be present on all platforms, not just UCRT.
I think we are confusing headers from the R package and headers from the C library. As Dirk writes, headers from the R package (your own headers) have to be copied to be in inst/ in the binary version of the package (and installed version of the package). But headers from the C library, which is in the toolchain, are available there and are not to be copied.
This is not the same as with r-winlib, because with r-winlib, the headers are not part of the toolchain, they are downloaded on demand by the package at installation time. They are copied there because otherwise the packages using say nloptr would have to be downloading the headers from r-winlib for nlopt...
I might start to understand your point @kalibera. However, I get errors from R-Hub windows 2022 platform because NLopt version in Rtools42 toolchain seems to be < 2.7. Did you update it the other day in the end?
I did update it, but R-hub apparently did not pick that up from me, yet. My builds are at https://www.r-project.org/nosvn/winutf8/ucrt3/ and specifically Rtools also linked from https://www.r-project.org/nosvn/winutf8/ucrt3/web/rtools.html
Without knowing the specifics of nloptr, I think in general packages should work even with older versions of the libraries, indeed possibly with reduced functionality (e.g. a missing algorithm that is only available in the new version). Having a hard dependency on particularly such a new version should be very rarely needed.
From the CI: https://github.com/astamm/nloptr/runs/4870876351?check_suite_focus=true, it fails though because it cannot find nlopt.h as I feared because I guess it cannot find in the PKG_CPPFLAGS a location where that file is found. Then, since I #include <nlopt.h> in some of my own C files, it fails.
Hmm, that's because of the "nlopt" prefix. It should help to add PKG_CPPFLAGS = -I$(R_TOOLS_SOFT)/include/nlopt or update the include directive (but there you would need to take care not to break it on other platforms). Some projects have such prefixes on one distribution but not on other.
But if I add PKG_CPPFLAGS = -I$(R_TOOLS_SOFT)/include/nlopt only to Makevars.ucrt, then it should only affect the Windows-devel platform, right?
Yes. So, if you have the prefix on all platforms, you can also instead change the #include <nlopt.h> to #include <nlopt/nlopt.h>. That also depends on your preference. Makevars.ucrt will apply to Windows (R-devel/R4.2), Makevars.win to Windows (older versions of R), Makevars to other platfoforms. Not all of these files have to exist - the details are in Writing R Extensions.
Yes I agree. But indeed, as you point it out, I do not have currently the prefix on other platform. But I can force them to. Thanks.
Right, then it makes sense to include just nlopt.h and update the PKG_CPPFLAGS in the Makevars file for the platforms that have the prefix.
Ok but there is no platform that have the prefix here, or am I missing something? By adding PKG_CPPFLAGS = -I$(R_TOOLS_SOFT)/include/nlopt to Makevars.ucrt, I should make nlopt.h includable with no prefix, shouldn't I?
I am asking because this is what I did in the last commit, but it still cannot find the header there.
Yes, this is what I meant. I can try on my system, which commit should I look at?
The very last one. Thanks!
It builds and checks fine on my system. There are only the visibility warnings from testthat code:
warning: visibility attribute not supported in this configuration; ignored [-Wattributes]
Hmm I cannot figure out why the CI fails on windows-2022 r-devel platform then.
I believe the issue is:
cp: cannot stat 'c:/rtools42/x86_64-w64-mingw32.static.posix/include/nlopt/*': No such file or directory
in which I believe c:/rtools42/x86_64-w64-mingw32.static.posix corresponds to $(R_TOOLS_SOFT).
In this run (https://github.com/astamm/nloptr/runs/4870876351?check_suite_focus=true), you didn't have the CPP FLAGS set, see
OE> gcc -I"C:/R/include" -DNDEBUG -I'D:/a/_temp/Library/testthat/include' -I"c:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -Wall -pedantic -c init_nloptr.c -o init_nloptr.o OE> init_nloptr.c:34:10: fatal error: nlopt.h: No such file or directory OE> 34 | #include "nlopt.h" OE> | ^~~~~~~~~
in the above, there is no -I directive adding the nlopt directory. My output has instead the "nlopt" there
gcc -I"C:/msys64/home/tomas/ucrt3/svn/ucrt3/R_PACK~1/rinst/include" -DNDEBUG -IC:/msys64/home/tomas/ucrt3/svn/ucrt3/r_packages/x86_64-w64-mingw32.static.posix/include/nlopt -I'C:/msys64/tmp/lib/testthat/include' -I"C:/msys64/home/tomas/ucrt3/svn/ucrt3/r_packages/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c init_nloptr.c -o init_nloptr.o
See this run: https://github.com/astamm/nloptr/runs/4871811401?check_suite_focus=true.
I believe that the issue lies in my last comment.
But I see that in my run, $(R_TOOLS_SOFT)points to c:/rtools42/x86_64-w64-mingw32.static.posix, while in yours, it points to C:/msys64/home/tomas/ucrt3/svn/ucrt3/r_packages/x86_64-w64-mingw32.static.posix, which means yours contain the nlopt folder whether mine does not. Question is: how can I ensure mine does?
Sorry, I am still puzzled which run I should look at. I don't see the message you cited. In https://github.com/astamm/nloptr/runs/4871811401?check_suite_focus=true. I see:
OE> gcc -I"C:/R/include" -DNDEBUG -Ic:/rtools42/x86_64-w64-mingw32.static.posix/include/nlopt -I'D:/a/_temp/Library/testthat/include' -I"c:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -Wall -pedantic -c init_nloptr.c -o init_nloptr.o OE> init_nloptr.c:34:10: fatal error: nlopt.h: No such file or directory OE> 34 | #include <nlopt.h> OE> | ^~~~~~~~~
which is puzzling to me. As if it were some quoting issue in the gihutb action implementation, as if it needed quotes...
The last commit switches to strategy where I copy the headers from $(R_TOOLS_SOFT)/include/nlopt into inst/include while the previous one uses PKG_CPPFLAGS = -I$(R_TOOLS_SOFT)/include/nlopt where both strategies ought to work if I understood correctly.
The last CI run you are referring to is related to the latter one.
Well I am not familiar with the system you use for github action. I instead used (and extended) a different one. Still, in mine, I set up whether to use "base" or "full" toolchain for the package. The "base" toolchain doesn't have all the libraries.
Certainly, you can also try on Winbuilder and if it works there, it will likely work also on the CRAN checks.
Ok. Please use the strategy of setting PKG_CPPFLAGS.
Will do. I am running on winbuilder-devel as we speak.
And I think you pointed out the issue. I should set up the CI script to use full toolchain.
Do you have documentation to help me do that please?
It does install on winbuilder devel, but seems to still have an NLopt version prior to 2.7: https://win-builder.r-project.org/oWMkq55cM38M/.
Ok, great. Could you simply not have version 2.7 as a requirement? You could test 2.7 features conditionally when the version is new enough.
Regarding your github actions, I see you are using
Workflow derived from https://github.com/r-lib/actions/tree/master/examples
Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help
So if you want to fix that, please try following up there. But certainly this is something that does not have to be fixed for a submission to CRAN.
I would like to enforce v2.7.0 or higher because I believe that softs depending on other softs should adapt to conform to the newest versions of their dependencies. My argument relies on the assumption that newer versions of softs are in general made of improvements.
On macOS or Linux, I use cmake to build from included sources if a system build of nlopt >= 2.7.0 is not found. On Windows, I thought of switching to the toolchain version for simplicity but if it cannot be guaranteed to be at least 2.7.0, I need to find out a way to build it, which brings me back to square one because the winbuilder devel machine does not have cmake.
Given the emergency here (nlopt scheduled for archiving next Friday, along with all of its hard downstream dependencies), one temporary solution is to allow the C API unit test suite to check for a version <2.7.0 by making the unit test that tests the version number more flexible. That would pass winbuilder checks I believe.
Testing that strategy right now. This should be however documented somewhere as it is making me realise that I never check for a version >=2.7.0 on Windows actually before installing. I only check it afterwards in the C API unit testing suite.
I would like to enforce v2.7.0 or higher because I believe that softs depending on other softs should adapt to conform to the newest versions of their dependencies. My argument relies on the assumption that newer versions of softs are in general made of improvements. On macOS or Linux, I use
cmaketo build from included sources if a system build ofnlopt >= 2.7.0is not found. On Windows, I thought of switching to the toolchain version for simplicity but if it cannot be guaranteed to be at least 2.7.0, I need to find out a way to build it, which brings me back to square one because the winbuilder devel machine does not havecmake.
I don't understand the last part, what did you mean by the need of building it? Did you mean it was hard to keep your package buildable also with older versions of the library? That would be a reason to make a hard dependency, yes.
Yes older versions of NLopt (~2.4) produced builds in which CRAN catches lots of warnings that were painful to resolve. I suspect these warnings might disappear as soon as we use a version of nlopt that relies on cmake but have no tests to back my guts here. I only used the latest version when I took upon the maintenance of the package. In other words, I am sure I want to use at least 2.5 but maybe not necessarily 2.7 as far as CRAN warnings are concerned. But, for my way of thinking these things (explained above), I would like to force 2.7 or higher.
By the time I wrote those lines, winbuilder devel got back at me positively. So I can for now just test for >= 2.6 in my unit tests to avoid them to fail on windows devel and we are good for now. I would though be happier in a recent future if we can guarantee >=2.7 on the rtools42 toolchain.
You could warn, but not die, if verson < 2.7.0 is detected. You could even do it on startup (if interactice() is TRUE and all that, of course).
Thanks. In this case it seems to me it is the right thing anyway to support somewhat older versions, but it is in the end up to you.
In either case, Rtools has 2.7.1, but it may really take a day or something before it gets updated on winbuilder.
Alright. Last weird thing that came up before submitting: https://artifacts.r-hub.io/nloptr_2.0.0.tar.gz-6614550c3a57464a9c779426f4a4c309/nloptr.Rcheck/
This says that on R-Hub windows 2022 r devel, your patch of Makevars.ucrt could not be applied. This makes me wonder: should we still use your patch since I updated Makevars.ucrt on my own?
Yes, the patch will have to be removed (it was described in an email sent to maintainers, but probably it was the previous maintainer). Are you ready to submit the package now? I can remove the patch, but even if it didn't propagate fast enough, that'd be fine, Uwe will help.
Yes I am submitting now.
Thanks, I've removed the patch from the repository.
Thanks to you, for your help and insightful discussion! I hope that submission will be accepted. For the sake of so many packages.
Thanks for maintaining the package. I also hope it will go through now, but if not, I can indeed move the patch back again. And in general, don't hesitate responding to the CRAN email if you didn't understand precisely what should be fixed/how/etc.