url.exists() gives false negatives on Windows computers

[Michael McCarthy]

For some reason, url.exists() has different behaviour between OSX, Linux, and Windows, where using the function in Windows falsely returns FALSE values on URLs that do exist. For example:

RCurl::url.exists(“https://www.netlify.com”)
#> FALSE

But on OSX or Linux it returns the correct value:

RCurl::url.exists(“https://www.netlify.com”)
#> TRUE

I’ve tested this on the following:

• OSX Mojave (local)
• Windows 8.1 Pro (local)
• OSX latest release (GitHub Actions runner)
• Linux latest release (GitHub Actions runner)
• Windows latest release (GitHub Actions runner)

Some other users on Stackoverflow seem to have the same problem, and Windows seems to be the common denominator.

Do you have any solutions to this problem?

[Duncan Temple Lang]

Hi Michael
I have a conjecture, but let's get some data. Can you issue the commands
RCurl::curlVersion()
and
RCurl::url.exists("https://www.netlify.com", verbose = TRUE)
and show us the output from the two commands, please? Thanks

[Michael McCarthy]

Hi Duncan,

Here are the results from my local Windows and OSX computers. Both computers are using the most recent CRAN release of RCurl.

Windows

RCurl::curlVersion()
$age
[1] 3

$version
[1] "7.40.0"

$vesion_num
[1] 468992

$host
[1] "x86_64-pc-win32"

$features
      ssl      libz      ntlm asynchdns    spnego largefile       idn      sspi 
        4         8        16       128       256       512      1024      2048 

$ssl_version
[1] "OpenSSL/1.0.0o"

$ssl_version_num
[1] 0

$libz_version
[1] "1.2.8"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps" 
[10] "ldap"   "pop3"   "pop3s"  "rtmp"   "rtsp"   "scp"    "sftp"   "smtp"   "smtps" 
[19] "telnet" "tftp"  

$ares
[1] ""

$ares_num
[1] 0

$libidn
[1] ""

RCurl::url.exists("https://www.netlify.com", verbose = TRUE)
[1] FALSE

OSX

RCurl::curlVersion()
$age
[1] 3

$version
[1] "7.54.0"

$vesion_num
[1] 472576

$host
[1] "x86_64-apple-darwin18.0"

$features
     ipv6       ssl      libz      ntlm asynchdns    spnego largefile   ntlm_wb 
        1         4         8        16       128       256       512     32768 

$ssl_version
[1] "LibreSSL/2.6.5"

$ssl_version_num
[1] 0

$libz_version
[1] "1.2.11"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"  
[11] "ldaps"  "pop3"   "pop3s"  "rtsp"   "smb"    "smbs"   "smtp"   "smtps"  "telnet" "tftp"  

$ares
[1] ""

$ares_num
[1] 0

$libidn
[1] ""

RCurl::url.exists("https://www.netlify.com", verbose = TRUE)
* Rebuilt URL to: https://www.netlify.com/
*   Trying 104.248.78.23...
* TCP_NODELAY set
* Connected to www.netlify.com (104.248.78.23) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=ca; L=San Francisco; O=Netlify, Inc; CN=*.netlify.com
*  start date: Jul  3 00:00:00 2019 GMT
*  expire date: Jul  7 12:00:00 2020 GMT
*  subjectAltName: host "www.netlify.com" matched cert's "*.netlify.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
Host: www.netlify.com
Accept: */*

< HTTP/1.1 200 OK
< Cache-Control: public, max-age=0, must-revalidate
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 19 May 2020 23:48:39 GMT
< Etag: "c0763cb5850d3ed513a7cf0a69525e01-ssl"
< Link: <https://www.netlify.com/>; rel="canonical"
< Strict-Transport-Security: max-age=31536000
< Age: 8
< Content-Length: 253471
< Connection: keep-alive
< Server: Netlify
< Set-Cookie: nf_ab=0.981820; expires=Wed, 19-May-2021 23:48:47 GMT; path=/
< X-NF-Request-ID: 09e89e2f-7399-4283-acf5-0974a20789f6-14993968
< 
* Connection #0 to host www.netlify.com left intact
[1] TRUE