Pandoc step hanging with knitrBoostrap 1.0.0 (jimhester/knitrBootstrap@76c41f0)
lcolladotor opened this issue · comments
Hi,
I'm asking here because you might know what the problem is. I am re-running an analysis I did before and for some reason it's taking much longer: basically pandoc hangs.
In particular, one case used to run in 5 hours and 32 mins (5 hrs 19 mins were spent on R analysis, the rest on the pandoc step). It now has hanged for 5 days (killed it), hanged a 2nd time (killed the process again), and is hanging for a third time (24 hrs already).
Currently, I'm using:
> library(knitrBootstrap)
> library(rmarkdown)
> library(knitcitations)
> devtools::session_info()
Session info-------------------------------------------------------------------
setting value
version R version 3.1.1 Patched (2014-10-16 r66782)
system x86_64, linux-gnu
ui X11
language (EN)
collate en_US.UTF-8
tz <NA>
Packages-----------------------------------------------------------------------
package * version date source
bibtex 0.3.6 2013-07-29 CRAN (R 3.1.1)
colorout * 1.0.2 2014-05-30 local
devtools 1.6.1 2014-10-07 CRAN (R 3.1.1)
digest 0.6.4 2013-12-03 CRAN (R 3.1.0)
evaluate 0.5.5 2014-04-29 CRAN (R 3.1.0)
formatR 1.0 2014-08-25 CRAN (R 3.1.1)
htmltools 0.2.6 2014-09-08 CRAN (R 3.1.1)
httr 0.5 2014-09-02 CRAN (R 3.1.1)
knitcitations * 1.0.2 2014-10-15 Github (cboettig/knitcitations@c7c5909)
knitr 1.7 2014-10-13 CRAN (R 3.1.1)
knitrBootstrap * 1.0.0 2014-10-15 Github (jimhester/knitrBootstrap@76c41f0)
lubridate 1.3.3 2013-12-31 CRAN (R 3.1.1)
markdown 0.7.4 2014-08-24 CRAN (R 3.1.1)
memoise 0.2.1 2014-04-22 CRAN (R 3.1.0)
plyr 1.8.1 2014-02-26 CRAN (R 3.1.0)
Rcpp 0.11.3 2014-09-29 CRAN (R 3.1.1)
RCurl 1.95.4.3 2014-07-29 CRAN (R 3.1.1)
RefManageR 0.8.34 2014-08-19 CRAN (R 3.1.1)
RJSONIO 1.3.0 2014-07-28 CRAN (R 3.1.1)
rmarkdown * 0.3.3 2014-09-17 CRAN (R 3.1.1)
rstudioapi 0.1 2014-03-27 CRAN (R 3.1.1)
stringr 0.6.2 2012-12-06 CRAN (R 3.1.0)
XML 3.98.1.1 2013-06-20 CRAN (R 3.1.0)$ pandoc --version
pandoc 1.12.3.1
Compiled with texmath 0.6.6, highlighting-kate 0.5.6.
Syntax highlighting is supported for the following languages:
actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
lua, makefile, mandoc, markdown, matlab, maxima, metafont, mips, modelines,
modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml,
octave, pascal, perl, php, pike, postscript, prolog, python, r,
relaxngcompact, restructuredtext, rhtml, roff, ruby, rust, scala, scheme,
sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl,
xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /home/bst/student/lcollado/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web: http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.The process has been stuck at:
/jhpce/shared/jhpce/core/JHPCE_tools/1.0/bin/pandoc basicExploration.utf8.md --to html --from markdown-hard_line_breaks+superscript+tex_math_dollars+raw_html+markdown_in_html_blocks-implicit_figures --output basicExploration.html -H /scratch/temp/2732570.1.shared.q/RtmpWjnxOa/knitr_bootstrap_full.html
I suspect that updating pandoc can help, but well, this used to work with the same version. Previously something similar got resolved by fixing a piece of rmarkdown #47 and I don't know if it has any relation to rstudio/rmarkdown#31.
The previous output of devtools::sessionInfo() is:
## setting value
## version R version 3.1.1 Patched (2014-10-05 r66720)
## system x86_64, linux-gnu
## ui X11
## language (EN)
## collate en_US.UTF-8
## tz
## package * version date
## acepack 1.3.3.3 2013-05-03
## AnnotationDbi * 1.27.19 2014-10-07
## base64enc 0.1.2 2014-06-26
## BatchJobs 1.4 2014-09-24
## BBmisc 1.7 2014-06-21
## bibtex 0.3.6 2013-07-29
## Biobase * 2.25.0 2014-04-26
## BiocGenerics * 0.11.5 2014-09-17
## BiocParallel 0.99.25 2014-10-03
## biomaRt 2.21.5 2014-10-08
## Biostrings 2.33.14 2014-09-09
## biovizBase * 1.13.11 2014-09-17
## bitops 1.0.6 2013-08-17
## brew 1.0.6 2011-04-13
## BSgenome 1.33.9 2014-09-13
## bumphunter 1.5.5 2014-09-09
## Cairo 1.5.6 2014-06-26
## checkmate 1.4 2014-09-03
## cluster 1.15.3 2014-09-04
## codetools 0.2.9 2014-08-21
## colorspace 1.2.4 2013-09-30
## DBI 0.3.1 2014-09-24
## derfinder * 0.99.5 2014-10-07
## derfinderHelper 0.99.2 2014-10-07
## derfinderPlot 0.99.5 2014-10-07
## devtools 1.6 2014-09-19
## dichromat 2.0.0 2013-01-24
## digest 0.6.4 2013-12-03
## doRNG 1.6 2014-03-07
## evaluate 0.5.5 2014-04-29
## fail 1.2 2013-09-19
## foreach 1.4.2 2014-04-11
## foreign 0.8.61 2014-03-28
## formatR 1.0 2014-08-25
## Formula 1.1.2 2014-07-13
## futile.logger 1.3.7 2014-01-25
## futile.options 1.0.0 2010-04-06
## GenomeInfoDb * 1.1.25 2014-10-07
## GenomicAlignments 1.1.30 2014-10-07
## GenomicFeatures * 1.17.19 2014-10-07
## GenomicFiles 1.1.19 2014-09-02
## GenomicRanges * 1.17.43 2014-10-07
## GGally 0.4.8 2014-08-26
## ggbio 1.13.13 2014-09-17
## ggplot2 * 1.0.0 2014-05-21
## graph 1.43.0 2014-04-26
## gridExtra * 0.9.1 2012-08-09
## gtable 0.1.2 2012-12-05
## Hmisc 3.14.5 2014-09-12
## htmltools 0.2.6 2014-09-08
## httr 0.5 2014-09-02
## IRanges * 1.99.30 2014-10-07
## iterators 1.0.7 2014-04-11
## knitcitations 1.0.2 2014-10-07
## knitr * 1.6.18 2014-09-19
## knitrBootstrap 0.9.0 2013-10-17
## labeling 0.3 2014-08-23
## lambda.r 1.1.6 2014-01-23
## lattice 0.20.29 2014-04-04
## latticeExtra 0.6.26 2013-08-15
## locfit 1.5.9.1 2013-04-20
## lubridate 1.3.3 2013-12-31
## markdown 0.7.4 2014-08-24
## MASS 7.3.35 2014-09-30
## Matrix 1.1.4 2014-06-15
## matrixStats 0.10.0 2014-06-08
## memoise 0.2.1 2014-04-22
## mgcv * 1.8.3 2014-08-29
## munsell 0.4.2 2013-07-11
## nlme * 3.1.118 2014-10-07
## nnet 7.3.8 2014-03-28
## OrganismDbi 1.7.5 2014-09-09
## pkgmaker 0.22 2014-05-14
## plyr 1.8.1 2014-02-26
## proto 0.3.10 2012-12-22
## qvalue 1.39.1 2014-05-30
## RBGL 1.41.1 2014-08-19
## RColorBrewer * 1.0.5 2011-06-17
## Rcpp 0.11.3 2014-09-29
## RCurl 1.95.4.3 2014-07-29
## RefManageR 0.8.34 2014-08-19
## regionReport * 0.99.3 2014-10-07
## registry 0.2 2012-01-24
## reshape 0.8.5 2014-04-23
## reshape2 1.4 2014-04-23
## RJSONIO 1.3.0 2014-07-28
## rmarkdown 0.3.3 2014-09-19
## R.methodsS3 1.6.1 2014-01-05
## rngtools 1.2.4 2014-03-06
## rpart 4.1.8 2014-03-28
## Rsamtools 1.17.34 2014-10-07
## RSQLite 0.11.4 2013-05-26
## rstudioapi 0.1 2014-03-27
## rtracklayer 1.25.19 2014-10-07
## S4Vectors * 0.2.4 2014-09-15
## scales 0.2.4 2014-04-22
## sendmailR 1.2.1 2014-09-21
## stringr 0.6.2 2012-12-06
## survival 2.37.7 2014-01-22
## TxDb.Hsapiens.UCSC.hg19.knownGene * 3.0.0 2014-10-07
## VariantAnnotation 1.11.35 2014-10-07
## XML 3.98.1.1 2013-06-20
## xtable 1.7.4 2014-09-12
## XVector * 0.5.8 2014-09-07
## zlibbioc 1.11.1 2014-04-26
## source
## CRAN (R 3.1.1)
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## Bioconductor
## Bioconductor
## Bioconductor
## Bioconductor
## Bioconductor
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## Bioconductor
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Github (lcolladotor/derfinder@da289c4)
## Github (lcolladotor/derfinderHelper@54a9721)
## Github (lcolladotor/derfinderPlot@08d7f07)
## Github (hadley/devtools@45a6ac0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## Bioconductor
## Bioconductor
## Bioconductor
## Bioconductor
## Bioconductor
## CRAN (R 3.1.1)
## Bioconductor
## CRAN (R 3.1.0)
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## Bioconductor
## CRAN (R 3.1.0)
## Github (cboettig/knitcitations@c7c5909)
## local
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## Bioconductor
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## CRAN (R 3.1.1)
## Github (lcolladotor/regionReport@204bf40)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Github (rstudio/rmarkdown@fd8dafd)
## CRAN (R 3.1.0)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Bioconductor
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Bioconductor
## Bioconductor
## CRAN (R 3.1.0)
## CRAN (R 3.1.1)
## Bioconductor
## Bioconductor
Notice that rmarkdown GitHub version fd8dafd is the same as version 0.3.3 (rstudio/rmarkdown@fd8dafd). The main difference is the knitrBootstrap version: basically I am re-running the analysis because I want to use the newer knitrBootstrap (not yet using the newest from #68)
You can find the Rmd file I'm using here which is called from regionReport::derfinderReport() defined here.
Cheers,
Leo
I don't know for sure what the problem is, although if I were to guess I would say that pandoc is incorrectly parsing some portion of the markdown because of a missing newline after one of the html blocks. In order to track down the problem I would inspect the .md file produced and try to produce a minimum working example, your report is too large (and takes too long to run) to figure out what is going wrong in a reasonable time frame.
I could reproduce this until recently when:
- I installed
knitr1.7.8 (might have been related to the knitr::kable() issue #70) - Reduced the size of an interactive DataTable to 500 entries instead of 2000.
- Have other jobs on the cluster using $TMPDIR which helps in terms of disk/network usage.
My guess is that either of the first 2 things helped. I'll close it for now. Hopefully I won't run into this problem again.
I should add that pandoc step still took 20 mins. But that's reasonable.