pandocomatic hangs on input for only some workflows

Question

pandocomatic hangs on input for only some workflows

ppenguin opened this issue 2 years ago · comments

I have been using pandocomatic with different process definitions successfully, but suddenly (OS reinstall) one of my workflows doesn't work anymore: pandoc hangs when executing it:

pandocomatic -b -c ~/.pandoc/pandocomatic/pandocomatic.yaml -o test.pdf test.md

If I Ctrl-C I see ¹.

If I only change to: pdf to to: latex in pandocomatic.yaml, the conversion to test.tex works without problems, and after that

lualatex test.tex

also produces test.pdf without problems.

If I copy the generated pandoc statement shown by pandocomatic on the cli and add test.md (i.e. the input) it also correctly generates test.pdf:

pandoc  --from=markdown \
        --to=latex \
        --pdf-engine=lualatex \
        --pdf-engine-opt=-shell-escape \
        --pdf-engine-opt=-interaction\=nonstopmode \
        --template=/home/jeroen/.pandoc/templates/1nnovatio/1nnoquote.latex \
        --lua-filter=/home/jeroen/.nix-profile/share/pandoc/filters/include-files.lua \
        --filter=/home/jeroen/.nix-profile/bin/pandoc-xnos \
        --filter=/run/current-system/sw/bin/pandoc-crossref \
        --filter=/home/jeroen/.nix-profile/bin/pandoc-plantuml \
        --toc \
        --citeproc \
        --number-sections \
        --variable=mainfont\=Calibri \
        --variable=fontsize\=12pt \
        --variable=papersize\=A4 \
        --variable=graphics\=true \
        --variable=templatepath\=\~/.pandoc/templates/1nnovatio \
        --variable=geometry\=a4paper,bindingoffset\=0.3in,left\=1in,right\=1in,top\=1in,bottom\=1.2in,footskip\=.6in \
        --verbose \
        --output=/home/jeroen/test/test.pdf

(The above is the generated pandoc statement without the manually added input)

Output after Ctrl-C:

    18: from /home/jeroen/.nix-profile/bin/pandocomatic:18:in `<main>'
    17: from /home/jeroen/.nix-profile/bin/pandocomatic:18:in `load'
    16: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/bin/pandocomatic:3:in `<top (required)>'
    15: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/pandocomatic.rb:105:in `run'
    14: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:98:in `execute'
    13: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:98:in `each'
    12: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:99:in `block in execute'
    11: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/command.rb:129:in `execute'
    10: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:86:in `run'
     9: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:135:in `convert_file'
     8: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:184:in `pandoc'
     7: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:184:in `chdir'
     6: from /nix/store/j23dgnp728255l6x1riazkg10z0npll5-ruby2.7.5-pandocomatic-0.2.8/lib/ruby/gems/2.7.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:222:in `block in pandoc'
     5: from /nix/store/vsc42jayhzqavkcm0amc54q232icgf38-ruby2.7.5-paru-0.4.3/lib/ruby/gems/2.7.0/gems/paru-0.4.3/lib/paru/pandoc.rb:157:in `convert'
     4: from /nix/store/vsc42jayhzqavkcm0amc54q232icgf38-ruby2.7.5-paru-0.4.3/lib/ruby/gems/2.7.0/gems/paru-0.4.3/lib/paru/pandoc.rb:313:in `run_converter'
     3: from /nix/store/wd9i3c2dszf3bjd1vf5cnrw5vlbg4zin-ruby-2.7.5/lib/ruby/2.7.0/open3.rb:101:in `popen3'
     2: from /nix/store/wd9i3c2dszf3bjd1vf5cnrw5vlbg4zin-ruby-2.7.5/lib/ruby/2.7.0/open3.rb:219:in `popen_run'
     1: from /nix/store/vsc42jayhzqavkcm0amc54q232icgf38-ruby2.7.5-paru-0.4.3/lib/ruby/gems/2.7.0/gems/paru-0.4.3/lib/paru/pandoc.rb:316:in `block in run_converter' /nix/store/vsc42jayhzqavkcm0amc54q232icgf38-ruby2.7.5-paru-0.4.3/lib/ruby/gems/2.7.0/gems/paru-0.4.3/lib/paru/pandoc.rb:316:in `read': Interrupt

↩

Huub de Beer · Answer 1 · Thu Mar 17 2022 22:00:29 GMT+0800 (China Standard Time)

Odd that conversion to PDF via pandoc fails, where it succeeds manually. Can you try removing the "--pdf-engine-opt" options from your template and see what happens. And the "--filter"/"--lua-filter" options. And both. Looking at the command, those look the most likely to cause "hanging".

However, to investigate, I need to reproduce the issue. And for that, I need a minimal example where this occurs. Can you supply a minimal example where this occurs? And can you also tell me what version of pandoc you're using?

ppenguin · Answer 2 · Sat May 28 2022 06:04:10 GMT+0800 (China Standard Time)

@htdebeer sorry for the (significant) delay...

I managed to look into it, and found that (given a few necessary filters are switched on), the thing that actually makes the difference is whether a TOC is generated. For me pandocomatic hangs when toc is switched on, but I can still generate a correct PDF (with TOC) if I manually execute (the exact) pandoc command line with added input md file.

See https://github.com/ppenguin/test-pandocomatic. (This repo uses the pandoc template from https://github.com/Wandmalfarbe/pandoc-latex-template)

ppenguin · Answer 3 · Sat May 28 2022 06:05:34 GMT+0800 (China Standard Time)

Oh, some version info:

> pandoc --version
pandoc 2.17.1.1
Compiled with pandoc-types 1.22.2, texmath 0.12.4, skylighting 0.12.3,
citeproc 0.6.0.1, ipynb 0.2
User data directory: /home/jeroen/.pandoc
Copyright (C) 2006-2022 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.
> pandocomatic --version
Pandocomatic version 0.2.8
© 2014—2020 Huub de Beer <Huub@heerdebeer.org>

Pandocomatic is free software; pandocomatic is released under the GPLv3.

For more information about pandocomatic run 'pandocomatic --help' or read its
documentation at https://heerdebeer.org/Software/markdown/pandocomatic/.

Huub de Beer · Answer 4 · Sat May 28 2022 16:43:35 GMT+0800 (China Standard Time)

Thanks for your reply. I've tried your minimal example, but there's a lot of assets in the template I don't have access to. These are the items templatepath, pandoc-include, pandoc-crossref, diagram-generator.lua, bibliography, testbib.bib, and csl in your file test.md. When I remove these items by commenting them, pandocomatic works, although it takes 9.3 seconds.

Can you add the filters, a minimal version of your bibliography, and the CSL file to your repository with the minimal example? Then I might be able to reproduce the issue.

ppenguin · Answer 5 · Sat May 28 2022 20:36:22 GMT+0800 (China Standard Time)

@htdebeer thanks for your (very prompt!) reply. I committed an update, which should be self-contained. (pandoc-crossref was in there because I thought it played a role, but I could reproduce hanging even without any filters).

So now running

./md2pdf.sh test_minimal.md

produces test_minimal.pdf

and setting toc: true makes the script hang (on my system)

The thing that is not self-contained is the needed texlive packages for the template, that is because a much simpler template didn't cause a hang (but that might also be (in hindsight) because my simpler docs didn't contain a TOC to begin with...)

I included my shell.nix for information (or if you have nix you could use it to (almost trivially!) completely reproduce the environment) in which all texlive packages are listed (line 32-37) that are installed in my test environment.

Huub de Beer · Answer 6 · Sat May 28 2022 21:11:10 GMT+0800 (China Standard Time)

Thanks for your update. I've pulled the latest version of your minimal example. When I run pandocomatic test_minimal.pdf, with or without toc: true, I don't get any issues. Idem when I run your md2pdf.sh script. The variant with the table of contents takes about three seconds longer to convert, but that's to be expected.

I've also updated my pandoc to the version you have, but to no avail. It keeps on working :-(

Can you tell me which version of texlive and paru you're using? I'm at 2020 and 0.4.3, respectively.

However, I see that you're only using the "pandoc" part of the pandocomatic template. That gives us another approach to investigate your issue:

As an alternative, can you convert your minimal pandocomatic template to a pandoc "defaults" file and use that instead? That'd tell us if the issue occurs with pandocomatic or with pandoc somehow. Thus:

A defaults.yaml file with contents:

from: markdown
to: pdf
pdf-engine: lualatex
template: eisvogel.tex
metadata:
- papersize=A4
- graphics=true
- geometry=a4paper,bindingoffset=0.3in,left=1in,right=1in,top=1in,bottom=1.2in,footskip=.6in
toc: true
toc-depth: 2
number-sections: true
# filter:
#   - pandoc-crossref
# lua-filter:
#   - diagram-generator.lua
citeproc: true # !!! http://lierdakil.github.io/pandoc-crossref/#citeproc-and-pandoc-crossref
bibliography:
- ./bib/testbib.bib
csl: ./bib/computer.csl

Subsequently, remove the whole pandocomatic_ thing from test_minimal.md. And then run pandoc like:

pandoc --defaults defaults.yaml test_minimal.md -o test.pdf

ppenguin · Answer 7 · Sat May 28 2022 22:26:54 GMT+0800 (China Standard Time)

Texlive is from this base (which is a bit different in terms of versions than "traditional" linux due to the way nix(os) works), the relevant (?) parts:

> tex --version
TeX 3.141592653 (TeX Live 2021/NixOS.org)
kpathsea version 6.3.3

> lualatex
This is LuaHBTeX, Version 1.13.0 (Web2C 2021/NixOS.org)

I'll do the defaults thing you mentioned. (I have also invited you to the test repo in case that's easier, no pressure though ;)

ppenguin · Answer 8 · Sat May 28 2022 22:31:36 GMT+0800 (China Standard Time)

pandoc --defaults defaults.yaml test_minimal.md -o test.pdf

works perfectly with the provided defaults.yaml

Huub de Beer · Answer 9 · Sun May 29 2022 01:26:41 GMT+0800 (China Standard Time)

That does suggest the issue is with pandocomatic, somewhere. Unfortunately, I cannot reproduce it. I am not quite sure how to proceed.

ppenguin · Answer 10 · Sun May 29 2022 04:07:39 GMT+0800 (China Standard Time)

It indeed appears a bit strange.

Do I understand correctly that pandocomatic pipes into the pandoc process (invoked by the command line that we see due to -b), whereas when we manually execute that command with added input (or the pandoc command above for that matter), we read from an input file. I.e. the difference between working and not-working is pandoc reading from stdin in the latter case?

Though executed via pandoc over stdin (make tstpipe, last commit) it also doesn't hang for me...

Huub de Beer · Answer 11 · Sun May 29 2022 04:14:11 GMT+0800 (China Standard Time)

Depending on the configuration of the actual pandoc call, yes. The input document is pushed on STDIN/STDOUT through the preprocessors, then through pandoc, and then through the postprocessors.

So it is not quite clear why the --toc option, of all options, would cause the hang-up. Or why this behavior seems different on your machine than mine. If it would just crash, we'd have a stacktrace to work on, but now it just hangs. Did you try to have it run for, say, a couple of minutes?

ppenguin · Answer 12 · Sun May 29 2022 04:14:50 GMT+0800 (China Standard Time)

Short of getting into the nitty-gritty of the "interrupt-trace" resp. debugging I'm also not sure, but maybe an informed guess where to e.g. add debug prints or similar could help, or a more thorough inspection of the trace?

> make test_minimal.pdf
pandocomatic -b --data-dir=/home/jeroen/devel/github.com/ppenguin/test-pandocomatic -o test_minimal.pdf test_minimal.md && okular test_minimal.pdf
pandoc  --from=markdown \
        --to=latex \
        --pdf-engine=lualatex \
        --template=templates/eisvogel.tex \
        --verbose \
        --variable=papersize\=A4 \
        --variable=graphics\=true \
        --variable=geometry\=a4paper,bindingoffset\=0.3in,left\=1in,right\=1in,top\=1in,bottom\=1.2in,footskip\=.6in \
        --toc \
        --toc-depth=2 \
        --number-sections \
        --citeproc \
        --bibliography=bib/testbib.bib \
        --csl=bib/computer.csl \
        --output=/home/jeroen/devel/github.com/ppenguin/test-pandocomatic/test_minimal.pdf
^C/nix/store/bkidj9hsnal7s495pfarzifsa27rqx33-ruby3.1.2-paru-0.4.3/lib/ruby/gems/3.1.0/gems/paru-0.4.3/lib/paru/pandoc.rb:316:in `read': Interrupt
        from /nix/store/bkidj9hsnal7s495pfarzifsa27rqx33-ruby3.1.2-paru-0.4.3/lib/ruby/gems/3.1.0/gems/paru-0.4.3/lib/paru/pandoc.rb:316:in `block in run_converter'
        from /nix/store/9vc8grx7y86rly8dk3sk8w5f4v6vhyz3-ruby-3.1.2/lib/ruby/3.1.0/open3.rb:228:in `popen_run'
        from /nix/store/9vc8grx7y86rly8dk3sk8w5f4v6vhyz3-ruby-3.1.2/lib/ruby/3.1.0/open3.rb:103:in `popen3'
        from /nix/store/bkidj9hsnal7s495pfarzifsa27rqx33-ruby3.1.2-paru-0.4.3/lib/ruby/gems/3.1.0/gems/paru-0.4.3/lib/paru/pandoc.rb:313:in `run_converter'
        from /nix/store/bkidj9hsnal7s495pfarzifsa27rqx33-ruby3.1.2-paru-0.4.3/lib/ruby/gems/3.1.0/gems/paru-0.4.3/lib/paru/pandoc.rb:157:in `convert'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:222:in `block in pandoc'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:184:in `chdir'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:184:in `pandoc'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:135:in `convert_file'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_command.rb:86:in `run'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/command.rb:129:in `execute'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:99:in `block in execute'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:98:in `each'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/command/convert_file_multiple_command.rb:98:in `execute'
        from /nix/store/ajydkxvj9z03v92nhqqy961mrn33ydir-ruby3.1.2-pandocomatic-0.2.8/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/lib/pandocomatic/pandocomatic.rb:105:in `run'
        from /nix/store/xb702nf5fimh6lgyvk6w6sf7hrxqibbn-ruby-gems/lib/ruby/gems/3.1.0/gems/pandocomatic-0.2.8/bin/pandocomatic:3:in `<top (required)>'
        from /nix/store/xb702nf5fimh6lgyvk6w6sf7hrxqibbn-ruby-gems/bin/pandocomatic:18:in `load'
        from /nix/store/xb702nf5fimh6lgyvk6w6sf7hrxqibbn-ruby-gems/bin/pandocomatic:18:in `<main>'
make: *** [Makefile:16: test_minimal.pdf] Interrupt

ppenguin · Answer 13 · Sun May 29 2022 04:15:54 GMT+0800 (China Standard Time)

Did you try to have it run for, say, a couple of minutes?

Yes, I let it hang over dinner, seemed permanent ...

ppenguin · Answer 14 · Sun May 29 2022 04:27:10 GMT+0800 (China Standard Time)

... that just gave me an idea: I set verbose: false in test_minimal.md (since supposedly it will use an output stream, though that should be stderr and not affect things?) and ....

🥁 🥁 🥁

> make test_minimal.pdf
pandocomatic --data-dir=/home/jeroen/devel/github.com/ppenguin/test-pandocomatic -o test_minimal.pdf test_minimal.md && okular test_minimal.pdf
[WARNING] Citeproc: citation eq:eqn01 not found
[WARNING] Citeproc: citation fig:plantfig not found
Pandocomatic needed 7.9 seconds to convert '/home/jeroen/devel/github.com/ppenguin/test-pandocomatic/test_minimal.md' to 'test_minimal.pdf'.

I.e. it worked 🚀

EDIT: the -b flag (which I also removed for above reason) is safe though; it works the same with it.

Huub de Beer · Answer 15 · Sun May 29 2022 16:49:57 GMT+0800 (China Standard Time)

\o/

This actually makes sense! I don't think I've ever used the "verbose" option in pandoc, so I just assumed it was harmless. But if it uses the standard streams, it'll invalidate pandocomatic's assumptions about those streams.

I'll try to resolve this. Maybe by adding a warning or so to pandocomatic to warn about using verbose. Or use a different stream, somehow.

Huub de Beer · Answer 16 · Fri Aug 19 2022 19:29:55 GMT+0800 (China Standard Time)

I've decided to disable pandoc's --verbose option to resolve this issue. This will be available in pandocomatic's next version.

pandocomatic hangs on input for only some workflows

Footnotes