Recursive includes not working

Question

Recursive includes not working

ksebby opened this issue 5 years ago · comments

I have a master.md file which has !include statements for dep1.md and dep2.md. dep2.md also has !include statements. When I run

$ pandoc master.md --filter pandoc-include -o processed.md -t markdown -s

the include statements in dep2.md are not processed. Documentation says that recursive includes are supported since v0.3.1. I am on v0.3.2. Is there anything else needed to make recursive imports work?

DCsunset · Answer 1 · Wed Jul 10 2019 09:23:32 GMT+0800 (China Standard Time)

Hi, @ksebby

Could you post a simple piece of code to reproduce the problem?
I have tested locally and can't reproduce it.

Karl Sebby · Answer 2 · Wed Jul 10 2019 23:31:32 GMT+0800 (China Standard Time)

Thanks @DCsunset. I am running

pandoc docs/templates/Tools.md --filter pandoc-include -o docs/ToolsGuide.md -t markdown -s

in the root directory of this repo https://github.com/xDBio-Inc/cwl-tools/tree/combined_karl

Karl Sebby · Answer 3 · Sun Jul 14 2019 13:49:13 GMT+0800 (China Standard Time)

I got rid of the need for recursion in my docs as it wasn't really necessary. If you still want to play around with it you can git checkout 888943a from the main branch. Thanks.

DCsunset · Answer 4 · Sun Jul 14 2019 14:49:21 GMT+0800 (China Standard Time)

Sorry for missing your comment 4 days ago (because Github didn't send to me any notification).

It seems that the paths in the file are not processed correctly.
I may fix the bug later.

DCsunset · Answer 5 · Mon Jul 15 2019 10:32:53 GMT+0800 (China Standard Time)

The bug has been fixed in v0.4.0.
The usage of recursive includes has been updated in README.md.

Normen Müller · Answer 6 · Thu Apr 30 2020 00:41:28 GMT+0800 (China Standard Time)

@DCsunset It looks like this bug reappears in v0.7.0 ... at least on my Mac. I have one Main.md which includes xxx/Main.md which in turn includes xxx/1.md and xxx/2.md. Running pandoc Main.md --filter pandoc-include -o Main.pdf results in

Traceback (most recent call last):
  File "/Users/normenmueller/Library/Python/3.7/bin/pandoc-include", line 8, in <module>
    sys.exit(main())
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 160, in main
    return pf.run_filter(action, doc=doc)
  File "/usr/local/lib/python3.7/site-packages/panflute/io.py", line 258, in run_filter
    return run_filters([action], *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/panflute/io.py", line 239, in run_filters
    doc = doc.walk(action, doc)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 265, in walk
    ans = list(chain.from_iterable(ans))
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 263, in <genexpr>
    ans = ((item,) if type(item) != list else item for item in ans)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 260, in <genexpr>
    ans = (item.walk(action, doc) for item in obj)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 276, in walk
    altered = action(self, doc)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 108, in action
    fn + ' ' + entry + ' ' + os.getcwd())
TypeError: can only concatenate str (not "NoneType") to str
Error running filter pandoc-include:
Filter returned error status 1

Traceback (most recent call last):
  File "/Users/normenmueller/Library/Python/3.7/bin/pandoc-include", line 8, in <module>
    sys.exit(main())
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 160, in main
    return pf.run_filter(action, doc=doc)
  File "/usr/local/lib/python3.7/site-packages/panflute/io.py", line 258, in run_filter
    return run_filters([action], *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/panflute/io.py", line 239, in run_filters
    doc = doc.walk(action, doc)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 265, in walk
    ans = list(chain.from_iterable(ans))
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 263, in <genexpr>
    ans = ((item,) if type(item) != list else item for item in ans)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 260, in <genexpr>
    ans = (item.walk(action, doc) for item in obj)
  File "/usr/local/lib/python3.7/site-packages/panflute/base.py", line 276, in walk
    altered = action(self, doc)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 132, in action
    raw, extra_args=pandoc_options)
  File "/usr/local/lib/python3.7/site-packages/panflute/tools.py", line 413, in convert_text
    out = inner_convert_text(text, in_fmt, out_fmt, extra_args)
  File "/usr/local/lib/python3.7/site-packages/panflute/tools.py", line 436, in inner_convert_text
    out = run_pandoc(text, args)
  File "/usr/local/lib/python3.7/site-packages/panflute/tools.py", line 337, in run_pandoc
    raise IOError('')
OSError
Error running filter pandoc-include:
Filter returned error status 1

DCsunset · Answer 7 · Thu Apr 30 2020 12:14:05 GMT+0800 (China Standard Time)

@normenmueller I fixed some bugs and release v0.7.2. I hope it solves this problem.

Normen Müller · Answer 8 · Sun May 03 2020 15:28:21 GMT+0800 (China Standard Time)

@DCsunset Thanks for the fast reply and patch!

However, it still does not work... guess there is a problem with spaces in path names? I tried !include 10\ Anhang/Main.md but

pandoc Main.md --filter pandoc-include -o Main.pdf
Traceback (most recent call last):
  File "/Users/normenmueller/Library/Python/3.7/bin/pandoc-include", line 8, in <module>
    sys.exit(main())
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 161, in main
    return pf.run_filter(action, doc=doc)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/io.py", line 266, in run_filter
    return run_filters([action], *args, **kwargs)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/io.py", line 247, in run_filters
    doc = doc.walk(action, doc)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/base.py", line 265, in walk
    ans = list(chain.from_iterable(ans))
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/base.py", line 263, in <genexpr>
    ans = ((item,) if type(item) != list else item for item in ans)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/base.py", line 260, in <genexpr>
    ans = (item.walk(action, doc) for item in obj)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/panflute/base.py", line 276, in walk
    altered = action(self, doc)
  File "/Users/normenmueller/Library/Python/3.7/lib/python/site-packages/pandoc_include.py", line 108, in action
    '%r %r %r' % (fn, entry, os.getcwd()))
ValueError: Included file not found: '10\xa0Anhang/Main.md' None '/Users/normenmueller/Worspace/eam/ads'
Error running filter pandoc-include:
Filter returned error status 1

DCsunset · Answer 9 · Mon May 04 2020 10:58:35 GMT+0800 (China Standard Time)

The spaces needn't be escaped. You might remove the backslashes and try it again.

Normen Müller · Answer 10 · Tue May 05 2020 16:26:03 GMT+0800 (China Standard Time)

@DCsunset I guess the error is on my site, but !include 10 Anhang/Main.md also does not work. No error is thrown, though, but it just quits normally with a PDF but without the included content.

DCsunset · Answer 11 · Wed May 06 2020 15:24:31 GMT+0800 (China Standard Time)

@normenmueller Could you post the !include lines in your Main.md and xxx/Main.md?
If you add some options to pandoc-include, you could post it as well.
Then I could try to reproduce it on my machine.

Normen Müller · Answer 12 · Wed May 06 2020 17:43:38 GMT+0800 (China Standard Time)

/doc
 |- 01 Intro/
 |- ...
 |- 10 Annex/
     |- Main.md
     |- 1.md
     |- 2.md
 |- Main.md

Main.md

---
title: '...'
subtitle: '...'
author: Normen Müller
abstract: |
  tbd
...

!include 10 Annex/Main.md

/10 Annex/Main.md

# Bla Blub

!include ./1.md

!include ./2.md

Shell:

$ cd /doc
$ pandoc Main.md --filter pandoc-include -o Main.pdf

DCsunset · Answer 13 · Wed May 06 2020 19:45:43 GMT+0800 (China Standard Time)

There seems no problem when compiling on my computer.
Did you update pandoc-include to v0.7.2?

Normen Müller · Answer 14 · Thu May 07 2020 00:25:39 GMT+0800 (China Standard Time)

My mistake. If you rename the resulting PDF but open the old PDF... then the error is in front of the computer ;-)