Error in $: Failed reading: not a valid json value

Question

Error in $: Failed reading: not a valid json value

PauloASilva opened this issue 5 years ago · comments

Hi,
Thank you for the work and effort put on pandoc-fignos development and maintenance.

I am trying to build a docker image on top of pandoc/latex.
After installing pandoc-fignos using the py package, every time I try to use it I get the following output

$ docker run --volume "`pwd`:/data" --user `id -u`:`id -g` my/pandoc-latex:latest --filter pandoc-fignos --from markdown -o README.pdf README.md
Error running filter pandoc-fignos:
Error in $: Failed reading: not a valid json value

I am not familiar with pandoc filters.
Can you please provide some guidance to solve this issue?

About my environment

pandoc

$ docker run -it --rm my/pandoc-latex:latest --version
pandoc 2.7.2
Compiled with pandoc-types 1.17.5.4, texmath 0.11.2.2, skylighting 0.7.7
Default user data directory: /root/.local/share/pandoc or /root/.pandoc
Copyright (C) 2006-2019 John MacFarlane
Web:  http://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

Python

$ docker run -it --rm --entrypoint /usr/bin/python my/pandoc-latex:latest --version
Python 2.7.16

Cheers,
Paulo A. Silva

Thomas J. Duck · Answer 1 · Tue Nov 05 2019 00:41:47 GMT+0800 (China Standard Time)

Hi, @PauloASilva. Thanks for your feedback. I am happy to try to help figure this out.

Can you please run pandoc-fignos --version and let me know what it reports?

Also, can you try to process your document outside of docker? I want to see if the problem is with pandoc-fignos or something else. If you could provide me with a minimal document and processing command that reproduces the problem, that would be great.

Thanks,
Tom

PauloASilva · Answer 2 · Tue Nov 05 2019 01:08:40 GMT+0800 (China Standard Time)

Hi @tomduck,
Thanks for your quick reply.

Regarding pandoc-fignos version

$ docker run -it --rm --entrypoint pandoc-fignos my/pandoc-latex:latest --version
pandoc-fignos 2.2.0

I'm running pandoc from within a docker container and trying to add pandoc-fignos to it to avoid polluting the system.
Would you agree to receive the Dockerfile to build the image, the minimal document, and the processing command to try to reproduce the problem?

Thanks,
Paulo A. Silva

Thomas J. Duck · Answer 3 · Tue Nov 05 2019 01:36:39 GMT+0800 (China Standard Time)

Thanks, @PauloASilva. I am not a docker user at present. We can probably work around this.

Below are a series of tests for you to try. I assume that they can be run within docker. You will want to use a minimal README.md for testing purposes.

Process the document using pandoc alone: pandoc README.md -o README.pdf. This should produce a valid pdf but without the cross-referencing.
Process with pandoc but write json to stdout instead: pandoc -t json README.md. This should complete without error.
Pipe the results of step 2 to pandoc-fignos: pandoc -t json README.md | pandoc-fignos. This should write json to stdout. If it fails, then please send me the output from step 2 which I can test.
Pipe the results of step 3 to pandoc: pandoc -t json README.md | pandoc-fignos json | pandoc - -f json -o README.pdf. If step 3 completed, then this should produce a valid pdf with the cross-referencing.

I would expect a failure in one of these steps.

Cheers,
Tom

PauloASilva · Answer 4 · Tue Nov 05 2019 17:11:19 GMT+0800 (China Standard Time)

Hi @tomduck,
Thanks for the detailed step-by-step guidance

1. Process the document using pandoc alone: `pandoc README.md -o README.pdf`.  This should produce a valid pdf but without the cross-referencing.

OK

2. Process with pandoc but write json to stdout instead: `pandoc -t json README.md`.  This should complete without error.

OK (valid JSON)

3. Pipe the results of step 2 to pandoc-fignos: `pandoc -t json README.md | pandoc-fignos`.  This should write json to stdout.  If it fails, then please send me the output from step 2 which I can test.

NOK

$ pandoc -t json TEST.md | pandoc-fignos 
usage: pandoc-fignos [-h] [--version] [--pandocversion PANDOCVERSION] fmt
pandoc-fignos: error: too few arguments

Providing the mandatory fmt to pandoc-fignos I get a valid JSON:

$ pandoc -t json TEST.md | pandoc-fignos json
{"meta": {}, "pandoc-api-version": [1, 17, 5, 4], "blocks": [{"c": [{"c": "See", "t": "Str"}, {"t": "Space"}, {"c": [["", [], []], [{"c": "1", "t": "Str"}], ["#fig:1", ""]], "t": "Link"}], "t": "Para"}, {"c": [{"c": [["fig:1", [], []], [{"c": "Figure", "t": "Str"}, {"c": [], "t": "Space"}, {"c": "1:", "t": "Str"}, {"c": [], "t": "Space"}, {"c": "This", "t": "Str"}, {"t": "Space"}, {"c": "is", "t": "Str"}, {"t": "Space"}, {"c": "fig", "t": "Str"}, {"t": "Space"}, {"c": "1", "t": "Str"}], ["./some-image.png", "fig:"]], "t": "Image"}], "t": "Para"}]}

4. Pipe the results of step 3 to pandoc: `pandoc -t json README.md | pandoc-fignos json | pandoc - -f json -o README.pdf`.  If step 3 completed, then this should produce a valid pdf with the cross-referencing.

Considering the change made to the given command (passing json as pandoc-fignos fmt) I get a PDF with the desired reference, nevertheless the figure's label now has a duplicate prefix.

This is my markdown

See {@fig:1}

![This is fig 1][fig-1]{#fig:1}

[fig-1]: ./assets/screenshots/rate-limiting-1.png

Running

$ pandoc -t json TEST.md | pandoc-fignos json | pandoc - -f json -o TEST.pdf

I get the PDF but figure's label is: "Figure 1: Figure 1: This is fig 1"
Without figures references labels are ok: no duplicate prefix.

Thanks,
Paulo A. Silva

PauloASilva · Answer 5 · Tue Nov 05 2019 17:28:53 GMT+0800 (China Standard Time)

I tried to build a PDF from my original markdown file, following your step-by-step list.

Running the following command outputs a PDF with references (still with the duplicate prefix but this is another issue)

$ pandoc -t json README.md | pandoc-fignos json | pandoc - -f json -o README.pdf

I tried to use pandoc's --filter, but then I get the "not a valid json value" error as below

$ pandoc --filter pandoc-fignos -o README.pdf README.md
Error running filter pandoc-fignos:
Error in $: Failed reading: not a valid json value

Running the same command for the very simple markdown file used for testing purposes it succeeds.

$ pandoc --filter pandoc-fignos -o TEST.pdf TEST.md

It sounds that there's something in my README.md file breaking pandoc-fignos but I does it work with pipes?

Cheers,
Paulo A. Silva

Thomas J. Duck · Answer 6 · Tue Nov 05 2019 20:44:07 GMT+0800 (China Standard Time)

Hi, @PauloASilva. Sorry, the fmt argument to pandoc-fignos should be latex. With this, there should be no difference running with pipes or --filter. --Tom

PauloASilva · Answer 7 · Tue Nov 05 2019 21:31:14 GMT+0800 (China Standard Time)

@tomduck I am a bit embarrassed since I was able to build the PDF from my original markdown file without any issues. Even the duplicate prefix "Figure #" is gone 🤔

$ docker run --rm --volume "`pwd`:/data" --user `id -u`:`id -g` my/pandoc-latex:latest --filter pandoc-fignos --from markdown -o README.pdf README.md

In fact removed all figures references from markdown and did all of them again. I swear I did it before opening this issue.
Any chance the JSON error being cause by figures references syntax errors in markdown?

I am very sorry for taking your time with this.

Best,
Paulo A. Silva

Thomas J. Duck · Answer 8 · Tue Nov 05 2019 22:07:46 GMT+0800 (China Standard Time)

No problem at all, @PauloASilva. The filter is by now pretty robust. I use it on some very large documents. But every now and then someone comes up with a use case that I didn't expect. I don't like crashes, and try to have the filter exit gracefully. So, if you come across a similar problem again, please don't hesitate to post an Issue. ☺︎

Cheers,
Tom

Brandon Williams · Answer 9 · Wed Feb 26 2020 06:36:36 GMT+0800 (China Standard Time)

Hi @tomduck, I think I'm having a similar problem, and I'm not sure if it's because I'm using a newer version of pandoc than what is supported (I'm on 2.9.2).

Basically if I run this document:

![Caption.](images/image.png){#fig:id}

See +@fig:id

with this command:

pandoc ch01.md --filter pandoc-fignos -o ch01.pdf

I get the error:

Error running filter pandoc-fignos:
Error in $: Failed reading: not a valid json value

This only seems to happen if I'm using some of the fancier features of fignos. Like if I drop the + it will work:

![Caption.](images/image.png){#fig:id}

See @fig:id

But then if I use the fignos-cleveref metadata value it will begin failing again.

Do you think this has to do with the newer version of pandoc?

Thanks for any help, and thanks for the wonderful library 🙂

Thomas J. Duck · Answer 10 · Sat Aug 01 2020 20:17:57 GMT+0800 (China Standard Time)

Hi @mbrandonw. Sorry, I missed your comment given that this Issue had been closed. Similar Issues were reported to pandoc-eqnos (tomduck/pandoc-eqnos#45) and pandoc-tablenos (tomduck/pandoc-tablenos#19). They have all now been resolved in the most recent releases. Cheers, Tom.