This LUA filter stopped working with the format `commonmark_x`, and is not displayed correctly with the format `markdown_github` on Codebraid
gusbemacbe opened this issue · comments
Hello!
- I use the latest version of @jgm's Pandoc;
- I use the latest version of @gpoore's Codebraid and Codebraid Preview;
- I use mostly the format
markdown_githubbecause of supported features.
Here is a small YAML default file code:
filters:
- assets/filters/lua/list-table.lua
template: "assets/templates/confluence"
variables:
confluence_html: true
languages:
- english
- español
- français
- portuguêsAnd VSCode's Codebraid settings:
{
"codebraid.preview.css.overrideDefault": true,
"codebraid.preview.css.useDefault": true,
"codebraid.preview.css.useMarkdownPreviewFontSettings": true,
"codebraid.preview.pandoc.defaultsFile": "confluence.yaml",
"codebraid.preview.pandoc.executable": "pandoc",
"codebraid.preview.pandoc.showRaw": true,
"codebraid.preview.pandoc.showStderr": "never",
"codebraid.preview.security.allowEmbeddedScripts": true,
"codebraid.preview.security.allowInlineScripts": true,
"codebraid.preview.security.allowLocalScripts": true,
"codebraid.preview.security.allowRemoteFonts": true,
"codebraid.preview.security.allowRemoteImages": true,
"codebraid.preview.security.allowRemoteMedia": true,
"codebraid.preview.security.allowRemoteScripts": true,
"codebraid.preview.security.allowRemoteStyles": true,
"codebraid.preview.scrollEditorWithPreview": true,
"codebraid.preview.scrollPreviewWithEditor": true,
"codebraid.preview.pandoc.build":
{
"*.md":
{
"reader": "markdown_github",
"preview":
{
"html":
{
"defaults": {},
"options":
[
"--preserve-tabs",
"--standalone",
"--wrap=auto",
]
}
},
"export": {}
},
},
}Here is a small Pandoc template code:
$if(confluence_html)$
<html $if(lang)$lang="$lang$" $endif$>
<head>
<meta charset="UTF-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>$if(title-prefix)$$title-prefix$ - $endif$$pagetitle$</title>
</head>
<body>
$endif$
<div class="ids-markdown">
Testing...
$body$
</div>
$if(confluence_html)$
</body>
</html>
$endif$I tested the simple list-table code in a Markdown file:
# Lorem ipsum dolor sit amet consectetur adipisicing elit. Fugit, minus?
Lorem ipsum dolor, sit amet consectetur adipisicing elit. Vero, adipisci sequi? Maiores autem quaerat praesentium?
:::list-table
* - row 1, column 1
- row 1, column 2
- row 1, column 3
* - row 2, column 1
-
- row 2, column 3
* - row 3, column 1
- row 3, column 2
:::I tested with the markdown_github format and here is the result. It failed to render the code:
I switched to the format commonmark_x and tested it. Here is the error:
In the commonmark_x case, the issue is that the Lua filter isn't compatible with the divs that Codebraid Preview inserts into the abstract syntax tree to provide scroll sync. Scroll sync for commonmark_x is different than for all other formats, because Pandoc provides source location information instead of Codebraid Preview having to attempt to reconstruct it. There's a note in the README about adapting filters. Basically, they need to skip elements with a data-pos attribute.
In the markdown_github case, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format as markdown_github+fenced_divs.
In the
markdown_githubcase, it appears that the issue is that the format doesn't automatically support fenced divs. So you need to specify the input format asmarkdown_github+fenced_divs.
I adjusted the confluence.yaml, updating from to markdown_github+fenced_divs, but Codebraid Preview informed the failure. The problem is probably in the Lua list-table code.
It seems to be working fine for me. I'd suggest checking the Markdown document between the sh code block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the * in the list are indented too far.
It seems to be working fine for me. I'd suggest checking the Markdown document between the
shcode block and whatever follows. For some reason, this is being treated as code instead of regular Markdown. There might be some sort of syntax error in the document. Maybe the*in the list are indented too far.
- I removed the
shpart; - I fixed the
*indentation;
But still the same error. I investigated the table indentation. I fixed, using VSCode's Ctrl + [, and it worked now:
:::list-table
Markup languages
* - Name
- Initial release
* - Markdown
- 2004
* - reStructuredText
- 2002
:::in comparison to the buggy code copied from this repository's README.md:
:::list-table
Markup languages
* - Name
- Initial release
* - Markdown
- 2004
* - reStructuredText
- 2002
:::Only the complicated tables do not work.
Ah, I reviewed my old Codebraid projects, I copied all the missing features for markdown_github variant format and all the complicated tables worked now.
The @pandoc-ext personnel to document the required features for any list-table table to work. They need to fix the table indentation bug.
Here are the missing features:
from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum
Please could you supply the full command line that shows the problem with the example below? Thanks.
With 3 indentation spaces:
- Using the
markdown_githubwith enabled on Codebraid Preview, it failed. - Using the full command
pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum --to html "README.md", it also failed.
Switching to commonmark_x.
- Using the Codebraid Preview, and disabling the scroll settings, it failed.
- Using the full command
pandoc --preserve-tabs --standalone --wrap=auto --defaults "confluence.yaml" --from commonmark_x+bracketed_spans+fenced_divs+raw_attribute+raw_html --to html "README.md" -o "result.html", it worked.
Thanks for that @gusbemacbe (I realize that I should have been able to extract this info from the existing discussion; sorry about that).
I think @gpoore's analysis is spot on, and that the main thing is that you need to enable the fenced_divs extension. With this input:
% cat spiz.md
:::list-table
- - zero-indented
:::
:::list-table
- - three-indented
:::
...this works for me:
% pandoc spiz.md -f markdown_github+fenced_divs -L list-table.lua
[WARNING] Deprecated: markdown_github. Use gfm instead.
<table>
<thead>
<tr class="header">
<th>zero-indented</th>
</tr>
</thead>
</table>
<table>
<thead>
<tr class="header">
<th>three-indented</th>
</tr>
</thead>
</table>
So why do you get the original codebraid error? I suspect (from the output shown below) that the indentation has somehow been changed. Is it possible to show precisely the markdown that's being passed to pandoc?
As for possible updates to the filter, I think that we could/should emphasise (in the documentation) that the fenced_div extension has to be enabled. Also, I suppose that if any of the supported pandoc markdown variants don't support 3-space indentation (are there any that don't?) then we could remove it from the examples. I also think that the filter should be adapted to work with codebraid (I can create a separate issue for that).
Here is the full command given by Codebraid:
pandoc "README.md"
--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css" \
--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css" \
--css="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css" \
--preserve-tabs --standalone --wrap=auto \
--defaults "$HOME/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" \
--lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" \
--lua-filter="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" \
--katex="$HOME/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/" \
--from markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum \
--to htmlIn the case of defaults, replace the path with where the defaults file is located. Copy my defaults code:
from: markdown_github+auto_identifiers+bracketed_spans+fenced_code_attributes+fenced_divs+inline_code_attributes+markdown_in_html_blocks+raw_attribute+raw_html+simple_tables+startnum
default-image-extension: '.png'
extract-media: dir
metadata:
author: "Gustavo Costa"
date: '2023-12-29'
keywords:
- codebraid
- css
- guide
- html
- markdown
- markdown_mmd
- multimarkdown
- pandoc
lang: "pt-PT"
title: "Codebraid Pandoc Markdowm Guide"
toc-title: "Table of Guides"
version: "1.0.0"
preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2
trace: true
track-changes: accept
# General writer options
filters:
- "list-table.lua"
highlight-style: kate
no-check-certificate: true
request-headers:
- ["User-Agent", "Mozilla/5.0"]
strip-comments: false
toc: true
toc-depth: 7
variables:
confluence_html: true
languages:
- english
- español
- français
- português
# Options affecting specific writers
ascii: false
email-obfuscation: references
html-q-tags: true
incremental: true
listings: true
markdown-headings: atx
number-sections: true
reference-links: true
reference-location: block
section-divs: true
self-contained: false
slide-level: 2
top-level-division: chapter
# Citation rendering
cite-method: biblatex
citeproc: true
# Math rendering in HTML
html-math-method:
method: katexAfter some investigation, I think that the problem is that the tab stop has been set to 2. Consider this example:
% pandoc <<<' two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>
Compare with this:
% pandoc <<<' two-indent' --tab-stop=4
<p>two-indent</p>
The CommonMark parser doesn't behave this way:
% pandoc <<<' two-indent' --tab-stop=2 --from commonmark
<p>two-indent</p>
I don't see any suggestion in the manual that --tab-stop should affect indented code block indentation, so is this a bug in the markdown reader?
I made another test, but different, with both --tab-stop=2 and --tab-size=4, on Codebraid Preview and using markdown_github and other variants:
* testing...
:::list-table
* - Name
- Description
- Origin
* - Apple
- Fruit
- China
* - Banana
- Fruit
- India
:::It worked with all the variants, except CommonMark, CommonMark X and gfm, on Codebraid Preview.
If I tab-indented the table part, it stopped working with all the variants, including CommonMark and CommonMark X, on Codebraid Preview.
@jgm, I'm not sure whether you saw this. Would you expect this behavior (see the last-but-one comment for more context)? Thanks.
% pandoc <<<' two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>
Yes, that is expected.
John Gruber's original Markdown documentation says:
To produce a code block in Markdown, simply indent every line of the block by at least 4 spaces or 1 tab.
IF we change a tab from 4 to 2 spaces, then 2 spaces will do it.
Thanks. I'd expect this if there was a tab in the input, but there are two spaces, not a tab. FWIW the CommonMark parser doesn't treat this as a code block.
commonmark just hard-codes 4 spaces tab stop, which I think is the best plan.
--tab-stop is sort of a vestigial feature -- I doubt many people use it. What are you using it for?
I do not know who you are addressing to, @jgm.
As for me, to work on Codebraid Preview, I would prefer using markdown_github and any variant, except commonmark_x and gfm.
Without depending on the Pandoc Lua filter list-table, any Markdown variant works on both Codebraid Preview and the terminal.
With the dependence of the Pandoc Lua filter list-table, and with 2 or 4 tab stop spaces, commonmark_x and gfm variants do not work on Codebraid Preview, but work on the terminal. The markdown_github, markdown_mmd and markdown_phpextra variants work on Codebraid Preview and on the terminal, but only if with the 2 tab stop spaces, and with the fenced_divs and simple_tables are enabled.
However, markdown_github has more features than commonmark_x and gfm, but I reconsider switching to markdown_mmd and markdown_phpextra, but I want to stop Pandoc from wrapping a p element in every line. @mb21 recommends me to use gfm, that stops Pandoc from wrapping p element in every line, but gfm will not work on Codebraid Preview with the dependence of the Pandoc Lua filter list-table.
Well. I have tabe-stop in the defaults file:
preserve-tabs: true
shift-heading-level-by: -1
tab-stop: 2Do you think that defaults file's tab-stop can be the cause of these bugs?
I definitely think you should try removing it, or changing it to 4!
@jgm, sorry for dragging you into this discussion. I'll create a pandoc issue to discuss the point that I raised with you.
Good evening, @wlupton
I have just made this test:
- With the removal of
tab-stopfrom thedefaultsfile or with the modification oftab-stop's value to4, it worked on Codebraid Preview with the Markdown variantmarkdown_github, but not with the other variantscommommark_xandgfm. I think it turns to be the bug oftab-stop.
However, I want to keep 2 tab stop spaces for the HTML elements when converting from Markdown to HTML. But it will not work with this Pandoc Lua filter on Codebraid Preview.
I hoped that this example from a couple of days ago would make it clear what's going on here. With a tab-stop of 2, indentation of 2 or more will be formatted as code.
% pandoc <<<' two-indent' --tab-stop=2
<pre><code>two-indent</code></pre>
If you want to use tab-stop=2 then you'll need to change your input file, because 3 > 2! BUT, are you sure that the tab-stop setting affects the generated HTML? I don't see why it would do!
I suggest using very simple input and defaults files (in fact, avoid using a defaults file if you can), and also making sure that it works as expected from the command-line before trying it with Codebraid.
Perhaps this example will make it clearer.
% cat spiz.md
:::list-table
* - 3
:::
:::list-table
* - 1
:::
% pandoc spiz.md --tab-stop=2
<div class="list-table">
<pre><code> * - 3</code></pre>
</div>
<div class="list-table">
<ul>
<li><ul>
<li>1</li>
</ul></li>
</ul>
</div>
I'm not using the list-table.lua filter here because I don't need to! It will fail, because it'll see a code block where it expects a list.
Let's assume that you have resolved the tab-stop problem, either by using tab-stop=4 (which is what I would recommend) or by editing the markdown.
Now you can try it with codebraid. You'll have (for now) to live with the restrictions explained here by @gpoore, which mean that it won't work with commonmark variants (commonmark, gfm and commonmark_x).
As I mentioned, I'd be happy to make list-table.lua work with codebraid's line numbering, but that should happen on a separate issue. I'll create this other issue tomorrow.
Does this all make sense?
Does this all make sense?
Yes, this does, @wlupton.
@gusbemacbe please could you try the feature/support-sourcepos branch with commonmark_x. Thanks.
Good morning, @wlupton!
I have just tested with both
from: commonmark_x;from: commonmark_x-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block(@gpoore's Codebraid Preview's Lua filter says not to supportalertsandtex_math_gfm);
It worked perfectly and fully with Codebraid Preview with enabled and disabled scrolling, except the third table:
Only - []{rowspan=2} `italic` doesn't work because it is not spanned with two rows.
And the last table failed to work with Codebraid Preview:
:::{.list-table}
- []{style="background: lightblue"}
- Ocean
- Area (sq. miles)
- - Atlantic
- 41.1 million
- - Pacific
- 63.8 million
:::
:::{.list-table style="background: lightsalmon"}
* - Fish
- Ocean
* :::{.list-table-body style="background: lightgreen"}
- - Arctic char
- Arctic
- - Humuhumunukunukuāpuaʻa
- Pacific
:::
* ------------- ---------
Cod Atlantic
Notothenioids Antarctic
------------- ---------
:::Here is the error:
Codebraid Preview Error
pandoc failed:
Error: Command failed: pandoc --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/%40vscode/codicons/dist/codicon.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/vscode-markdown.css --css=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/media/codebraid-preview.css --defaults "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/defaults/JCaZoD3OG8INzm4MPjk-TFkkqtepMQqgUZeBa0McErk.yaml" --standalone --wrap=auto --defaults "/home/gus/Tests/confluence/_codebraid/temp_defaults/_codebraid_preview.yaml" --standalone --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/sourcepos_sync.lua" --katex=https://file%2B.vscode-resource.vscode-cdn.net/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/node_modules/katex/dist/ --lua-filter="/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/filters/show_raw.lua" --from "/home/gus/.vscode-insiders/extensions/gpoore.codebraid-preview-0.15.0/pandoc/readers/commonmark_x.lua-ascii_identifiers-east_asian_line_breaks-implicit_figures-pipe_tables-smart-sourcepos-wikilinks_title_after_pipe-wikilinks_title_before_pipe+attributes+autolink_bare_uris+bracketed_spans+definition_lists+emoji+fancy_lists+fenced_divs+footnotes+gfm_auto_identifiers+hard_line_breaks+implicit_header_references+raw_attribute+raw_html+rebase_relative_paths+strikeout+subscript+superscript+task_lists+tex_math_dollars+yaml_metadata_block+sourcepos" --to html
Error running filter assets/filters/lua/list-table-sourcepos.lua:
assets/filters/lua/list-table-sourcepos.lua:169: expected list item to have row attrs at
------------------------------------------------------------------------
stack traceback:
assets/filters/lua/list-table-sourcepos.lua:51: in upvalue 'assert_'
assets/filters/lua/list-table-sourcepos.lua:169: in function <assets/filters/lua/list-table-sourcepos.lua:134>
With markdown_github and with the updated list-table filter, the spanned rows worked:
I suspend CommonMark X does not support it.
Thanks. The rowspan problem was my fault (I hadn't skipped the data-pos Divs in this case). It should now be fixed.
The other problem was that the table shown below is a "simple table", but commonmark_x only supports pipe tables, so I've changed this README.md example to use a pipe table.
* ------------- ---------
Cod Atlantic
Notothenioids Antarctic
------------- ---------
I believe that all the examples are now working in Codebraid with commonmark_x. Do you agree?
I believe that all the examples are now working in Codebraid with
commonmark_x. Do you agree?
- It almost worked, but it does not solve the
colspanorrowspanproblem. - I enabled
pipe_tablesand it still failed:
Please double-check that you have the latest filter code. I believe that I have commented out the test that would lead to that error.
I have just tested the latest list-table codes with the latest Lua filter code on Codebraid Preview with the commonmark_x+pipe_tables and it worked fully and perfectly!
Great! Note that you shouldn't need the +pipe_tables (commonmark_x already enables pipe tables).