squidfunk / mkdocs-material

Contribution guidelines

I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

... the problem doesn't occur with the mkdocs or readthedocs themes
... the problem persists when all overrides are removed, i.e. custom_dir, extra_javascript and extra_css
... the documentation does not mention anything about my problem
... there are no open or closed issues that are related to my problem

Description

I'm trying to get code highlighting to work but when including the documented option to enable, the site crashes

markdown_extensions:
  - pymdownx.highlight:
      anchor_linenums: true

Expected behaviour

The site should start up and code blocs should be highlighted

Actual behaviour

When serving the site, the crashes

mkdocs serve
INFO     -  Building documentation...
INFO     -  Cleaning site directory
ERROR    -  Error reading page 'index.md': format() argument must be a formatter instance, not a class
Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 61, in format
    formatter.format(tokens, realoutfile)
  File "C:\Python310\lib\site-packages\pygments\formatter.py", line 94, in format
    return self.format_unencoded(tokensource, outfile)
  File "C:\Python310\lib\site-packages\pygments\formatters\html.py", line 981, in format_unencoded
    source = self.wrap(source)
TypeError: BlockHtmlFormatter.wrap() missing 1 required positional argument: 'outfile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python310\Scripts\mkdocs.exe\__main__.py", line 7, in <module>
  File "C:\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Python310\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Python310\lib\site-packages\mkdocs\__main__.py", line 181, in serve_command
    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
  File "C:\Python310\lib\site-packages\mkdocs\commands\serve.py", line 63, in serve
    config = builder()
  File "C:\Python310\lib\site-packages\mkdocs\commands\serve.py", line 58, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "C:\Python310\lib\site-packages\mkdocs\commands\build.py", line 292, in build
    _populate_page(file.page, config, files, dirty)
  File "C:\Python310\lib\site-packages\mkdocs\commands\build.py", line 174, in _populate_page
    page.render(config, files)
  File "C:\Python310\lib\site-packages\mkdocs\structure\pages.py", line 175, in render
    self.content = md.convert(self.markdown)
  File "C:\Python310\lib\site-packages\markdown\core.py", line 268, in convert
    newRoot = treeprocessor.run(root)
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 491, in run
    code.highlight(
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 392, in highlight
    code = highlight(src, lexer, formatter)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class

Steps to reproduce

Create a new docs project
Add provided configuration
Attempt to serve mkdocs serve

Package versions

Python: python --version
3.10.2
MkDocs: mkdocs --version
1.3.0
Material: pip show mkdocs-material | grep -E ^Version
8.2.10

Configuration

site_name: My Site
theme:
  name: material
markdown_extensions:  
  - pymdownx.highlight:
      anchor_linenums: true

System information

Operating system: Windows 10
Browser: Tested in Brave and Chrome

This is already fixed and released in 9.4

Of pymdownx-extensions that is

Just ran into the same issue with the following package versions.

mkdocs-material-8.2.10
pymdown-extensions-9.4
Pygments-2.12.0

Tried to pin the pygments version with pygments>=2.10,<2.12, then got the error Pymdownx Highlight requires at least Pygments 2.12+ if enabling Pygments

@giswqs Your issue is not the same. Pymdown Extensions 9.4 fixes the issue with Pygments 2.12 and requires Pygments 2.12, so if you pin Pygments to less than 2.12, then you need to also use Pymdown Extensions 9.3. If you want to use the latest of both, don't pin them.

@facelessuser I understand. Initially, I did not pin versions, and I got the error TypeError: format() argument must be a formatter instance, not a class. Then I pinned pygments>=2.10,<2.12 and got the error Pymdownx Highlight requires at least Pygments 2.12+ if enabling Pygments. So I guess I also need to pin pymdown-extensions<9.4 as a temporary workaround
https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:212

It would be great if mkdocs-material can fix the issue so that I don't need to pin versions.

@giswqs

So I guess I also need to pin pymdown-extensions<9.4 as a temporary workaround

No, I don't see why you need to pin <9.4. You need to use the latest of both. It is possible maybe the packages were out of sync when you tested, and you didn't get the latest of both.

I was able to test on a py3.7 - p3.10 with the latest Pygments and Pymdown Extensions with no issues. I was able to deploy the documentation for the project with the latest Pygments, Pymdown Extensions, and MkDocs Material.

TypeError: format() argument must be a formatter instance, not a class

This error is specifically caused because you are using the latest Pymdown Extensions but pinning an older Pygments version.

It would be great if mkdocs-material can fix the issue so that I don't need to pin versions.

From what I see, there are no more issues, and mkdocs-material doesn't need to fix any issues.

Can you provide more specific details? I'm happy to fix anything on my end that I can, but what you've posted is a known error when using incompatible Pygments and Pymdown Extensions versions. You need to use the lastest of both.

@facelessuser No, it fails to build if I don't pin versions even with the latest mkdocs-material. See the following three scenarios. Only the third scenario with pin versions of the both pygments and pymdown-extensions works.

requirements_docs.txt

First scenario:

mkdocs-material

Commit: https://github.com/giswqs/geemap/pull/1038/files (no changes to requirements_docs.txt)
Error: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:213

Second scenario:

mkdocs-material 
pygments>=2.10,<2.12

Commit: gee-community/geemap@5cb3d12
Error: https://github.com/giswqs/geemap/runs/6150975478?check_suite_focus=true#step:11:64

Third scenario:

mkdocs-material 
pygments>=2.10,<2.12
pymdown-extensions<9.4

commit: gee-community/geemap@7049f86
pass: https://github.com/giswqs/geemap/runs/6151239311?check_suite_focus=true

For the first scenario above, it installed all packages with the latest version. However, it still throws the error TypeError: format() argument must be a formatter instance, not a class.

mkdocs-material-8.2.10
pymdown-extensions-9.4
Pygments-2.12.0

Pygments: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:77
mkdocs-material: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:51
pymdown-extensions: https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:168

@giswqs But this isn't a Mkdocs Material issue or a Pymdown Extensions issue. You have a conflict as I believe Juypter is also requiring Pygments. It's using a Mistune markdown parser as well.

https://github.com/giswqs/geemap/runs/6150975478?check_suite_focus=true#step:11:64

This error clearly fails due to Pymdown Extensions needed the most recent Pygments, and it is not there. pygments-2.11.2 pymdown-extensions-9.4.

https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:11:213

This one isn't even failing in Mkdocs Material, it is failing in Jupyter. This is not a Material issue. You have a lot of libraries, and they have incompatible requirements. Juypter seems to be running an entirely different Markdown parser (Mistune) than Mkdocs is using.

You have a special case where you need to pin both pymdown-extensions and pygments, but that is because of additional 3rd party things you are pulling in. That has nothing to do with Material specifically though.

It installed Pygments 2.12.0, which is the latest version, newer than 2.11.2 you mentioned.

https://github.com/giswqs/geemap/runs/6150893473?check_suite_focus=true#step:10:77

This error clearly fails due to Pymdown Extensions needed the most recent Pygments, and it is not there. pygments-2.11.2 pymdown-extensions-9.4.

I can't tell what version of Pygments is installed as I see this:

Requirement already satisfied: Pygments>=1.6 in /opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages (from grip->-r requirements_docs.txt (line 4)) (2.12.0)

Also, the failure is not in Mkdocs Material or Pymdown Extensions, it is in Jupyter somewhere via mkdocs_jupyter. That's a Mkdocs Jupyter/Jupyter issue.

Hey so I got a little lost in here... are we off topic or did I miss a turn somewhere? I don't use Jupyter and I just installed mkdocs-material using the docs on the site. I need version 9.4? I followed the instructions which led to 8.*

The version number (2.12.0) can be found at the end of the line you pasted. The error message is the same as @jstafford5380 posted.

Once I pinned pygments and pymdown-extensions versions, then it built successfully. I agreed that it is probably not a mkdocs-material issue, but I don't think it is a mkdocs-jupyter either @danielfrg.

pygments>=2.10,<2.12
pymdown-extensions<9.4

@jstafford5380 sorry, we are a bit off topic. The failure was in pymdown-extensions, which I maintain. Version 9.4 of that package now works with Pygments 2.12+ only, but should fix the failure.

The off topic part is that some plugins for MkDocs may not support the latest pygments. That is not specifically a Material issue though. If you need to pin packages because some other external MkDocs plugin has a conflict still, then you'll have to pin multiple things. If not, you can use the latest Pygments with the latest pymdown-extensions.

@giswqs It is not the same

The opening post originates in pymdownx.highlight, something that Material uses, that is why it is relevant:

  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 491, in run
    code.highlight(
  File "C:\Python310\lib\site-packages\pymdownx\highlight.py", line 392, in highlight
    code = highlight(src, lexer, formatter)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "C:\Python310\lib\site-packages\pygments\__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class

Yours originates in Mistune/Jupyter, a completely different Markdown parser originating from MkDocs Jupyter. The final line is the same, but that does not mean the same libraries are responsible.

  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1141, in output_loose_item
    body += self.tok()
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1063, in tok
    return getattr(self, 'output_%s' % t)()
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mistune.py", line 1085, in output_code
    return self.renderer.block_code(
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/mkdocs_jupyter/nbconvert2.py", line 219, in block_code
    return highlight(code, lexer, formatter)
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/pygments/__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/opt/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/site-packages/pygments/__init__.py", line 69, in format
    raise TypeError('format() argument must be a formatter instance, '
TypeError: format() argument must be a formatter instance, not a class
Error: Process completed with exit code 1.

Pymdown Extensions, which Material relies on, is fully tested with Pygments 2.12: https://github.com/facelessuser/pymdown-extensions/actions/runs/2217033680. I even test that it builds with the latest MkDocs Material: I even test that it builds with MkDocs Material: https://github.com/facelessuser/pymdown-extensions/runs/6149575142?check_suite_focus=true#step:7:47.

If you can provide a minimal example showing otherwise, I'm happy to dig in and try to fix it, but the trace you've linked to seems to indicate this fails in the MkDocs Juypter plugin, it does not originate in MkDocs Material or Pymdown Extensions (what the opening post showed).

@jstafford5380 sorry, we are a bit off topic. The failure was in pymdown-extensions, which I maintain. Version 9.4 of that package now works with Pygments 2.12+ only, but should fix the failure.

The off topic part is that some plugins for MkDocs may not support the latest pygments. That is not specifically a Material issue though. If you need to pin packages because some other external MkDocs plugin has a conflict still, then you'll have to pin multiple things. If not, you can use the latest Pygments with the latest pymdown-extensions.

Ok interesting. So I re-accomplished the install on a different machine and it works fine. Would/should it be in scope to list these minimum versions? I'm new to the ecosystem, but in other frameworks with package managers, usually the versions are worked out using semantic versioning and the manager will install or update to the correct versions.

Sorry I'm also new to the project -- i noticed that annotations don't work but the docs note that it's experimental. Does that mean it only works in the insider version?

Thanks for reporting. 68bcb1a pins Python Markdown Extensions to >=9.0,<9.4. I don't have the time right now to look into what's needed to support the latest versions ob both, PR appreciated. Other than that, I'll look into it later this week.

I've released 8.2.11 which temporarily pins Pygments and Python Markdown until this issue is resolved, so users can build their documentation projects. As already said, I'll look into it as soon as I got some time on my hands, but if somebody wants to go ahead and provide a PR, I'm happy to merge. We need to make sure that all code examples still work.

Thanks. Beautiful stuff btw :)

I will likely bump Material via PR. If there are issues, I may fix the easy ones, but unfamiliar areas, like maybe code annotations and such, I may defer to you @squidfunk to remedy those changes at your leisure.

This will not guarantee that that there will be no conflicts outside of Material, such as other MkDocs plugins, but we at least now have 8.2.11 which people can pin to if all there requirements are not ready to support the latest Pygments.

@facelessuser sounds great! If you run into any issues, it would be amazing if you could document them as part of the PR, so I can look into fixing them. I really hope that the latest Pygments doesn't break features such as code annotations, but I'm pretty confident that it should work with little effort.

The latest pygments did only a couple of things. It changed the HTMLFormater wrap method so it no longer accepted an out_file parameter. This is why things "broke". Pymdown Extensions subclasses that class to do a couple of things. That is now fixed. Everything else deals with the HTML output.

It now wraps the table output of code blocks (code blocks with line numbers that aren't using inline mode) with a <div class="highlight">. That class is no longer attached to the child <div> that wraps the <pre> block with the code. This isn't necessarily a problem from what I tested with Material, but I did not exhaustively test all features.

This means that SuperFences now attaches extra classes and attributes to the table code's outer <div> wrapper as well.

That's the gist of the changes. Specific enough to break what pymdownx was doing previously, but in the end, the HTML changes weren't that big. It might affect some CSS depending on how it targeted things related to code blocks in the table form, but all other cases will be fine. And, it may not affect any table code blocks if the CSS (maybe JS) is crafted in such a way that it doesn't care about these changes. That's the part I need to check.

I'll definitely document any Material issues I find.

It now wraps the table output of code blocks (code blocks with line numbers that aren't using inline mode) with a <div class="highlight">. That class is no longer attached to the child <div> that wraps the <pre> block with the code. This isn't necessarily a problem from what I tested with Material, but I did not exhaustively test all features.

This is likely to break code annotations in code blocks with line numbers (tables), but likely nothing that can't be fixed.

Yep, that is what I figured. The initial smoke test indicated that code blocks were generally working, but I suspected there would be some broken JS "magic" and maybe some minor CSS tweaks.

It now wraps the table output of code blocks (code blocks with line numbers that aren't using inline mode) with a <div class="highlight">. That class is no longer attached to the child <div> that wraps the \<pre\> block with the code. This isn't necessarily a problem from what I tested with Material, but I did not exhaustively test all features.

This is likely to break code annotations in code blocks with line numbers (tables), but likely nothing that can't be fixed.

When will code annotations be available in the free version? Really want to use it right now :)

Code annotations have been available in the free version since version 8.0.0, released on November 28, 2021, as they were part of the $4,000 found goal which was reached during that time.

@squidfunk So far the only issues I see are with annotations with Tabled Code blocks, everything else seems to be working. Copy to clipboard, display of code blocks, code block titles, line highlight, etc. all work. I am running on the public Material. I'll see how easy it is to fix annotations on tables. If I get bogged down, I'll let you know.

Okay, perfect. Thanks for your time! My guess is that this location needs to be adjusted:

mkdocs-material/src/assets/javascripts/components/content/code/_/index.ts

Lines 181 to 184 in d8d734d

    
           const container = el.closest([ 
        
             ":not(td):not(.code) > .highlight", 
        
             ".highlighttable" 
        
           ].join(", "))

It'll probably be a little more involved than just that, but that is definitely the area 🙂

Think I got it, it was easier than I thought.

@squidfunk Curious though. Is the removal of the comment # something only on insiders? Or is that portion on the public as well and broken?

I suspect the removal of # is only on insiders...I imagine if the Pymdown/Pygments fix was backported there the # would not be there.

Seems margins of table code blocks are not removed if they have line numbers under content tabs. I imagine that is an easy fix.

I think I found and fixed everything.

I suspect the removal of # is only on insiders...I imagine if the Pymdown/Pygments fix was backported there the # would not be there.

Jup, this is currently only in Insiders:
https://squidfunk.github.io/mkdocs-material/reference/code-blocks/#stripping-comments