Use AnnotatedIOBuffer + StyledStrings for handling ANSI codes.

[Timothy]

With the impending release of StyledStrings, I think the ability to capture output styling with an AnnotatedIOBuffer and then represent it as HTML might be of interest to Documenter as an eventual replacement for the current "parse ANSI escape characters" approach.

Here's an example of what this could look like:

aio = Base.AnnotatedIOBuffer()
old_stdout = getglobal(Base, :stdout)
try
    setglobal!(Base, :stdout, aio)
    printstyled("hey", bold=true, color=:magenta)
finally
    setglobal!(Base, :stdout, old_stdout)
end
read(seekstart(aio), Base.AnnotatedString)

When run in the REPL, you should see output like this:

This works thanks to a specialised prinstyled(::AnnotatedIOBuffer, ::AnnotatedString) method, but won't work for manually outputted ANSI escape codes.

There are three major reasons why I think this functionality might be of interest:

• Because this handles styling information more "directly", instead of reverse-engineering it by processing ANSI codes, it should be a more robust method of translating styling information to HTML
• StyledStrings constructed output can contain more styling information that can be represented with the basic ANSI styling codes. Some can be represented with terminal-specific ANSI extensions (e.e. a yellow wavy underline), other attributes cannot (e.g. font/height).
• With the way that StyledStrings works, it should be rather straightforward to write "writers" for various output formats (HTML and LaTeX are of immediate interest, plaintext is trivial and then you've got the interest in other formats like Typst).

[Morten Piibeleht]

Just as a note: I think this would need to be implemented in IOCapture. Or, at least, IOCapture is relevant too, since we do do some ANSI code handling there.

This works thanks to a specialised prinstyled(::AnnotatedIOBuffer, ::AnnotatedString) method, but won't work for manually outputted ANSI escape codes.

So.. I haven't followed StyledStrings closely enough to have a super informed opinion here. But, this feels like a major downside -- our current approach seems strictly better, since it handles StyledStrings and other outputs. ANSI codes not being able to see the non-standard styling is a fair point.. but neither will the user using the REPL.. at least in most terminal emulators..?

I think I need some more information here as to what StyleStrings actually does on a low level, and maybe a sketch of what an implementation would look like here, and how it differs from what we have now, to make a call here.

All that said, I do think that, if StyledStrings is the future, which it appears to be, then it does feel right that we should natively support it in Documenter.

[Timothy]

Hi Morten,

I think it might be best if I actually address your comments in reverse order.

All that said, I do think that, if StyledStrings is the future, which it appears to be, then it does feel right that we should natively support it in Documenter.

I think it's too early to say what "the" future looks like, but I'm hoping that StyledStrings will be adopted fairly broadly for the benefits it brings to handling styled content in many forms. It's about to be used in the Markdown stdlib, and I believe that Julius is interested in using it with Makie's styled text for example.

This is a bit funny since it's sort-of-terminal but not entirely. The design of StyledStrings is such that it covers terminal capabilities, but also supports other aspects that are only possible in non-terminal contexts (namely font selection and text sizing). I'm not entirely sure how this should work out, but I suspect it would be useful to be able to show StyledStrings content in docs.

I think I need some more information here as to what StyleStrings actually does on a low level, and maybe a sketch of what an implementation would look like here, and how it differs from what we have now, to make a call here.

Simply put, what StyledStrings does is take the new AnnotatedString type from Base (in 1.11+), which allows you to add labelled metadata to regions of a string, and introduces a "styling information" struct (called Face, think "typeface") to put in a metadata slot labelled :faces.

There's a bit more to make it work, make it user-customisable, etc. but that is the core of it.

ANSI codes not being able to see the non-standard styling is a fair point.. but neither will the user using the REPL.. at least in most terminal emulators..?

Well, it does makes drawing attention to those capabilities in @repl docs a bit harder 😛

So.. I haven't followed StyledStrings closely enough to have a super informed opinion here. But, this feels like a major downside -- our current approach seems strictly better, since it handles StyledStrings and other outputs.

Yea, while I would love to see everybody using StyledStrings not manually outputting ANSI codes, that definitely isn't the reality today, so I wouldn't suggest abandoning the SGR-parsing functionality, but I think it would be nice to also be able to work with some of the more advanced capabilities that StyledStrings offers.

I'm not sure what the best path forward is, but that's why I've opened this issue — so we can discuss it.