Is still utf16 with text-2?
dpwiz opened this issue · comments
I see there is a mix of "16" and "8" length in the code, text-2 in deps and utf16 in package name. It's all very confusing now.
A readme notice about what's going on would be helpful.
The indexing operations and so on are measured in UTF-16 but the internal representation is that of Text, i.e. UTF-8. So there's some remapping going on internally. The reason for keeping up the indexing using UTF-16 code units after upgrading to text-2 is for interfacing with external tools that use that kind of indexing (e.g. LSP).
Readme contributions would be welcome.