pyupgrade 3.16.0 improperly breaks on emoji/formatted multiline string

[Hemanth Kini]

Using Python 3.12.4 and pyupgrade 3.16.0 results in a bug:

hkini$ pip show pyupgrade
Name: pyupgrade
Version: 3.16.0
Summary: A tool to automatically upgrade syntax for newer versions.
Home-page: https://github.com/asottile/pyupgrade
Author: Anthony Sottile
Author-email: [e-mail]
License: MIT
Location: /Users/hkini/.asdf/installs/python/3.12.4/lib/python3.12/site-packages
Requires: tokenize-rt
Required-by:
hkini$ cat test.py
def test(test_arg1):
    content = f"""
    <table style="width: 100%; height: 100%; font-size: 1.1em;">
        <tr>
            <td style="width: 100%; height: 100%;">🔗 This is a test {test_arg1} </td>
        </tr>
    </table>
    """
    return content
hkini$ pyupgrade test.py
Rewriting test.py
hkini$ cat test.py
def test(test_arg1):
    content = f"""
    <table style="width: 100%; height: 100%; font-size: 1.1em;">
        <tr>
            <td style="width: 100%; height: 100%;">🔗 This is a test {te{test_arg1} </td>
        </tr>
    </table>
    """
    return content

This used to work with Python 3.12.2 - I suspect it's a combination of the use of an emoji, the variable within the curly brace, and some change in the tokenizer/untokenizer logic.

[Hemanth Kini]

Note - I think this relates to PyCQA/docformatter#282 (which I also filed)

[Anthony Sottile]

please make an issue against cpython, they must have regressed fstring positions again