`tokenizer` extension

[Anton]

What's about add tokenizer extension containing class PhpToken?

[Dariusz Rumiński]

-1 for partial polyfill
if you claim that tokenizer extension is polyfilled, yet provide only PhpToken but not token_get_all function - PHP CS Fixer will fail to work

[Anton]

@keradus it's polyfill for php less 8.0 like as intl Normalizer, tokenizer extension still required.

[Nicolas Grekas]

if you claim that tokenizer extension is polyfilled, yet provide only PhpToken but not token_get_all function - PHP CS Fixer will fail to work

I agree, but can't this be solved by not claiming that this would polyfill ext-tokenizer (since it doesn't)?

[Anton]

of course, we can implement token_get_all() (I saw few solutions), but this is extremely expensive at maintaining

[Nicolas Grekas]

We'd better not do this :)

[Nicolas Grekas]

It could be fine to add this class to the Php80 polyfill and define the class only if \function_exists('token_get_all').
Can you please send a PR @WinterSilence? (with some test cases of course)

[Anton]

@nicolas-grekas ok, PHP8 tests to compare native PhpToken and polyfill? PHP7 tests to compare polyfill and predefined expected results?

[Nicolas Grekas]

The CI should handle this already because it runs on php 8 and php 7.

[Anton]

@nicolas-grekas send PR to current repo or symfony/polyfill-php80?

[Nicolas Grekas]

To the current repo please, the other is an automatic subtree split.

[Anton]

@nicolas-grekas As I'm see, package use custom legacy PhpUnit... I'm don't have time to explore this history museum and can write tests only for PhpUnit8+.

Also, why composer.json not contain autoload-dev section - how correctly add my test class to autoloader?

Class Symfony\Polyfill\Php80\PhpToken in /src/Php80/PhpToken.php, Symfony\Polyfill\Tests\Php80\PhpTokenTest in /tests/Php80/PhpTokenTest.php and

if (!\class_exists('PhpToken') && \function_exists('token_get_all')) {
    class PhpToken extends p\PhpToken {}
}

into /src/Php80/bootstrap.php, right?

[Nicolas Grekas]

That's fine, just send the PR with what you have, we'll find a way.

[Anton]

@nicolas-grekas also, if "symfony/phpunit-bridge": "^5.3|^6.0" require same version of phpunit, then need update phpunit.xml.dist: xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/5.0/phpunit.xsd" or xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/6.0/phpunit.xsd" to calm "barking" IDE (:

[Nicolas Grekas]

That's fine, polyfills cannot use cutting edge tools by definition 🤷

[Anton]

@nicolas-grekas it's not fine or true - just outdated DTD, but I'm don't want argue about it