Giters

sebastianbergmann / php-token-stream

Wrapper around PHP's tokenizer extension.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

PHP 8 and trailing newline in comment token

sebastianbergmann opened this issue · comments

commented

@theseer will (probably) have to adapt theseer/tokenizer for this as well.

commented

Checking...

commented

The changes made in 19cf736 re-attach the newline to the comment token and effectively reverse the change in PHP 8 so that code that uses this component continues to work without change in PHP 8.

test.php

<?php declare(strict_types=1);
require __DIR__ . '/vendor/autoload.php';

$i = 1;

foreach ((new PHP_Token_Stream(file_get_contents(__DIR__ . '/tests/_fixture/issue_95.php'))) as $token) {
    print $i++ . "\t" . get_class($token) . "\t'" . $token . "'\n";
}

PHP 7.4

1	PHP_Token_OPEN_TAG	'<?php '
2	PHP_Token_DECLARE	'declare'
3	PHP_Token_OPEN_BRACKET	'('
4	PHP_Token_STRING	'strict_types'
5	PHP_Token_EQUAL	'='
6	PHP_Token_LNUMBER	'1'
7	PHP_Token_CLOSE_BRACKET	')'
8	PHP_Token_SEMICOLON	';'
9	PHP_Token_WHITESPACE	'
'
10	PHP_Token_IF	'if'
11	PHP_Token_WHITESPACE	' '
12	PHP_Token_OPEN_BRACKET	'('
13	PHP_Token_STRING	'false'
14	PHP_Token_CLOSE_BRACKET	')'
15	PHP_Token_WHITESPACE	' '
16	PHP_Token_OPEN_CURLY	'{'
17	PHP_Token_WHITESPACE	'
    '
18	PHP_Token_COMMENT	'// @codeCoverageIgnoreStart
'
19	PHP_Token_WHITESPACE	'    '
20	PHP_Token_COMMENT	'# ...
'
21	PHP_Token_WHITESPACE	'    '
22	PHP_Token_COMMENT	'// @codeCoverageIgnoreEnd
'
23	PHP_Token_CLOSE_CURLY	'}'
24	PHP_Token_WHITESPACE	'
'

PHP 8

1	PHP_Token_OPEN_TAG	'<?php '
2	PHP_Token_DECLARE	'declare'
3	PHP_Token_OPEN_BRACKET	'('
4	PHP_Token_STRING	'strict_types'
5	PHP_Token_EQUAL	'='
6	PHP_Token_LNUMBER	'1'
7	PHP_Token_CLOSE_BRACKET	')'
8	PHP_Token_SEMICOLON	';'
9	PHP_Token_WHITESPACE	'
'
10	PHP_Token_IF	'if'
11	PHP_Token_WHITESPACE	' '
12	PHP_Token_OPEN_BRACKET	'('
13	PHP_Token_STRING	'false'
14	PHP_Token_CLOSE_BRACKET	')'
15	PHP_Token_WHITESPACE	' '
16	PHP_Token_OPEN_CURLY	'{'
17	PHP_Token_WHITESPACE	'
    '
18	PHP_Token_COMMENT	'// @codeCoverageIgnoreStart
'
19	PHP_Token_WHITESPACE	'    '
20	PHP_Token_COMMENT	'# ...
'
21	PHP_Token_WHITESPACE	'    '
22	PHP_Token_COMMENT	'// @codeCoverageIgnoreEnd
'
23	PHP_Token_CLOSE_CURLY	'}'
24	PHP_Token_WHITESPACE	'
'

Differences between PHP 7.4 and PHP 8

$ php test.php > php-74.txt   
$ php-80 test.php > php-8.txt 
$ diff -u php-74.txt php-8.txt
commented

Work on this is not complete.