toml-spec-tests/values/qa-array-inline-nested-1000 and qa-table-inline-nested-1000 are broken on master
antoneliasson opened this issue · comments
tomlc99 at cdbb9de. toml-spec-tests at 64effd796
make; cd test2; bash build.sh; bash run.sh|grep -v OK
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG -c -o toml.o toml.c
ar -rcs libtoml.a toml.o
cc -shared -o libtoml.so toml.o
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG toml_json.c libtoml.a -o toml_json
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG toml_cat.c libtoml.a -o toml_cat
Cloning into 'toml-spec-tests'...
remote: Enumerating objects: 119, done.
remote: Counting objects: 100% (119/119), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 402 (delta 17), reused 112 (delta 11), pack-reused 283
Receiving objects: 100% (402/402), 98.65 KiB | 0 bytes/s, done.
Resolving deltas: 100% (37/37), done.
toml-spec-tests/values/qa-array-inline-nested-1000 ... [FAILED]
toml-spec-tests/values/qa-table-inline-nested-1000 ... [FAILED]
These tests are expected to fail as we do not validate Unicode strings.
@cktan Those are different tests.
The Note.txt in this repo mentions:
toml-spec-tests/errors/string-basic-multiline-out-of-range-unicode-escape-2.toml ... [FAILED]
toml-spec-tests/errors/string-basic-out-of-range-unicode-escape-2.toml ... [FAILED]
But they pass. Instead qa-array-inline-nested-1000 ... fail which don't rely on unicode AFAIK.
I just ran the tests and they seem to passed:
% bash run.sh | grep qa | grep nested
toml-spec-tests/values/qa-array-inline-nested-1000 ... [OK]
toml-spec-tests/values/qa-table-inline-nested-1000 ... [OK]
The tests fail on my machine exactly as reported by @antoneliasson .
Yes. Here's a clean reproduce of this for linux/musl:
docker build --rm -t tmp-repro . -f-<<EOF
FROM alpine
RUN apk add bash gcc musl-dev make git jq go && \
cd && \
git clone https://github.com/cktan/tomlc99 --depth 1 && \
cd tomlc99 && \
make && \
cd test2 && \
bash build.sh && \
bash run.sh
EOFRun that in an empty directory. The errors are:
toml-spec-tests/values/qa-array-inline-nested-1000 parse error: Exceeds depth limit for parsing at line 1, column 2132 ... [FAILED]
toml-spec-tests/values/qa-table-inline-nested-1000 parse error: Exceeds depth limit for parsing at line 1, column 897 ... [FAILED]
I cannot reproduce the failure:
% bash -v test.sh
git clone https://github.com/cktan/tomlc99
Cloning into 'tomlc99'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 230 (delta 2), reused 4 (delta 1), pack-reused 220
Receiving objects: 100% (230/230), 114.64 KiB | 0 bytes/s, done.
Resolving deltas: 100% (125/125), done.
Checking connectivity... done.
cd tomlc99/
make
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG -c -o toml.o toml.c
ar -rcs libtoml.a toml.o
cc -shared -o libtoml.so toml.o
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG toml_json.c libtoml.a -o toml_json
cc -std=c99 -Wall -Wextra -fpic -O2 -DNDEBUG toml_cat.c libtoml.a -o toml_cat
cd test2
bash build.sh
Cloning into 'toml-spec-tests'...
remote: Enumerating objects: 119, done.
remote: Counting objects: 100% (119/119), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 402 (delta 17), reused 112 (delta 11), pack-reused 283
Receiving objects: 100% (402/402), 98.65 KiB | 0 bytes/s, done.
Resolving deltas: 100% (37/37), done.
Checking connectivity... done.
bash run.sh | grep FAIL
%
Please use the command I provided which shows that the test fails on a clean linux system. Clearly there is some difference on your system or with your config that doesn't produce the failure.
I am quite sure the error comes from the 'jq' program, not from tomlc99.
My version of jq:
% jq --version
jq-1.5-1-a5b5cbe
The jq version in the repro command I posted comes from the alpine repos is is version 1.6. I can test this on my normal linux install later, but I remember the tests also failing there with an older jq version.
By the way, the unicode tests that are "expected to fail", never failed for me on any system 🤷
I don't think it's jq. I also have jq-1.5-1-a5b5cbe from Debian Buster. With the latest master the two tests that fail are qa-{array,table}-inline-nested-1000.
Here are the qa-{array,table}-inline-nested-1000.json.out from a failing test run: https://gist.github.com/antoneliasson/6c963eb29c50db9b43d71849fded5f53
As you can see they are significantly smaller than the reference files.
Help me chase it down? Run the following commands in test2/ to nail down which command failed:
fname='toml-spec-tests/values/qa-array-inline-nested-1000'
../toml_json $fname.toml > x.json.out
jq -S . x.json.out > x.json
diff x.json $fname.json