tree-sitter-cli@0.22.2 breaks tree-sitter-css tests
jeandudey opened this issue · comments
Problem
The tests in tree-sitter-css fails with tree-sitter-cli 0.22.2. Also reported on: tree-sitter/tree-sitter-css#47.
After playing a bit with the test file I've found that the problem might not be the grammar but tree-sitter-cli itself as when removing the colons from the test name it works as intended.
For example, changing from this:
To this:
=========================
slotted pseudo element
=========================
Works without problems.
Steps to reproduce
git clone https://github.com/tree-sitter/tree-sitter-css --depth 1
cd tree-sitter-css; tree-sitter generate; tree-sitter test
Expected behavior
Tests should run without problem.
Tree-sitter version (tree-sitter --version)
tree-sitter 0.22.2
Operating system/version
GNU Guix 4d79a9cd6b5f0d8c5afbab0c6b70ae42740d5470
FWIW the test parser works (albeit more strict with suffixes). Maybe it's time to use it internally. ;)
If there's a way without doing it with cyclic dependencies then yes, we don't use the bundled generated files in GNU Guix so it means we'd have to keep an older version of tree-sitter-cli to build tree-sitter-test to build tree-sitter-cli 👀
There won't be any cyclic dependencies when #3069 is merged since parsers will only depend on tree-sitter-language. Assuming we want this, tree-sitter-cli will depend on the (not yet published) tree-sitter-test crate which will include the generated files, even when they're removed from version control in the future. If Nix/Guix has its own special way of building tree-sitter-cli then it may also need its own special solution.
It'd still be a cyclic dependency for Guix since we remove generated files from grammar repositories, i.e.: generated bindings, grammar.json, parser.c, etc. So to build tree-sitter-test one needs an existing tree-sitter-cli to generate those files to then build the newer tree-sitter-cli.
You won't need to build tree-sitter-test in order to build tree-sitter-cli if you just run cargo build --release and let it use the published crate which will always include the generated files.
Generated files are not allowed in Guix (even Rust crates are patched to remove bundled code) so the cyclic dependency is always there, i.e. we'll always generate tree-sitter-test files again, it'd basically introduce a bootstrapping problem in tree-sittter.
I guess if tree-sitter-test ends up being used in tree-sitter-cli we could always make patches to remove tree-sitter test command and thus the dependency on it so that we can generate tree-sitter-test to build a feature complete tree-sitter-cli later.
Sounds to me like this is a non-issue: tree-sitter-css simply needs to adjust to the intentionally changed (improved) test format, which is possible in a backward-compatible way. I'm sure a PR would be appreciated.
(And any -- so far hypothetical -- future changes are yours to deal with, as you apparently do with every packaged software.)
Sounds to me like this is a non-issue: tree-sitter-css simply needs to adjust to the intentionally changed (improved) test format, which is possible in a backward-compatible way.
This is actually an issue with the regex used to parse the headers, which is rather complicated at this point.
Generated files are not allowed in Guix (even Rust crates are patched to remove bundled code)
Does Nix also do that or did Guix decide to be extra special? In any case, if your package manager messes with packages that do not belong to it, it should deal with any issues that introduces.
Does Nix also do that or did Guix decide to be extra special?
Guix decides to be extra special to ensure that software always has a path to build fully from source, to avoid problems like the one in xz where the exploit was in auto generated files.
I'm not aware of Nix's guidelines on packaging though (or if they even have guidelines).
It is not just for security or saying meh don't trust anyone but having this capability using grammars in other forms not conceived by tree-sitter project itself. i.e.: as packages in Guix are code one could create a fork of tree-sitter and replace the official tree-sitter in those grammars and generate code for other languages instead of C for example, while also keeping the official packages, which is to say unlikely to be done but possible.
Or if for example, a vulnerability in the code generated of tree-sitter is found fixing the grammars should be as simple as updating the packaged tree-sitter.
it should deal with any issues that introduces.
You're right on this one, I was just pointing an issue that'd also benefit non-Guix users to be able to build tree-sitter without generated files, it'd make maintenance of tree-sitter in Guix easier anyway.
(And any -- so far hypothetical -- future changes are yours to deal with, as you apparently do with every packaged software.)
And I do, I mean, to fix this I just patched the tree-sitter-css test file in Guix at build time, this issue is more of a formality to open this issue.
The reason this issue was found in the first place is because we remove those files, re-generate them and then test as I'm sure other distributions do as well in some form or another, e.g. just testing.
Sorry if I offended you guys in some way or another in my previous comments, thanks for the help on this issue, just expressed my thoughts and you can feel free to ignore them, was just adding my POV as a packager.
No worries, appreciate the heads-up about tree-sitter-css test files being broken! (I just wanted to prevent this issue being derailed by the unrelated tree-sitter-test discussion.)