cannot parse modern rust due to reliance of syntex_syntax

[Eric Huss]

Since ctest is using a ~2 year old version of libsyntax, it cannot handle any language changes since then. Are there any plans or ideas on how to move forward? I noticed you looking at this at dtolnay/syn#588, but I was wondering if you've found anything since.

[gnzlbg]

Sadly no, this is also part of #23 .

Basically syntex was deprecated, but no replacement was provided. syn is just missing too many pieces for it to be usable:

• syn parses a file into an AST, but there is no way to expand macros
• syn parses only a file at a time, we'd need a way to parse lib.rs into an AST, step through the mod xyz; declarations, find the module (handle rust2016,2018,#[path]...), replace the mod xyz; with its content, if macros were exported try to expand those macros, etc.
• many macros are set globally, e.g., #[cfg(...)], so a way to set target_arch/os/feautre/env/pointer_width... (or to just select a target) and expand them in the syn AST.

For the use cases in libstd, syntex has recently started failing to parse parts of the file due to errors (e.g. packed(N)), but those errors can be ignored for the time being, so...

[gnzlbg]

@ehuss i've forked syntex_syntax, and added support for repr(packed(N)) and repr(transparent) to it .

If there is something from modern Rust missing there that you need, feel free to open a PR to my syntex fork.

[gnzlbg]

So I'm closing this issue as well, since there is nothing I can do here.

I don't know why those involved decided to deprecate syntex_syntax without providing a replacement, and I can't take the workload of re-releasing libsyntax on crates.io .

I do maintain a fork of the old syntex crate (syntex_syntax2 on crates.io) where I have hacked in support for some new language constructs used on FFI (repr(packed), repr(transparent), etc.) and that "works for me". If something should be hacked in there, you can open an issue in https://github.com/gnzlbg/syntex or even better, send a PR with a hack. It is completely unsupported and use at your own risk: there is no CI, no tests are run, and as long as everything continues to work in the rust-lang/libc repo which uses the fork, I don't mind modifying the library further if "it works for you".

cc @dtolnay @Centril who might know more about what happen with the syntex crates and what the plan was for crates that were using it to migrate to something else.

[David Tolnay]

what happen with the syntex crates and what the plan was for crates that were using it

I no longer needed it for Serde so it didn't make sense to keep putting in time keeping up with rustc. Someone who wants that functionality should feel free to maintain a fork, but note that it's a lot of work. Almost everything that used it was able to switch to either syn or rustc-ap-syntax.

[gnzlbg]

Maybe we could switch to rustc-ap-syntax, it appears that it currently doesn't build, but i'm re-opening this until someone evaluates how easy / hard would it be to migrate to that (if possible at all).

[Wim de With]

I've done some experiments trying to use the rust-ap-* crates but I encountered an incredible amount of issues with getting it to compile. rustc-ap-syntax is now rust-ap-rustc_ast, but it does not only need that crate, but at least the following list:

• rustc-ap-rustc_ast
• rustc-ap-rustc_parse
• rustc-ap-rustc_span
• rustc-ap-rustc_feature
• rustc-ap-rustc_expand
• rustc-ap-rustc_attr
• rustc-ap-rustc_session

At this point I was compiling half of rustc in a rather unsupported way, leading to errors such as:

    Checking rustc-ap-rustc_attr v727.0.0
error: environment variable `CFG_RELEASE` not defined
   --> /home/wf/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc-ap-rustc_attr-727.0.0/src/builtin.rs:594:47
    |
594 |             let rustc_version = parse_version(env!("CFG_RELEASE"), true).unwrap();
    |                                               ^^^^^^^^^^^^^^^^^^^
    |
    = note: this error originates in the macro `env` (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

error: could not compile `rustc-ap-rustc_attr`

To learn more, run the command again with --verbose.

This could be 'solved' this by setting CFG_RELEASE to something manually, which is... not great.

In between the time syntex was written and now the internal API of the compiler has changed quite a bit; some effort would be needed to adapt ctest to the current API in the rustc crates. Although I believe it is theoretically possible to get this to work, it would make ctest incredibly fragile, because every time the internal rustc API changes, it will fail to build. Also, I'm pretty sure rustc cannot be built with a stable rustc because it uses feature flags, meaning it would need nightly compilers which change every day. I remember when clippy wasn't a part of rustup and had the same problem, which wasn't fun.

I also had to use an old nightly (nightly-2021-07-07) because those rustc-ap-* crates are very stale.