Network requests in build.rs and heavy build dependencies

[Armin Ronacher]

The alpha release of the crate requires tokio and reqwest during build time which are very heavy dependencies. More importantly it seems to be doing HTTP requests in build.rs which is a bad idea.

Could this be implemented in different ways?

[Titus]

There are two network requests but they’re only fired if there is no file yet, e.g.,:

markdown-rs/build.rs

Lines 15 to 23 in af96212

 let value = if let Ok(value) = fs::read_to_string(data_url) { 
     value 
 } else { 
     let value = reqwest::get(url).await.unwrap().text().await.unwrap(); 
  
     fs::write(data_url, value.clone()).unwrap(); 
  
     value 
 }; 

.

I’ve noticed Rust gets very slow with build scripts though.

Could this be implemented in different ways?

Pretty sure you know more about Rust (ecosystem) than I do, so I’m all ears on what a good way to build files in a Rust project is, other than, ehh, a build script 😅

[Armin Ronacher]

The build script itself is not the issue, it's mostly about how heavy it is. A reasonable alternative would be to put this logic behind an optional feature. On second look the build script also does something it's not supposed to do which is to change the contents in src (for the unicode) file.

In some ways what this build script does is probably better done in a separate utility for development purposes only and to check-in the output of it regularly. That's what I also generally do myself for similar situations.

[Titus]

Why are build scripts not supposed to change src?

separate utility for development purposes only and to check-in the output of it regularly

That’s exactly what I’m using build scripts for. I guess that isn’t what build scripts in the rust ecosystem are supposed to do. Rust ecosystem is weird sometimes.

[Armin Ronacher]

Why are build scripts not supposed to change src?

Build scripts are supposed to place content in OUT_DIR and then have a script in src/ reference it. If you do place stuff there you end up with bugs like this: rust-lang/cargo#3076

That’s exactly what I’m using build scripts for. I guess that isn’t what build scripts in the rust ecosystem are supposed to do. Rust ecosystem is weird sometimes.

build scripts in the rust ecosystem are just there to run additional time at build time. This however seems to be a task that is run during development not at build time (eg: artifacts are checked out).

[Titus]

This however seems to be a task that is run during development

Correct.

Is this something you would like to work on, or can you otherwise share how you would do this?

[Titus]

Fixed in https://github.com/wooorm/markdown-rs/releases/tag/1.0.0-alpha.5!