JET doesn't analyze expression given to `@btime`

[Alexandre Bergel]

Consider this example:

julia> JET.report_text("""
       using BenchmarkTools
       @btime (println("This ran!");1+"1") samples=1 evals=1
       """, concretization_patterns=[:x_])
This ran!
═════ 1 toplevel error found ═════
┌ @ top-level:2
│ MethodError: no method matching +(::Int64, ::String)
...

The example above shows that JET executes the macro in addition to running the usual checks. Is there a way for JET to not evaluate specific macros but only analyze the expression the macro operates on?

[Shuhei Kadowaki]

The concretization_patterns=[:x_] option tells JET to actually execute all provided (in order to to achieve better analysis accuracy). This means JET will execute code like the following, regardless of whether or not macros are being used:

julia> report_text("""
       println("This ran!")
       1 + "1"
       """; concretization_patterns=Any[:x_])
This ran!
...

To prevent this, I suggest you might want to consider not specifying concretization_patterns at all.

[Alexandre Bergel]

Thanks @aviatesk for the prompt reply.
However, it seems that removing the concretization_patterns prevents JET from finding obvious error. E.g.,

julia> JET.report_text("""
              using BenchmarkTools
              @btime (println("This ran!");1+"1") samples=1 evals=1
              """; target_defined_modules = true)
No errors detected

It should complain with 1+"1"

[Shuhei Kadowaki]

Indeed, the code passed to @btime is internally represented as an Expr object, and BenchmarkTools.jl fundamentally executes this using Core.eval. JET doesn't analyze these Expr objects, becauset Julia's metaprogramming features can generate an unlimited number of such invalid Expr objects, and it would be endless to report every single one. From the static analysis point of view, the difficulty of analyzing your case is essentially same as the one of analyzing code using Core.eval.

[Alexandre Bergel]

Thanks for your reply. It seems there is no solution to my initial problem then.

[Shuhei Kadowaki]

Indeed. Well, alternatively, JET could consider specially analyzing expressions provided to frequently used idiom such as @btime as a special case. To enable this, a sort of plugin system would be required.