Startup and shutdown overhead

[Uwe Fechner]

By default we have quite a high startup and shutdown time:

ufechner@ubuntu:~/repos/LogFiles$ time asysimg --project -t auto -e ""
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/asysimg-2022-09-20T10-49-17.212.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_9uDsjwxerC-snoop.jl
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/snoop-file.jl

real	0m3.351s
user	0m3.309s
sys	0m1.459s

compared to:

ufechner@ubuntu:~/repos/LogFiles$ time julia --project -t auto -e ""

real	0m0.175s
user	0m0.105s
sys	0m0.105s

[Uwe Fechner]

First idea how to reduce that:

Create a second startup script with the name runjulia/runjulia.bat in the project folder that includes a hard coded path to the system image, like

ufechner@ubuntu:~/repos/LogFiles$ time julia --project -J /home/ufechner/.julia/asysimg/1.8.1/4Hy5DP/asysimg-2022-09-20T10-49-17.212.so -t auto -e ""

real	0m0.331s
user	0m0.310s
sys	0m0.670s

While this is not as fast as a default Julia terminal, it is 10 times faster than the default asysimg execution.

Please, comment on this idea!

[Petr Vana]

The problem is that some precompile statements for this packages are not included correctly. It seems, there is a bug somewhere in the following code. I'll investigate it.

AutoSysimages.jl/src/AutoSysimages.jl

Lines 494 to 510 in 47128ab

 # First collect precompile statements for a dummy run (e.g., with -e "") 
 @info "AutoSysimages: Collecting precompile statements for empty run (-e \"\")" 
 empty_precompiles = tempname() 
 julia_cmd = joinpath(Sys.BINDIR::String, Base.julia_exename()) 
 readlines( 
     pipeline( 
         `$julia_cmd -J $loaded_image -e "" --trace-compile stderr`, 
         stderr = empty_precompiles, 
     ), 
 ) 
 precompiles = String[] 
 for line in readlines(empty_precompiles) 
     if startswith(line, "precompile(") && endswith(line, ")") 
         push!(precompiles, line[12:end-1]) 
     end 
 end 
 _append_statements(precompiles) 

[Petr Vana]

I'll keep this open before we solve all the problems. Now, the time is reduced to about 1 second in #22.

[Petr Vana]

Maybe the simplest solution is to write a short independent script that would be called via -L and return the same arguments without loading the whole AutoSysimages package.

AutoSysimages.jl/scripts/unix/asysimg

Line 3 in 415f613

 asysimg_args=`$JULIA_EXE -e "using AutoSysimages; print(julia_args()); exit();" "$@"` 

[Uwe Fechner]

I don't think that it is resolved:

ufechner@tuxedi:~/repos/Plotting$ time asysimg --project -e ""
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/asysimg-2022-09-18T00-41-33.838.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/23ZRuw
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_NCFqZMj8nO-snoop.jl
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl

real	0m5,282s
user	0m5,268s
sys	0m1,642s

[Petr Vana]

@ufechner7 This may happen in dev mode because the package is not included into sysimage automatically. You can force it by

AutoSysimages.add("AutoSysimages")

Does it helps?

[Petr Vana]

You can check loaded packages by

$ julia -J /home/petr/.julia/asysimg/1.7.2/3Ptgat/asysimg-2022-09-21T12-48-53.219.so
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Base.loaded_modules |> collect .|> println;
Downloads [f43a241f-c20a-4ad4-852c-f6b1247861c6] => Downloads
PackageCompiler [9b87118b-4619-50d2-8e1e-99f35a4d4d9d] => PackageCompiler
Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5] => Unicode
...

[Uwe Fechner]

Latest test results:

ufechner@tuxedi:~/repos/Plotting$ time asysimg --project -e ""
┌ Info: The package AutoSysimages.jl started!
│  Loaded sysimage:    /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/asysimg-2022-09-24T00-55-54.112.so
│  Active directory:   /home/ufechner/.julia/asysimg/1.8.1/23ZRuw
│  Global snoop file:  /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl
└  Tmp. snoop file:    /tmp/jl_an8LiFUVqT-snoop.jl
[ Info: AutoSysimages: Copy snooped statements to: /home/ufechner/.julia/asysimg/1.8.1/23ZRuw/snoop-file.jl

real	0m2,919s
user	0m2,969s
sys	0m1,199s
ufechner@tuxedi:~/repos/Plotting$ time julia --project -e ""

real	0m0,084s
user	0m0,062s
sys	0m0,054s

So yes, a slight improvement, but not enough to close this issue...

To reproduce my results please clone https://github.com/ufechner7/Plotting and follow the instructions in README.md

[Petr Vana]

As the first step, I proposed to add some more information to "Info:" section to make sure which version of the package is used.

[Uwe Fechner]

(Plotting) pkg> st
Status ~/repos/Plotting/Project.toml
[4eb35182] AutoSysimages v0.2.5
[4c0ca9eb] Gtk v1.2.2
[d0351b0e] InspectDR v0.4.3
⌅ [91a5bcdd] Plots v1.32.0
Info Packages marked with ⌅ have new versions available but cannot be upgraded. To see why use status --outdated