vtjnash / SciMLBenchmarks.jl

Benchmarks for scientific machine learning (SciML) software and differential equation solvers

SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Differential Equation Solver Software

SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks for the SciML Scientific Machine Learning Software ecosystem, including cross-language benchmarks of differential equation solvers and methods for parameter estimation, training universal differential equations (and subsets like neural ODEs), and more.

Results

Static outputs in pdf, markdown, and html reside in SciMLBenchmarksOutput.

Table of Contents

• Multi-Language Wrapper Benchmarks
  • ODE Solver Multi-Language Wrapper Package Work-Precision Benchmarks (MATLAB, SciPy, Julia, deSolve (R))
  • Torchdiffeq vs DifferentialEquations.jl (/ DiffEqFlux.jl) Benchmarks
  • torchdiffeq vs Julia DiffEqFlux Neural ODE Training Benchmark
  • torchsde vs DifferentialEquations.jl / DiffEqFlux.jl
  • JITCODE vs SciPy vs DifferentialEquations.jl on large network dynamics
  • DifferentialEquations.jl vs Mujuco and DiffTaichi
  • DiffEqFlux.jl / DifferentialEquations.jl vs Jax on an epidemic model
  • DifferentialEquations.jl vs SciPy vs NumbaLSODA on a stiff ODE
  • DifferentialEquations.jl vs SciPy vs NumbaLSODA
• Non-stiff Ordinary Differential Equations (ODEs)
  • Linear Work-Precision Diagrams
  • Three-Body Work-Precision Diagrams
  • Pleides Work-Precision Diagrams
  • Rigid Body Work-Precision Diagrams
  • Fizhugh-Nagumo Work-Precision Diagrams
  • Lotka-Volterra Work-Precision Diagrams
  • Direct vs MATLAB Benchmark
  • Runge-Kutta vs Taylor Integration on Pleides
• Stiff Ordinary Differential Equations (ODEs)
  • Van der Pol Work-Precision Diagrams
  • ROBER Work-Precision Diagrams
  • Orego Work-Precision Diagrams
  • Hires Work-Precision Diagrams
  • Pollution Work-Precision Diagrams
  • BCR (1122 ODE) Work-Precision Diagrams
• Differential-Algebraic Equations (DAEs)
  • ROBER DAE Work-Precision Diagrams
  • OREGO DAE Work-Precision Diagrams
  • Chemical Akzo Nobel Differential-Algebraic Equation (DAE) Work-Precision Diagrams
• Method of Lines PDEs
  • Filament PDE Discretization Work-Precision Diagrams
  • Allen-Cahn Finite Difference Work-Precision Diagrams
  • Allen-Cahn Pseudospectral Work-Precision Diagrams
  • Burger's Finite Difference Work-Precision Diagrams
  • Burger's Pseudospectral Work-Precision Diagrams
  • KdV Finite Difference Work-Precision Diagrams
  • KdV Pseudospectral Work-Precision Diagrams
  • Kuramoto–Sivashinsky Finite Difference Work-Precision Diagrams
  • Kuramoto–Sivashinsky Pseudospectral Work-Precision Diagrams
• Dynamical ODEs
  • Single Pendulum Comparison Benchmark
  • Henon-Heiles Energy Conservation Benchmark
  • Quadrupole Boson Hamiltonian Energy Conservation Benchmark
• N-body problems
  • Acceleration function benchmarks
  • Liquid argon benchmarks
• Nonstiff SDEs
  • Simple Nonstiff SDE Strong Work-Precision Diagrams
  • Simple Nonstiff SDE Weak Work-Precision Diagrams
  • Lotka-Volterra SDE Work-Precision Diagrams
• Stiff SDEs
  • Stochastic Heat Equation Investigation
  • Quadratic Diffusion Noise Investigation
  • Oval2 Long Run
  • Oval2 Long Timings
  • Oval2 Timings
• Nonstiff DDEs
  • Constant Delay DDEs
    • Mackey and Glass Work-Precision Diagrams
    • Wheldon, Kirk, and Finlay Work-Precision Diagrams
• Stiff DDEs
  • Quorum Sensing Work-Precision Diagrams
• Jump Equations
  • Diffusion Model
  • Mendes Multistate Model
  • Negative Feedback Gene Expression Model
  • Negative Feedback Marchetti Model
• Parameter Estimation
  • Lorenz Equation Parameter Estimation by Optimization Methods
  • Bayesian Lotka-Volterra Parameter Estimation
  • Bayesian Lorenz Equation Estimation
  • Bayesian FitzHugh-Nagumo Equation Estimation
  • Lotka Volterra Equation Parameter Estimation by Optimization Methods
  • FitzHugh-Nagumo Equation Parameter Estimation by Optimization Methods
• Physics-Informed Neural Network (Neural Network PDE Solver) Cost Function Benchmarks
  • Allen-Cahn PDE Physics-Informed Neural Network (PINN) Loss Function Error vs Time Benchmarks
  • Diffusion Equation Physics-Informed Neural Network (PINN) Loss Function Error vs Time Benchmarks
  • Hamilton-Jacobi PDE Physics-Informed Neural Network (PINN) Loss Function Error vs Time Benchmarks
  • Level Set PDE Physics-Informed Neural Network (PINN) Loss Function Error vs Time Benchmarks
  • Nernst-Planck PDE Physics-Informed Neural Network (PINN) Loss Function Error vs Time Benchmarks
• Physics-Informed Neural Network (Neural Network PDE Solver) Optimizer Benchmarks
  • Diffusion Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks
  • 1D Nernst-Planck Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks
  • Allen-Cahn Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks
  • Berger's Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks
  • Hamilton-Jacobi Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks
  • Poisson Equation Physics-Informed Neural Network (PINN) Optimizer Benchmarks

The following tests were developed for the paper Adaptive Methods for Stochastic Differential Equations via Natural Embeddings and Rejection Sampling with Memory. These notebooks track their latest developments.

• SDE Adaptivity

  • qmax Determination Tests
  • Adaptive Efficiency Tests

Current Summary

The following is a quick summary of the benchmarks. These paint broad strokes over the set of tested equations and some specific examples may differ.

Non-Stiff ODEs

• OrdinaryDiffEq.jl's methods are the most efficient by a good amount
• The Vern methods tend to do the best in every benchmark of this category
• At lower tolerances, Tsit5 does well consistently.
• ARKODE and Hairer's dopri5/dop853 perform very similarly, but are both far less efficient than the Vern methods.
• The multistep methods, CVODE_Adams and lsoda, tend to not do very well.
• The ODEInterface multistep method ddeabm does not do as well as the other multistep methods.
• ODE.jl's methods are not able to consistently solve the problems.
• Fixed time step methods are less efficient than the adaptive methods.

Stiff ODEs

• In this category, the best methods are much more problem dependent.
• For smaller problems:
  • Rosenbrock23, lsoda, and TRBDF2 tend to be the most efficient at high
    tolerances.
  • Rodas4 and Rodas5 tend to be the most efficient at low tolerances.
• For larger problems (Filament PDE):
  • QNDF and FBDF does the best at all normal tolerances.
  • The ESDIRK methods like TRBDF2 and KenCarp4 can come close.
• radau is always the most efficient when tolerances go to the low extreme (1e-13)
• Fixed time step methods tend to diverge on every tested problem because the high stiffness results in divergence of the Newton solvers.
• ARKODE is very inconsistent and requires a lot of tweaking in order to not diverge on many of the tested problems. When it doesn't diverge, the similar algorithms in OrdinaryDiffEq.jl (KenCarp4) are much more efficient in most cases.
• ODE.jl and GeometricIntegrators.jl fail to converge on any of the tested problems.

Dynamical ODEs

• Higher order (generally order >=6) symplectic integrators are much more efficient than the lower order counterparts.
• For high accuracy, using a symplectic integrator is not preferred. Their extra cost is not necessary since the other integrators are able to not drift simply due to having low enough error.
• In this class, the DPRKN methods are by far the most efficient. The Vern methods do well for not being specific to the domain.

Non-Stiff SDEs

• For simple 1-dimensional SDEs at low accuracy, the EM and RKMil methods can do well. Beyond that, they are simply outclassed.
• The SRA and SRI methods both are very similar within-class on the simple SDEs.
• SRA3 is the most efficient when applicable and the tolerances are low.
• Generally, only low accuracy is necessary to get to sampling error of the mean.
• The adaptive method is very conservative with error estimates.

Stiff SDEs

• The high order adaptive methods (SRIW1) generally do well on stiff problems.
• The "standard" low-order implicit methods, ImplicitEM and ImplicitRK, do not do well on all stiff problems. Some exceptions apply to well-behaved problems like the Stochastic Heat Equation.

Non-Stiff DDEs

• The efficiency ranking tends to match the ODE Tests, but the cutoff from low to high tolerance is lower.
• Tsit5 does well in a large class of problems here.
• The Vern methods do well in low tolerance cases.

Stiff DDEs

• The Rosenbrock methods, specifically Rodas5, perform well.

Parameter Estimation

• Broadly two different approaches have been used, Bayesian Inference and Optimisation algorithms.
• In general it seems that the optimisation algorithms perform more accurately but that can be attributed to the larger number of data points being used in the optimisation cases, Bayesian approach tends to be slower of the two and hence lesser data points are used, accuracy can increase if proper data is used.
• Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package perform the best.
• Another algorithm being used is the QuadDIRECT algorithm, it gives very good results in the shorter problem case but doesn't do very well in the case of the longer problems.
• The choice of global versus local optimization make a huge difference in the timings. BBO tends to find the correct solution for a global optimization setup. For local optimization, most methods in NLopt, like :LN_BOBYQA, solve the problem very fast but require a good initial condition.
• The different backends options available for Bayesian method offer some tradeoffs beteween time, accuracy and control. It is observed that sufficiently high accuracy can be observed with any of the backends with the fine tuning of stepsize, constraints on the parameters, tightness of the priors and number of iterations being passed.

Interactive Notebooks

To run the tutorials interactively via Jupyter notebooks and benchmark on your own machine

1. Run Weave for the file (or folder) you are interested in
2. Activate the appropriate environment
3. Open and run the notebook.

Note: Since notebooks default to looking for a Project.toml file at the same level or parent folder, you might need to move the notebook to the folder with the appropriate Project.toml.

Example (starting from the project root folder)

]activate .
]instantiate
using SciMLBenchmarks
SciMLBenchmarks.weave_file("benchmarks/Jumps", "Diffusion_CTRW.jmd", [:notebook])
]activate benchmarks/Jumps

Then move Diffusion_CTRW.ipynb to "benchmarks/Jumps" and open the notebook.

Contributing

All of the files are generated from the Weave.jl files in the benchmarks folder. To run the generation process, do for example:

]activate SciMLBenchmarks # Get all of the packages
using SciMLBenchmarks
SciMLBenchmarks.weave_file("NonStiffODE","linear_wpd.jmd")

To generate all of the files in a folder, for example, run:

SciMLBenchmarks.weave_folder("NonStiffODE")

To generate all of the notebooks, do:

SciMLBenchmarks.weave_all()

Each of the benchmarks displays the computer characteristics at the bottom of the benchmark. Since performance-necessary computations are normally performed on compute clusters, the official benchmarks use a workstation with an Intel Xeon CPU E5-2680 v4 @ 2.40GHz to match the performance characteristics of a standard node in a high performance computing (HPC) cluster or cloud computing setup.

Inspecting Benchmark Results

To see benchmark results before merging, click into the BuildKite, click onto Artifacts, and then investigate the trained results.