Differentiable physics benchmark
An extensible benchmark framework for differentiable physics solvers that treats gradient quality as a first-class criterion alongside forward accuracy and throughput.
What this measures, and why
If you train a model or run an optimizer through a physics simulation, the simulation has to return two things: a correct prediction (the forward pass) and a correct gradient of that prediction with respect to your inputs (the backward pass, or vector–Jacobian product, VJP). Most solver benchmarks only check the forward pass. Mosaic checks both, because a solver that runs fast and predicts accurately is still useless for gradient-based learning if its VJP is wrong, noisy, or numerically ill-conditioned.
Concretely, every solver is asked to compute an output \(y = f(x)\) and the VJP \(\bar{x} = (\partial f / \partial x)^\top \bar{y}\), and we score it on three axes:
Forward
Does the forward solution match a trusted reference?
Gradient
Does the gradient match a finite-difference ground truth?
Cost
How much wall-clock time and memory does each cost?
How solvers are compared
Built on tesseract-core, which wraps each solver in a Docker container exposing a uniform apply / vjp interface, and tesseract-jax, which calls those containers as native JAX functions (with full JIT and grad support) from the benchmark harness. This lets a single harness compare solvers built on incompatible AD backends (JAX, PyTorch, Julia Zygote, hand-written C++ adjoints) without any shared dependency, by talking only to the common apply / vjp interface.
Benchmark domains
The full catalog — per-solver numerical scheme, AD strategy, image name, schema fields, and known limitations — is on the Solver Reference page.
See how the solvers compare
Each domain has a results page with the task setup, pass/fail checks for every solver, and every plot:
Evaluation protocol
All metrics are computed uniformly across solvers through the Tesseract interface. The protocol is solver-agnostic: it operates on apply and vjp and never inspects solver internals. Mosaic reports several categories of metrics:
Setup compatibility. For each solver–task pair: whether the solver produces a usable gradient, fails numerically, or cannot run due to structural constraints (e.g. periodic-only BCs on a channel domain).
Gradient accuracy. Central finite differences (FD) through the Tesseract interface serve as the solver-agnostic ground truth. We never inspect a solver’s AD machinery, only the gradient it returns. Directional derivatives are evaluated along \(K\) random perturbation vectors (\(K = 6\text{--}20\)) and reported as:
- Cosine similarity between the AD gradient \(g_\text{ad}\) and the FD gradient \(g_\text{fd}\) (direction agreement, \(1\) = identical direction)
- Relative \(L^2\) error \(\dfrac{\lVert g_\text{ad} - g_\text{fd} \rVert}{\lVert g_\text{fd} \rVert}\) (magnitude agreement, \(0\) = exact)
The FD step size is swept over \(\varepsilon \in \{10^{-6}, \dots, 10^{-1}\}\) and the minimum error is reported, avoiding both the truncation regime (large \(\varepsilon\)) and the roundoff regime (small \(\varepsilon\)). For Navier–Stokes problems (F2, F3), the full Jacobian is additionally computed and its singular-value spectrum inspected. A wide spread of singular values signals an ill-conditioned gradient that will stall an optimizer.
Gradient through chained calls. Training a model through a solver calls the solver once per interval and differentiates through the whole chain. Its loss compares every interval with a target, so the cotangents each call receives are far smaller than those of the single-call checks. On a forced, turbulent Kolmogorov flow, the gradient of such a loss through 16 solver calls is compared with central differences along 8 divergence-free directions. A VJP that is accurate only for cotangents of order one, for example because its adjoint solves stop at an absolute tolerance, passes the single-call checks and fails this one.
Performance. Forward and VJP wall-clock time, their ratio, and peak memory, measured on each solver’s intended hardware target across 3–4 problem sizes. All timings are averaged over 3 runs after 1 warmup iteration (to pre-populate JIT caches and stabilize memory allocation).
Forward accuracy. Resolution sweep against a reference: an analytical solution where one exists (e.g. the 2D Taylor–Green vortex), the Schäfer–Turek literature values for the 2D cylinder, a converged high-resolution spectral run for the 3D Taylor–Green vortex and the forced 2D Kolmogorov flow, and otherwise the trimmed-mean consensus of all solvers. The Kolmogorov flow is chaotic, so instead of a fixed error bound a solver must converge towards the reference on a ladder of grids, with an observed order of at least 0.8. Physical-law adherence (e.g. a divergence-free velocity field, \(\nabla \cdot \mathbf{u} = 0\)) is checked where applicable.
Optimization convergence. The ultimate test: can you actually optimize through the solver? Each task runs a gradient-based optimizer up to a fixed iteration cap: L-BFGS for H (1000) and F3 (100, projected onto divergence-free fields), Adam for S (2500) and F2 (100, stopping early at a plateau). A solver succeeds if it brings the objective to at most half its initial value. For F2, it must reach at most 70% of its initial drag and end within 30% of the other solvers’ final drag. This catches solvers whose gradients pass the FD check pointwise but still fail to drive a full optimization loop.
Benchmark suites
The CLI exposes four suites, each running a set of experiments:
| Suite | CLI command | Experiments | What it measures |
|---|---|---|---|
| forward | mosaic run --suites forward |
baseline, agreement, physical_laws |
Forward accuracy vs. reference and analytic solutions |
| cost | mosaic run --suites cost |
spatial_cost, temporal_cost, vjp_cost |
Wall-clock scaling with problem size |
| gradient | mosaic run --suites gradient |
fd_check, param_sweep, horizon_sweep, jacobian_svd |
Gradient correctness and conditioning |
| optimization | mosaic run --suites optimization |
domain-specific (e.g. recovery, drag_opt, topopt) |
End-to-end optimization convergence |
Results are saved to mosaic-results/<problem>/<suite>/ as JSON (metrics), NPZ (arrays), and PNG/PDF (plots). Per-domain breakdowns with embedded plots live under the Results menu.