Methodology notes
A failed query counts in the geometric mean with a penalty ratio: twice the worst time posted for that query by any compared solution, divided by the baseline’s time. Queries the baseline fails are left out. A solution whose run failed before producing any result is listed with its exit code instead of a score.
Times are wall-clock times of each query; the cold run follows a cache flush, the hot runs repeat the query immediately.
Benchmark methodology: see Methodology and motivation(opens in new tab).