cicdbench · independent CI runner benchmark · pilot round 1

Round 1 results

Round 1 · Nov 16 to 17, 2026
Per provider · each card says which runs it covers

Leaderboard

One tie rule throughout: values equal at the precision shown share a rank (times to 0.01 min, green rates to 1%, queue wait to 1 s, cost to 1 cent), and a tie for the fastest run gives each tied provider the win. A provider with no data for a metric is listed without a rank.

After a small edit · the run teams do most

Time to green per repo, edit run

From the push returning to the last check finishing, read from GitHub's timestamps, in minutes. Bars are scaled per repo: each card has its own scale. A failed run has no bar and no rank. Grey GitHub rows are a reference from the first group only (see method) and are not ranked.

Every run

Cold, warm and edit, in minutes

● marks the fastest green run in each row; runs tied at 0.01 min both get it. * marks a partial run, which is never ranked. Cold: new branch and PR, which starts with an empty branch-scoped cache. Caches that are shared across a whole repo were not verified empty for round 1 (some repo copies had run setup checks before it); the protocol for later scored rounds is to clear them first and record how; for the 2026-11-17 reruns a cache reset was not verified. Warm: the same commit re-run. Edit: one comment line pushed to a core source file.

Cost

What a run costs at list price

Billed job time under each provider's own rounding, at list price: Depot $0.012/min, Luxor $0.008/min, Instabox $0.034/hour, Boat $0.036/hour. Boat's rate is an estimate: Boat only lists its 2 vCPU size ($0.018/hour), so the 4 vCPU size is assumed to cost double. Boat's and Instabox's rates still have to be checked against an invoice. Plan fees not included. The per edit run column is cost estimated from measured job time at these rates (not from invoices), including runs that failed or were partial (labelled). The 1,000-run column weights 80% edit, 15% warm and 5% cold runs, and is projected only where all three runs were green and complete; otherwise it shows a dash and why, since a failed run's cost says nothing about a passing one.

Failures

What broke, and whose fault it was

Method and limits

Read these numbers as a pilot