TLDR
- Our code sets prices on live Booking.com and Airbnb listings and Claude writes most of it, so every deploy runs the whole suite of about 53,000 tests.
- On the price list Ubicloud is about 5x cheaper than Depot per vCPU-minute, but its vCPUs are hyperthreads, so per test it’s only about 3x, which is why I compare runners by what a test costs.
- testmon only sees Python and missed most of the failures from SQL changes, so our own plugin also tracks SQL, GraphQL and fixtures, and one small PR ran 18 tests instead of about 48,000.
- A quarter of PR runs still run the whole suite, and it’s far too easy to run up massive CI bills with agents, so I keep cutting what runs.
Most software for short-term rentals is about operations, like messaging guests, cleaning and check-in. We do the revenue side, so our code sets prices, promotions and discounts on live Booking.com and Airbnb listings. Claude writes most of it now (I wrote about how in The IDE is dead), so tests are how I know it works.
Why tests matter for a pricing tool
A bug in our code can put a wrong price on a live listing, and if a guest books it before anyone notices then that’s the price they pay. So every PR that touches the backend reruns the tests on the code that decides prices and promotions, and fails if they cover less than 80% of it.
Coverage only tells you the code ran, so we also do mutation testing, where a tool changes the discount and price maths in about 1,150 small ways, one at a time, and the tests have to fail for each one. Right now they catch about 90%.
Every deploy runs the whole suite, which is about 53,000 tests now and has doubled since early September. We run tons of agents in parallel, each pushing to its own PR, so CI also turned into a real bill. On Blacksmith we went from about $130 in June to being on track for nearly $2k a month by August.
We’ve used four CI runners since June
We moved from GitHub’s own runners to Blacksmith in June. In August I found that Blacksmith rounded every job up to a whole minute, and most of our jobs run for seconds, so we paid for a lot of waste, especially when runs got killed. The rounding was over a quarter of the bill. Depot bills per second and it was a one line switch, so we moved there.
Ubicloud is 5x cheaper than Depot per minute but 3x per test
In November we moved again, to Ubicloud. On the price list its standard runners are about 5x cheaper per vCPU-minute than Depot’s. A vCPU is the virtual CPU you rent, and on Depot each one was a full core, but on Ubicloud it’s a hyperthread, i.e one of two threads sharing a physical core, so 8 vCPU is really 4 cores. The same test takes longer there, so per test Ubicloud is only about 3x cheaper.
In the first two days a lot of test jobs hit their time limits, which hadn’t been happening on Depot. So we moved PR runs from 4 vCPU to 8, which costs about the same per run because it’s twice the price per minute for about half the time. I compare runners by what a test costs now, because the price per minute doesn’t tell you how much work you get done in that minute.
testmon only sees Python, so we track SQL changes too
We use testmon, a pytest plugin that records which code each test runs and then picks only the tests whose code changed. It only sees Python though, and when we replayed one merge it missed 9 of the 13 real failures that SQL view changes caused. So our own plugin on top also records the SQL, GraphQL, fixtures and product guides each test reads.
In late October 8 out of 10 PR runs were still running the whole suite, about an hour each, because every PR changed something global like the settings. A PR’s first run now starts from the selection recorded on main, so a test PR that changed one helper and one test ran 18 tests instead of about 48,000.
A PR run that picks its tests takes about 10 minutes now. Changes the selection can’t follow, like dependencies, test settings or CI itself, still run everything, and a quarter of PR runs last week ran the whole suite at about 35 minutes each. That’s probably bad and needs optimizing later.
Every deploy still runs everything
Picking tests can miss things, so every deploy runs the whole suite and nothing goes live until it passes. As the suite grew, merge to live went from about 20 minutes in early September to over 40 in early October (the chart at the top), and the tests were the biggest part of that.
So we split the deploy’s tests over four machines. The parts are planned by whole test file from last time’s durations, so a file’s fixtures (the test data it sets up) build once on one machine, and the deploy fails if the parts don’t cover every file between them. That roughly halved the test time for about 50% more compute, since each machine does its own setup. Merge to live is about 30 minutes now, still slower than in early September because the suite has doubled and the vCPUs are slower.
Small fixes that added up
- Work stealing, where a test worker that runs out takes tests from a busy one, plus one worker per vCPU, cut the test run by about a quarter.
- Postgres with fsync off, so it doesn’t wait for writes to reach the disk, which is fine because the test database is thrown away anyway.
- Our MCP server’s tests ran twice, in the main test job and in their own coverage job, and on some PRs that second copy was most of the main job’s work.
- Agents open PRs as drafts, so the expensive checks wait until a PR is ready.
- We deleted automerge, which took about 40,000 CI runs for fewer than 300 merges, so now agents review and a human merges.
What’s still not fixed
When an agent pushes again, the run already going on that PR gets cancelled, and in our first few days on Ubicloud those cancelled runs were about a quarter of the daily spend. A few tests that depend on Postgres query plans still fail now and then too.
Even on Ubicloud, CI with this many agents came to about $400 for the first 10 days of November, which is more than I’d like, so I have to keep cutting what runs. I’ve set a spend alert on it too, because it’s far too easy to run up massive CI bills now with agents.
