How we cut our build from forty minutes to nine

Field notes from the platform team · 6 min read

For two years nobody touched the build. It worked, it was slow, and every engineer had quietly arranged their day around it: start a build, make coffee, answer messages, come back, start again.

This spring we finally measured it. What follows is what we found, what we changed, and the one change we would skip if we did it again.

Where the time went

We expected the compiler to be the villain. It was not. Compiling the whole monorepo took eleven minutes on a cold machine; the other twenty-nine were spent downloading, unpacking and testing things that had not changed since the previous run.

A single trace of one build, laid out on a timeline, made the case better than any meeting could. Long grey bars of waiting, short coloured bars of work.

Three things accounted for almost all of it: a dependency download that ignored its own cache, a test suite that ran every test on every change, and a packaging step that rebuilt images nobody had asked for.

The fastest build step is the one you can prove you do not need to run.

Caching the right things

Our first cache stored the output of every step under a key made of its inputs. It helped less than we hoped, because the keys included timestamps, absolute paths and the version of a tool that changed every night.

Once the keys described only what a step actually reads, the hit rate went from a third to nine in ten.

key = hash(sources, lockfile, toolchain)
if cache.has(key):
    restore(key)
else:
    run_step()
    cache.put(key, outputs)

We also stopped caching things that were cheaper to rebuild than to download. Small steps with large outputs were slower from the cache than from scratch, and they crowded out the entries that mattered.

The lesson is boring and true: a cache is only as good as the honesty of its key.

Splitting the tests

The test suite ran on one machine, in one long line. We split it by measured duration, not by folder, so that eight machines finish within seconds of each other.

• Slow tests are spread first, fast ones fill the gaps.
• Each shard reports how long it took, and the next split learns from it.
• A flaky test is quarantined automatically after two failures.

Nine minutes of tests became seventy seconds, and the quarantine list became the most honest document in the company.

Remote execution

The last big step was to stop building on laptops at all. Actions run on a shared pool of machines close to the cache, and a laptop only sends what changed.

The first week was rough. Tests that quietly depended on a file in someone’s home folder failed one after another, and each failure was a small archaeology project.

This is the part that took longest and saved least. It is worth it for a large team; for us, the cache and the split tests had already done most of the work.

What we would skip

If we started again we would measure first, fix the cache keys second and split the tests third, and only then ask whether remote execution is still needed.

The build now takes nine minutes on a bad day. Nobody makes coffee during it any more, which, we are told, is its own kind of loss.