In praise of boring systems

The most valuable technology in my last three jobs was Postgres and a queue. Everything else we ran was, on a good day, a tool we used to ship the work — and on a bad day, a tax we paid to keep the lights on.

This is not the most fashionable position. It is, however, the one I keep arriving at, and I have stopped pretending otherwise.

What “boring” actually means

When I say “boring” I do not mean obsolete, slow, or undeserving of attention. I mean it has been around long enough that you can answer all of these without Googling:

• How does it fail?
• How do I know it failed?
• What’s the recovery story?
• Who in my org has a strong opinion about it that I trust?

A new tool fails in ways nobody has named yet. A boring tool fails in ways someone wrote a runbook about in 2014. The runbook still works. That alone is worth more than everything in the new tool’s marketing site.

The concrete trade

There is a real cost to picking the boring option. You give up the elegance of whatever the new option does better. You sometimes pay in performance. You usually pay in lines of code. You almost always pay in the polish of the developer experience.

In return you get something that is hard to overstate: the ability to solve the actual problem without the system fighting you. The boring system has a thousand StackOverflow answers and a thousand blog posts and a thousand engineers in your network who can help you debug it at 2am. It has accumulated a tail of tools and integrations and migration guides because it is old enough to have been wrong about things, recanted, and recovered.

The new system is one production incident away from a thread that begins “wow has anyone else hit this?”

The four-question test

Before adopting a new piece of infrastructure I now ask, in order:

1. Can Postgres do this badly enough that we can ship today and revisit in a quarter?
2. If not, is the new thing solving an actual problem we have, or a problem we anticipate having?
3. If anticipated, are we anticipating it because the situation requires it, or because the new thing is fun?
4. If we adopt this, what is our exit strategy when it stops being maintained?

That last one is the question that catches the most. A surprising number of “load-bearing dependencies” in any given system are projects with a single maintainer who is two life-events away from never writing code again. You don’t need to avoid those projects. You do need to know that’s what you’re picking, and to be honest with yourself about your capacity to pick up the pieces.

The exception

The exception is when the new thing solves a categorical problem the boring thing cannot — when it is not a 2x improvement but a 100x one, because the new approach removes a constraint the old one couldn’t. This is rarer than people think.

Most “we should rewrite this” arguments are 2x at best. Most “we should switch our database” arguments are 1.5x and a six-month freeze on shipping anything else. Be very honest with yourself about which kind of argument you are in the middle of.

The Postgres-and-a-queue stack is not a destination. It is a default. The right time to leave the default is when you have a specific reason. The wrong time to leave the default is when the new option has a logo you find appealing.