Meeeeester James Steinberg claims:
Until you are making a million a year in revenue, it is a waste of time to write tests.
We must accept this argument from authority! Señor Steinberg has imploded a startup, and sold others off (although with no mention of total internal return, but let’s not let boring old notions from finance get in the way of opinions from lottery winners)!
If you’re the kind of person who likes to play scratch-its with your career and production stability, it’s entirely reasonable to test everything by hand every time you change anything; that’s the mindset I’d expect from someone playing with scratch-its.
Those of us who make calculated bets instead of playing the startup lottery, who understand alien technologies like expected value, total internal rate of return, and compounding returns take a markedly different approach from the throw-up-a-landing-page-and-sell-a-startup-on-the-basis-of-mailing-list-signups crowd. In that we make investments that pay accelerating dividends. Continuous Delivery as a superpower, you know? If only we could inquire with Herr J. S.: “What about continuous delivery? Is that for millionaires as well? What tooling is worth investing in? Should I just tar up my php dir and shit it directly onto my Digital Ocean VPS?”
Not writing tests is a false economy. “Run through your service/app/site manually as a user, sure”, the man says. But how many times are you to do that? If you’re working on a change deep in a user flow, will Steinberg’s Methodology of Virtuous Sloth force you to click through ten screens to validate each change? If that’s your workflow, what amount of time have you lost being unable to programmatically determine if your changes are working? Moreover, how do you develop confidence that the rest of your system remained functional? Do you click through that as well?
Investing in tests is daunting, and if it’s not a set of muscles one’s in the habit of flexing (and been forced to live without), the upside is entirely opaque. It’s the less-than-glamorous engineering side of our work. One has to have intuition for what to test, and how to test it effectively. Setting up testrunners is a laborious and sometimes arduous process, all the more so the longer the project’s been delayed. It’s the work we do when we’re not delivering features so that we can deliver features at a serious cadence. It’s the work we do in the engine room, emerging greasy and glorious in the eyes of our fellow engineers, but a complete mystery to our colleagues in Product. A failure to write and automate testing at the outset is just as expensive as failing to write deployment automation at the outset.
And no matter how fast you feel like your engineers are going today, on a greenfield project, with no tests, you will hit the ceiling on throughput quickly, as manual testing and regression fixes begin to absorb your entire engineering team’s attentional budget.
Tests, just like deployment automation, is an investment with compounding returns. If your engineers universally have confidence that they can make changes and confirm the change works in very quick loops, they will universally move more quickly than if they’re testing every change by hand. If you want to onboard new engineers and they have to boot your codebase laboriously and manually every time, and start clicking around to see if their changes work and to understand if they’ve broken anything, you’ll lose time there. Your new engineers likely won’t even know the critical flows to ensure they haven’t broken, and your engineering leads will have to assign them low-risk tasks because you don’t have guards against fixing things, and then your merge gates will similarly have to involve manually testing everything. For every single change.
“Tech debt” feels like a cutesy name, but it’s also more accurate than a tightly-collimated beam. As you rack up debt (in terms of unreadable and cross-cutting implementations), the cost of making each change will increase. Adding another engineer to the project will slow the whole thing down as they extract tribal knowledge from their colleagues. As your featureset expands, the amount of manual testing necessary to validate every change increases, and you have a combinatorial explosion to keep an eye on as your engineering team grows and wants to merge and validate more changes on an ongoing basis.
Perhaps the good mister is right after all, but let’s give it some more color. Tests are for engineering teams who want to become millionaires slowly, and then quickly. This is precisely the Get Rich Slowly approach to becoming a high performing engineer. It’s not enough to just have 5 years of experience in React, that just makes you a React developer with 5 years of experience riding a surfboard on the shifting sands of JS. Strong engineers a) have experience in multiple contexts (and no, Next in addition to vanilla React doesn’t count), b) know how to collaborate on software in a team context and c) build systems to take their code from development to production without tears.
A strong engineer is a millionaire in competence. They codify what would otherwise be implicit project knowledge in tests. They start the project with tests, so that it’s already cleanly factored and prepared for automated testing throughout its lifetime.
But perhaps this perspective is too alien. If you only work with folks with 3 years of writing JQuery and 2 writing React, it’s likely that they won’t be competence millionaires out of the gate. You’ll hire them for their ability to smash out changes in JS, not for their broad mastery of the collaborative software development process. Is this really what you want, though? Just because you’re smashing features out to get your product to the magical flywheel point where user-to-user recommendations bring more users into your funnel than you lose in the churn doesn’t mean that you can afford to add features that reduce churn. Failing to consider the impact of quality and reliability on churn is that classic myopic startup mentality, that eg “only MRR matters” or “only signup rate matters”, or that “if we just fix the top of the funnel, our churn rate won’t matter”.
Churn matters. The experience of quality matters. Users lost to a broken experience that nobody realized was broken because there’s no automated testing matter. High velocity in software development matters, and making investments that compound their return over time is the only way to get there. There are no silver bullets (in software or any other production process), there are only a complexly interrelated set of tools that leveraged together unlock successively more and more magical engineering workflows. You can’t have preview environments until you have deployment automation. It’s hard to stomach the risk of continuous deployments to production without automated tests and robust monitoring. QA staff will balloon in count unless you teach them (and insist that they!) automate their work; and you’ll have to invest in tooling to support their automation until you have the budget for dedicated testing engineers.
So put it another way: tests are necessary but insufficient to become a millionaire, unless you’re playing the lottery. Just like compounding your personal savings over time is necessary to have a hope in hell of retiring before 75 in the modern American economy, unless you’re into playing the lottery with your career.
Just remember: lotteries are -EV, and winners always overvalue their own influence over outcomes.
On my way to a weekend ceramics party with my wife and children, I got scammed out of a whole sixty dollars! I pulled over to help someone in a late-model Cadillac. I expected that he needed a jump (although who needs a jump on an on ramp?!), but it turns out he wanted some cash. I’m generally willing to help folks out (provided I don’t get skeezed out with meth-head vibes out of the gate), so I offered the dude twenty bucks out of the gate. He then waved a fat sparkly ring at me, and said he’d be willing to sell it. I eyeballed it, estimated the value (if it were 18K gold) at a conservative 500 bucks, and then estimated the likelihood that I was getting scammed at 90%. Then, and here’s the embarrassing moment, I completely fucked up the expected value calculation. Instead of calculating EV as:
