Category Archives: Uncategorized

Tests are for getting rich slowly

Meeeeester James Steinberg claims:

Until you are making a million a year in revenue, it is a waste of time to write tests.

We must accept this argument from authority! Señor Steinberg has imploded a startup, and sold others off (although with no mention of total internal return, but let’s not let boring old notions from finance get in the way of opinions from lottery winners)!

If you’re the kind of person who likes to play scratch-its with your career and production stability, it’s entirely reasonable to test everything by hand every time you change anything; that’s the mindset I’d expect from someone playing with scratch-its.

Those of us who make calculated bets instead of playing the startup lottery, who understand alien technologies like expected valuetotal internal rate of return, and compounding returns take a markedly different approach from the throw-up-a-landing-page-and-sell-a-startup-on-the-basis-of-mailing-list-signups crowd. In that we make investments that pay accelerating dividends. Continuous Delivery as a superpower, you know? If only we could inquire with Herr J. S.: “What about continuous delivery? Is that for millionaires as well? What tooling is worth investing in? Should I just tar up my php dir and shit it directly onto my Digital Ocean VPS?”

Not writing tests is a false economy. “Run through your service/app/site manually as a user, sure”, the man says. But how many times are you to do that? If you’re working on a change deep in a user flow, will Steinberg’s Methodology of Virtuous Sloth force you to click through ten screens to validate each change? If that’s your workflow, what amount of time have you lost being unable to programmatically determine if your changes are working? Moreover, how do you develop confidence that the rest of your system remained functional? Do you click through that as well?

Investing in tests is daunting, and if it’s not a set of muscles one’s in the habit of flexing (and been forced to live without), the upside is entirely opaque. It’s the less-than-glamorous engineering side of our work. One has to have intuition for what to test, and how to test it effectively. Setting up testrunners is a laborious and sometimes arduous process, all the more so the longer the project’s been delayed. It’s the work we do when we’re not delivering features so that we can deliver features at a serious cadence. It’s the work we do in the engine room, emerging greasy and glorious in the eyes of our fellow engineers, but a complete mystery to our colleagues in Product. A failure to write and automate testing at the outset is just as expensive as failing to write deployment automation at the outset.

And no matter how fast you feel like your engineers are going today, on a greenfield project, with no tests, you will hit the ceiling on throughput quickly, as manual testing and regression fixes begin to absorb your entire engineering team’s attentional budget.

Tests, just like deployment automation, is an investment with compounding returns. If your engineers universally have confidence that they can make changes and confirm the change works in very quick loops, they will universally move more quickly than if they’re testing every change by hand. If you want to onboard new engineers and they have to boot your codebase laboriously and manually every time, and start clicking around to see if their changes work and to understand if they’ve broken anything, you’ll lose time there. Your new engineers likely won’t even know the critical flows to ensure they haven’t broken, and your engineering leads will have to assign them low-risk tasks because you don’t have guards against fixing things, and then your merge gates will similarly have to involve manually testing everything. For every single change.

“Tech debt” feels like a cutesy name, but it’s also more accurate than a tightly-collimated beam. As you rack up debt (in terms of unreadable and cross-cutting implementations), the cost of making each change will increase. Adding another engineer to the project will slow the whole thing down as they extract tribal knowledge from their colleagues. As your featureset expands, the amount of manual testing necessary to validate every change increases, and you have a combinatorial explosion to keep an eye on as your engineering team grows and wants to merge and validate more changes on an ongoing basis.

Perhaps the good mister is right after all, but let’s give it some more color. Tests are for engineering teams who want to become millionaires slowly, and then quickly. This is precisely the Get Rich Slowly approach to becoming a high performing engineer. It’s not enough to just have 5 years of experience in React, that just makes you a React developer with 5 years of experience riding a surfboard on the shifting sands of JS. Strong engineers a) have experience in multiple contexts (and no, Next in addition to vanilla React doesn’t count), b) know how to collaborate on software in a team context and c) build systems to take their code from development to production without tears.

A strong engineer is a millionaire in competence. They codify what would otherwise be implicit project knowledge in tests. They start the project with tests, so that it’s already cleanly factored and prepared for automated testing throughout its lifetime.

But perhaps this perspective is too alien. If you only work with folks with 3 years of writing JQuery and 2 writing React, it’s likely that they won’t be competence millionaires out of the gate. You’ll hire them for their ability to smash out changes in JS, not for their broad mastery of the collaborative software development process. Is this really what you want, though? Just because you’re smashing features out to get your product to the magical flywheel point where user-to-user recommendations bring more users into your funnel than you lose in the churn doesn’t mean that you can afford to add features that reduce churn. Failing to consider the impact of quality and reliability on churn is that classic myopic startup mentality, that eg “only MRR matters” or “only signup rate matters”, or that “if we just fix the top of the funnel, our churn rate won’t matter”.

Churn matters. The experience of quality matters. Users lost to a broken experience that nobody realized was broken because there’s no automated testing matter. High velocity in software development matters, and making investments that compound their return over time is the only way to get there. There are no silver bullets (in software or any other production process), there are only a complexly interrelated set of tools that leveraged together unlock successively more and more magical engineering workflows. You can’t have preview environments until you have deployment automation. It’s hard to stomach the risk of continuous deployments to production without automated tests and robust monitoring. QA staff will balloon in count unless you teach them (and insist that they!) automate their work; and you’ll have to invest in tooling to support their automation until you have the budget for dedicated testing engineers.

So put it another way: tests are necessary but insufficient to become a millionaire, unless you’re playing the lottery. Just like compounding your personal savings over time is necessary to have a hope in hell of retiring before 75 in the modern American economy, unless you’re into playing the lottery with your career.

Just remember: lotteries are -EV, and winners always overvalue their own influence over outcomes.

Tooling and workflows for high-output engineering teams (and their stakeholders)

[1]I like that “stakeholder” imparts this image of someone holding a pointy wooden bit to ones chest…My team at SeaWorld is, depending on how you look at things, either stretched ridiculously thin, or incredibly efficient at what we do. We’re responsible for: branded toolchains on around iOS and Android codebases (producing builds for each of our brands on each platform from two distinct codebases); a handrolled API gateway frequently described as “the mobile API”, which caches CMS content and abstracts over several different; and a Next.js application that gets variously webviewed into the native apps and delivered on its own. We get all that done, delivering not no features, and all on a team of about 10 engineers, 1.5 product people, 1 designer and 1 QA (and no project managers, if you can believe it! Jokes, I want a PM like I want a static compiler for Python). I contest that the only thing that makes this remotely possible is our continuous deployment automation, and I further assert that focusing on engineering productivity is an excellent locus for IC attention as you move up in seniority and into management/the chain of command. Join me for a tour of the last two years of evolution in CI and CD with my brand new team.

Continue reading Tooling and workflows for high-output engineering teams (and their stakeholders)

References

References
1I like that “stakeholder” imparts this image of someone holding a pointy wooden bit to ones chest…

Happy Imbolc!

I personally turn into a dead zombie of death over the winter months. Living at the 45th parallel in the stupid little microclimate of the Portland metropolitan area guarantees that I can’t get a decent amount of UV in my face and so my vitamin D levels drop and I have to (literally) dose myself with reptile lamps in order to keep my immune system functional and from falling into a deep well of emotional pain for 6 months (it takes ~3 months after summer ends for me to deplete my vitamin D reservoirs and for my energy levels to begin drooping).

While some folks like Christmas, for me it always spells the beginning of the Season of Doom and Gloom, and I much prefer what happens at the end of the holiday season, after the solstice and as we go into spring with days lengthening (but before the sun god even starts thinking about showing us his hallowed face again.

Which is why I light the Christmas tree on fire every year.

Happy Imbolc!

Rocket Stove

I built a rocket stove. Tallulah helped me cram it full of firewood.

I’m not super pleased with how little flame it produces but that’s the efficiency/vroom tradeoff at work.

The perfect cup of coffe

The perfect cup of coffee is perfectly repeatable. Variables are constrained, and you enter a regime of hunting local maxima. Small and controlled changes to understand their impact on your brew.

There is no perfect cup, everyone has their own preferences on oils and particulate matter and so on. Some folks love the rich mouth-feel of a cup from the French press, other folks like it strong out of the Aeropress, some folks won’t touch it unless it came from a ten-thousand dollar espresso machine lovingly kept in tune all day by a barista with a Master’s of Social Work.

Here’s how I make coffee, in the most repeatable and reliable fashion. Minimal tooling, minimal overhead, minimal complexity, maximal repeatability.

Tools:

  • burr grinder
  • Clever coffee dripper

Instructions:

  1. Measure out and boil 12 ounces of water.
  2. As soon as the water boils, turn the kettle off, weigh out and grind 20 grams of fresh beans.
  3. Line the Clever with a paper filter
  4. Dump ground beans into the Clever
  5. Pour in just enough water to soak the grounds (this allows the CO2 to “bloom” off, and prevents the grounds from floating and not soaking fully in the brewing medium)
  6. Bloom the grounds for 1 minute
  7. Pour the rest of the water into the Clever
  8. Soak grounds for another 3 minutes
  9. Decant

Simply, perfectly, fool-proof-edly repeatable.

Get 3 Clevers, because they’re guaranteed to go out of production in the next decade, and then you’ll be stuck doing pour-overs again, with all the asinine variance that imposes on the process.

Sleep training (cues, backoff, kindness)

The child misses you, her constant companion. One of you is always there. Close in physical touch, near at hand for a fall or a fear or to laugh at a fart. Night comes (sweet one, for us all, sooner than we wish, but prepare thyself for it to take everyone and everything you love; you before them if not them before you. Death comes for us all, oh tiny love, remember that and dance!) and it is dark. So dark. Dark because “dark helps me sleep”, we say, for in truth light regulates the circadian, and oh my sweet thou needst it bright in the morning and dark in the evening. These environmental cues, these rhythms to the day, this is what life is made of. To wake, to see the light, to play and run and fight and scream, to watch the sun descend, to clear the table and prep to eat. To sit with family, to cherish thirty minutes of dedicated togetherness before, yes, before we brush our teeth and hair and go to bed.

But surely we are not just to stay abed, are we? I simply cannot! And you cannot make me!

Yes sweet one, we are to stay abed. Canst thou not lie quietly and go to sleep? This is our mandate from heaven; or at the very least your mandate from me: go the fuck to sleep.

I know, I know. Should I chastise thee? No, anger is no tone on which to bed down sweetly. Entering the dark of the night, from which we cannot know that we are who we were before we closed our eyes, is not something that anyone should ask anyone else to do with heightened emotions. How are we to square this circle? Sweet one, full unknowing of the joys of  uninterrupted sleep, why should you sleep? Why should you stay abed? Why not pile out full of sunbeams and comical utterances, demanding more of the love and attention that otherwise defines every moment of your life?

Because we all must sleep, beloved. We all must build a practice for ourselves (I for you, as with so many things) of winding down to a mood where sleep comes quickly, and where rest carries us deeply through the night. I must sleep. And before I can sleep, you must sleep (motherfucker). Cleaving you from me, and I from you is pain and sorrow. A distance unbearable for flesh of my flesh. I killed alternate selves and sunk ghost ships of my own life to be here for you, in the interstices of other obligations. And to part? To sleep? Intolerable, I know.

How about this, child of mine, love unparalleled: you lie in your bed for a bare thirty seconds, and I’ll come back. I always come back. Watch! I’ll be back in a mere half minute.

Oh dear, you emerged. “Didn’t make it”, did you? It’s fine, expected. In what world, and at what cruel hand would we ever willingly spend time apart? (I tell you, you know it not now, but even our love, tiny one, does not extend to your intruding upon my personal time after you go the fuck to sleep). Come, let me carry you back to bed, tuck you in again. I love you so much! May I kiss your cheek? Hum the refrain to your bedtime song again? Darling, I’ll return oh so soon. Five seconds this time. Something achievable, something you can do. An easy victory, all I ever want for you (at least when we’re working on hard things together…).

Good night my love! I’ll check on you again in, shall we say, seven seconds? Seven arduous, agonizing seconds? Agonizing for you, fraction enormous of your life. No sense of time, it must weigh upon you like a blanket of eternity itself. When will he return? Will he? I hope he does. Will he? Will…he?

Of course I do. See? Goodnight, little one. I love you more than the moon and the stars, more powerfully than the thermocline and the jetstream. I’ll come back and check in on you in thirty seconds this time, do you think you can make it?

And so it goes. Greater gaps every time. If someone has trouble staying in their bed, reduce the gap until you find what they can handle. Sometimes check-ins happen at 5 second intervals while you build trust with the little one that you will return to check in on them.

The gaps naturally extend. They want to build these muscles. As much of a trap as it is, they want to please you (they also want to please themselves, and salve their own anxiety, so love them when they pop out to say hello before your timer goes off). You will get to the point where that first check-in after the door closes is 2 minutes, then 3, then 5, and then ultimately the door closes and they go to sleep on their own.

No tears, no fear, no anger, no yelling, no shame. Just love and sympathy and caring. We always come back, don’t we? It’s a foundational message, and trust never comes for free but is earned through sweat and always following through.

Flatland Skateboard Training Program

Here’s the training program that I’m currently working. I believe that it’s about on the money for someone with 3-5 years of dedicated practice and 95th-percentile core physical fitness. This is not meant to get you oodles of new tricks, it’s the drill-and-mastery oriented practice practice practice approach to emulating the endless time and unswerving focus of the 15-year old.

In addition to a minimum of 1 hour skateboarding every day:

  • Alternate daily between (40 minutes):
    • lower body (squats, heel raises, weighted vertical jumps, weighted lateral jumps, weighted single-leg jumps, weighted 180º jump-rotations)
      AND
    • upper body (exclusively on the olympic rings: push up, plank, mountain climbers, dips, pull up progressing to full muscle up, lower leg-raise progressing to full pike)
  • minimum 30 minutes of yoga per day (it’s pretty hard to beat Yoga with Adrienne)

Skate routine (everything rolling. don’t drill tricks not rolling, you’ll handicap yourself brutally. Ask how I know):

  • Ollies.
  • Shuvits, frontside and backside. Progress to pop shuvits. Progress to tre flips.
  • 180s, frontside and backside. Progress to bigspin. Progress to biggerspin.
  • Kickflips
  • Heelflips
  • Tailslides and noseslides, frontside and backside. Progress to shuv out. Progress to kickflip in. Progress to kickflip out.
  • 5050s. Progress to kickflip in. Progress to kickflip out.

Full-routine progressions:

  • Do everything over a parking bumper.
  • Do everything over 2 stacked parking bumpers.

Nothing in life is free, nothing worth having is easily got. Get out there and motherfucking drill.