It’s Never Too Early To Do The Right Thing

Is This Thing On?

Welcome to the inaugural post of the KUKA ATX Tech Blog.  In this blog, we’ll hear from the developers on the KUKA ATX team who work on KUKA Connect and related projects.  We’re pretty proud of our people and our technology, and we’re eager to share it with the world.  We also think this is a pretty great place to work, and this blog might give you an idea of the types of technologies and culture you’d be exposed to if you were to join us. I’ll start by introducing myself.  My name is Charles Crain, and I am the lead architect of the cloud team here.  I’ve been writing software professionally for nearly 20 years, and I’ve worked on everything from real-time and embedded software to desktop applications to compilers to large scale distributed applications.  Writing high-quality software with a talented team of motivated folks — and having a ton of fun doing it — is one of my favorite things.  I sometimes can’t believe they pay me for this.  Did I say that out loud?

The pressure of coming up with a worthy topic for a first blog post was pretty intense.  I spend a lot of my time reading, talking, and thinking about how to write software better.  Better than what?  I guess, better than I’ve done it or seen it done in the past.  Better — I think — than most folks do it.  If I met someone with more than 5 years of experience in software development that hadn’t been involved in a project that they absolutely hated, I’d be surprised.  If I could find anyone in the modern world that hadn’t used a piece of software they’d absolutely hated, I’d be even more surprised.  I conclude that most people (including myself) are pretty bad at writing software.  One of my life goals is to be part of as many software projects as I can where the team feels like we’re generally “doing the right thing.”  I intuit that a development team that feels that way almost can’t help but deliver delightful products to their customers.  So far, I have yet to be proven wrong, but only because projects where the team feels that way are so rare.  I hope to change that.

Lots Of Graphs That Say The Same Thing

One of my favorite blog posts ever on the subject of writing effective software is The Solution To Technical Debt.  I admire the ambitious title for one thing, but I also like the way it concisely states a central conundrum of writing good software.  Namely, doing the right thing almost always slows you down right now compared to doing the wrong thing.  This makes doing the right thing a hard sell (particularly to the short-sighted), even though the long-term payoff might be massive.  The author illustrates this point with two graphs, that I really think are the same graph drawn two different ways:

Both of those graphs show two lines: velocity when doing the right thing (stopping writing crappy code and cleaning up old crap, respectively) and when doing the wrong thing (keeping writing crappy code and keeping old crap).  In both cases, doing the right thing is shown to be hard at first but breaks even with doing the wrong thing at some point and pays off thereafter.  If you search the web for images like “technical debt graph” you turn up many similar images from similar posts:


I stared at graphs like these for a long time.  They all say the same thing, essentially.  At some point, you have a choice of whether to do something one way or the other.  One way looks good for a while then peters out.  The other looks tough for a while then gets awesome.  In every case, there is this interesting area in the graph that seemed to me to contain some sort of holy grail to doing software the right way, or at least the keys to unlocking why so many teams don’t.  The area I am talking about is the part where doing the wrong thing seems like a better decision than doing the right thing.  You know… I realized that any time I’ve been part of a decision not to do the right thing (that I later regretted), it had always stemmed from a perception that this area was “too big.”  I’ve been part of too many discussions to count that have gone something like, “we all agree X is the right thing to do, but some of us think it’s too costly to justify doing right now.”  Every time the “X is the right thing, just not right now” camp has triumphed, X never happened, not once.  I have been part of a project where we consciously decided to delay doing the right thing — then scrupulously came back and did it when the time was right — zero times.  And I am not talking about trivial things like “I’ll comment this code later” or “I’ll shore up the test coverage of this class after next iteration.”  I am talking about stuff that’s big enough to require discussions about whether to do things a certain way across an entire team.  Whether to run iterations of this or that length, how to even define what it means to be done at the end of an iteration, how to write effective automated tests, how to evolve and maintain cross component APIs, etc.  For decisions of any size greater than trivial, it seemed, a decision not to do it now was — for all practical purposes — equivalent to a decision never to do it.  For years, I wondered at why this was.  Then, after staring at graphs like the ones above, it became clear:

  • That area — the area where doing it wrong is faster and better than doing it right — is your technical debt.  It’s the stuff you have to “clear out” before doing it the right way is also the natural way.
  • Technical debt, like financial debt, grows exponentially.  It makes more of itself at a rate proportional to how much of it there is.
  • Therefore, there is no such thing as waiting for the situation to get better all by itself.  On the contrary, an exponentially growing problem gets unmanageably large with breathtaking speed.
  • Ergo, doing it later is logically equivalent to never doing it.
  • The kicker: all projects start with zero(*) technical debt.

What All These Graphs Really Say

Here is my version of all these graphs from above.

Any time you have a decision in a software project where you can do the expensive-but-right thing or the cheap-but-wrong thing, this is the math you’re faced with.  You have some amount of “cleaning up” or “prep work” to do before the right thing is as easy or easier to do than the wrong thing.  From then on, things become awesome.  The only variable is how much work you have to do before you get there, and that work is — exactly — your technical debt, or at least the portion relevant to whatever decision you’re considering.  The more technical debt you have, the longer you’ll spend in the “red zone” before breaking even, which may tempt you to put things off.  However, the more technical debt you have, the more quickly it will grow if you don’t do anything.  Pretty quickly — shockingly quickly in my experience — you may be faced with an intractable choice: spend more time than you have paying off your technical debt, or give up more velocity than you can afford by not paying it off.  At this point, your developers usually start looking for other jobs.

The only reliable solution to this is to start doing the right thing as soon as you reasonably understand what that is.  If you can establish effective platforms and architectural patterns early on, your “red zone” is effectively zero.  There’s never a better time to start doing the right thing than the very beginning.  This is what the graph looks like for a project you are just starting.

I put a (*) next to my assertion above that all new projects begin with zero technical debt, because that’s not strictly true.  The project I’m currently leading involved learning a new language (Scala), several new open source technologies (Kafka, Cassandra, Kubernetes, etc.), and some new architectural patterns (microservices, reactive programming).  We could have chosen not to do that and instead begun the project with familiar languages, technologies and architectures, which would have seemed a lot faster at first but quickly ground to a halt.  However the decision to take the time to lay a proper foundation for the project was never easier or cheaper to make than at the very beginning.

The Real Payoff

There’s another factor at play here that might be even more important than making the right decisions or having an effective architecture.  Teams that spend too long doing the wrong thing don’t just give up the long term gains they’d get from having effective architectures and platforms with low technical debt to build on.  They also lose velocity to miserable developers.  Being good at writing software isn’t just important because we want to write good software; it’s important because we want to have fun doing it.  And developers having fun — who believe in what they’re doing and how they’re doing it — write better software.  The projects we’re doing at KUKA ATX are the first ones in my career that I’ve truly believed in, and when I see the same belief in others, it motivates me to excel.  This is the real payoff of doing the right thing immediately.

More To Come

I touched on a lot of subjects I will get into much more detail in later blog posts.  The phenomenon of exponential growth and how to manage it is something I’ll give a more in-depth treatment for instance, including how that subject touches on the microservice pattern and Conway’s Law.  I’ll also talk about some specifics behind “doing the right thing” and how you can use certain patterns, practices, and philosophies to avoid being blindsided by technical debt.  We’ll also hear from other members of the team on some nitty gritty technical hurdles we’ve had to overcome.  Stay tuned!

Charles Crain
Distinguished Software Architect