A Taxonomy of Tech Debt (2018)

167 pointsposted 13 hours ago
by jakey_bakey
(technology.riotgames.com)

40 Comments

abc-1

12 hours ago

Contagion is exactly why interfaces are one of the most important pieces of design and should be given significant thought. A beautiful interface with a suboptimal implementation can be easily cleaned up when time is allotted. The reverse is rarely true.

majormajor

3 hours ago

I don't disagree but I think commonly you are missing one of two things that are necessary for a proper design:

1) time to design it 2) knowledge of exactly what it needs to do today and in a year

Sometimes you're missing both.

In which case I think you can prevent contagion from being too terrible by enforcing smaller modules and single responsibility in a compositional way. That doesn't require as much knowledge of the future or time, but just requires you to avoid high-surface-area interfaces that end up with lots of behavioral variants controlled via parameters in a nesting-doll style. Instead, move your config/parsing/behavioral decisions to the edges of your logic instead of letting them seep into all your underlying models too.

appplication

2 hours ago

Agree, but I’ve found designing robust, future proof interfaces to be one of the hardest problems in developing software. Even intentionally setting out to avoid tech debt at all costs, it’s just hard to do correctly. It requires more than technical bravado and architectural vision. It really does get into the realm of predicting the future.

abc-1

2 hours ago

Look at how mathematicians build minimal yet complete definitions for inspiration. An algebraic system can be created with a set of operations such as multiplication and addition, and existing concepts can be mapped to this system, such as money, but the underlying algebraic system will never change. It is complete.

Much of the system can be complete like this with forethought. The pieces that cannot can be factored out to the edges.

appplication

an hour ago

You’re not wrong in a theoretical sense, but building useful interfaces that your average dev can grok enough to build on top of requires higher level abstractions, approximations, and “reasonable defaults”. My experience is that only a small number of devs actually well understand the codebases they work in (and care enough to be thoughtful in interfacing with it).

The majority of devs generally are happy to tack on their features and PRs to whatever random scaffolding they can, without regard or awareness for how their individual component fits into the larger system, or how it may be extended. And to be honest it’s not necessarily a bad thing, because they do need to get work done, and merging PRs shouldn’t be reserved for the enlightened.

I guess I’m just pessimistic. The reason we don’t see perfect software is because we are not capable of producing it. At a certain point it all becomes spaghetti. If you work with software that isn’t spaghetti, it’s only because the people who care about it not becoming spaghetti haven’t left yet. This is good, but eventually they will leave, standards will decline, and you will become one with the pasta.

abc-1

an hour ago

Keep fighting the good fight. It’s more satisfying, even if entropy inevitably wins ;)

yodsanklai

6 hours ago

Which is why I like languages that make interfaces very explicit, like OCaml or Ada. Most of the time, I don't want to see the implementation, just a properly documented interface. If people can't describe in simple terms the behavior of an interface, something is wrong.

resonious

8 hours ago

I gotta say, it's pretty amazing to me that this was written by an engineering manager. None of the EMs I've worked with would be capable of discussing our codebase at this level of technical detail. Even the ones that used to be engineers.

Although to be fair, we don't have any EMs who were promoted from within. We have a bad habit of hiring managers from outside, as nobody internally really wants to stop doing engineering (myself included).

bbor

12 hours ago

Great article, from a technical perspective! I would say it’s more a “nomenclature” than a “taxonomy” because it’s neither exhaustive nor discrete (by design), but I might be mistaken there. I loved the physical examples for each especially, really thought provoking.

As always, I have a philosophical nit to pick: the “three axes” introduced at the top are just “Return” and “Investment” from good ol’ RoI, with a subcategory added for a particular type of forward-looking/conditional Return. I’m guessing this decision has worked in practice and I don’t expect video game development practices to be absolutely scientifically sound, but some extra philosophical certainty never hurts!

leni536

11 hours ago

One important aspect is when you knowingly take on tech debt in return of some short-term benefit. Then this benefit becomes an other axis to weigh against.

jamesfinlayson

5 hours ago

I've heard this called "tactical debt" instead of "technical debt".

ooterness

12 hours ago

Great article. The "contagion" factor is a useful concept that I hadn't seen before. Needs a [2018] tag.

APublicMan

12 hours ago

My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.

When you have a proper backlog of tickets, including tech debt tickets, the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity.

bbojan

12 hours ago

> the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity

I have yet to visit this misterious universe you describe.

hinkley

4 hours ago

I and one, maybe two other coworkers will fix some of the tech debt while everyone else tries to avoid making eye contact, and we fantasize about a world where voodoo dolls actually work.

xarope

4 hours ago

you need a smart PM who works closely with the CTO to craft the narrative to sales, that the next critical feature milestone is gated behind fixing said tech debt...

Swizec

11 hours ago

> I have yet to visit this misterious universe you describe.

The trick is to have 1 backlog. Tech debt and features live on the same list and it is up to the PM to prioritize. Engineering’s job is to argue cost.

Good PMs will prioritize relevant tech debt or pull it in with feature work in the same area. They understand the tradeoff of go slow to go fast. They also understand when tech debt will never become relevant (because the feature is getting nixed, or hasn’t shown desired impact yet, or because the cost of interest is waaaay lower than the cost of paying it off in many cases).

This only works when engineers have the discipline to look stinky awful code in the eye and say “not today” and stay within agreed timeboxes. You blow this estimate once or twice, get the PM in hot water with leadership, and you’ve lost the trust.

NAHWheatCracker

10 hours ago

All of the teams I've been on have used one list. I've never seen a PM prioritize the technical work. I still think it's a good idea for it to be one list, but it's not sufficient.

For teams that don't have a good PM, you also need a tech champion. Failing that, engineers need to inflate estimates and do tech work under other stories. Then everything becomes less predictable and teams never develop trust.

rqtwteye

9 hours ago

All PMs I have seen so far were just passing on management’s desire for more features quickly. The only approach I have seen work is if engineering adds refactoring as part of the normal work that needs to be done without asking for permission.

NAHWheatCracker

9 hours ago

That's the practical advice to engineers who are stuck in a dysfunctional organization where they can't really effect change, which is probably 90%+ of all organizations.

jamesfinlayson

5 hours ago

I remember working in a team where the backlog was controlled by the PM and he created a separate backlog that developers got to use - unsurprisingly, pretty much nothing ever got moved out of the separate backlog.

patrickmay

9 hours ago

> For teams that don't have a good PM, you also need a tech champion.

That's part of the role of a Technical Program Manager. The Eng Manager, Product Manager, and TPM should form a holy trinity of mutual support, filling in for each other's gaps. When that happens, you get much better odd of having a high performing team.

Source: I've been both an engineering manager and a TPM. Never the PM, though.

NAHWheatCracker

9 hours ago

Perhaps that can work, but I'm skeptical whenever the solution is "another manager".

Swizec

10 hours ago

> For teams that don't have a good PM, you also need a tech champion

Yes. And to add some nuance, you need a [trusted] engineer who can say “This will take 3 weeks because of tech debt items A, B, C. We can fix those in 1 week and then take 1 week to implement this. How would you like to proceed?”

Any decent PM will take the 2 week option that also cleans up the codebase.

But if fixing the tech debt would take 3 weeks and then another 2 weeks to build the feature, then any decent PM will take the option that doesn’t fix tech debt unless there’s a bunch more stuff coming in this area in which case taking 3 weeks to fix stuff is totally worth it.

Their job is to make those tradeoffs. Our job is to highlight the tradeoffs they’re making so they can make informed decisions.

gregmac

4 hours ago

> “This will take 3 weeks because of tech debt items A, B, C. We can fix those in 1 week and then take 1 week to implement this. How would you like to proceed?”

I've experienced something like this, but only on a project that mostly had the original team that built it (including me) still working on it. We were able to keep things in check, and in the above case would just do it that way without really asking.

On many other projects I've been involved in, there's years of tech debt that has accumulated: the typical retrospectively incorrect design decision, followed by layers and layers of band-aids, each time making the real fix more complicated and a bigger scope.

These things undoubtedly increase the cost of everything else, but it's really hard to articulate. The fixes take weeks, the break-even won't come until months later, the long-term team members are a mix of skeptical and defensive of their work (eg: don't want to do the real fix). In some cases, there's a war story "we heard that about x, but that caused so many bugs we had to revert and abandon it, why is this going to be different?"

Any tips for anyone working in this environment?

NAHWheatCracker

9 hours ago

I agree with you completely that you need trust.

> Our job is to highlight the tradeoffs they’re making so they can make informed decisions.

This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.

If you have shit engineers and great PMs, the best outcome is likely to shift decision making to PMs. If you have great engineers and shit PMs, decision making should shift towards engineers.

If they are both equivalently shit or great, it should be a balance. I believe this is the most likely scenario. I believe that balance is thrown out the window if engineers "highlight the tradeoffs" while the actual decision making is lies with the PMs.

How to actually achieve balance is extremely idiomatic to the team and organization. It's hard to get people to have adult, non-confrontational discussions about this sort of thing, however. Too many people will treat it as a negotiation.

Swizec

8 hours ago

> This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.

I think of it more as a partnership.

If I’m in charge of getting groceries and you’re in charge of budgets, we need to have an informed discussion on what exactly is our budget and what food we need so we don’t starve. Sure I could blow the whole budget on steak and I might even love eating nothing but steak for 3 days, but eventually some carbs would be nice. Likewise neither of us will be happy if I go max stingy and buy nothing but bags of rice for the week.

The reason I think PMs should make the final call is not that engineers are subordinate, it’s that PMs are accountable. (RACI – responsible, accountable, consulted, informed). The person whose ass is on the line makes the call.

Usually when I ask engineers if they want to be accountable for making the call (and its outcome), things get real quiet real fast :)

NAHWheatCracker

6 hours ago

If PMs are accountable, then I'm with you. Decision making should lie with those accountable.

From what I've seen, accountability doesn't mean much. Could be the places I've worked. Poor PMs get promoted despite running projects into the ground, good engineers get held back despite pushing through adverse project plans, vice versa.

FridgeSeal

9 hours ago

> it is up to the PM to prioritize. Engineering’s job is to argue cost.

That’s a lot of words to say “more features lol” which is basically what every PM I’ve worked with has only wanted.

hinkley

4 hours ago

That just lets the lazy devs scapegoat the PM for “not letting the “ work on the tech debt.

Most people don’t want to work on it. That’s why there is so much. Generating it is like eating candy. It’s unhealthy but you just want to have something sweet right now and the bowl is in reach…

FridgeSeal

2 hours ago

Hmmm, not sure I buy the argument.

Most co-workers I’ve had would have _loved_ to fix the shortcuts and hacks that were done to meet deadlines, but were never given the time. “Refactor while you do new features” works sometimes, but doesn’t work on anything larger scale - e.g. if your overall architecture is collapsing under its own weight, it’s hard to “sneak in” the sort of major work you need to do to fix it.

hinkley

10 minutes ago

Oh there’s always a few of those, but then there’s the corner cutters who make more of the mess than everyone else, and few impostors who fade into the bushes when there’s a gap in the schedule.

rqmedes

9 hours ago

Agile is perfectly optimised for creating tech debt. Corporate software is almost always impossible to change once released so it’s obvious that frequent iterative deliverables that you can only code around or on top of propagate technical debt

hinkley

4 hours ago

Waterfall has time to bury the debt and let the grass grow over the crime scene before people come asking questions.

This is dev culture not agile culture.

loloquwowndueo

12 hours ago

“Not enough feature tickets to exhaust capacity” - I don’t think I’ve ever seen this happen :) PMs and sales always manage to book all available capacity.

hinkley

4 hours ago

I’ve done it once or twice. One particular time there was a lot of hand wringing about how there was nothing to work on. I about saw red. Tech debt and bugs. That’s what you work on.

That incident really changed my perspective on people who talk about how tech debt is bad. Some of them will roll up their sleeves, but some just want to look high minded without putting in the effort.

deknos

12 hours ago

if tech debt would depend on some kind of methodology it would not pop up with XP/Kanban/waterfall.

techdebt can even pop up in unorganized slowmo opensource software.

arrjayh

11 hours ago

> My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.

Yeah, this is 100% correct. I comically left Riot after ~6 months for this exact reason. Obviously it's a large company with many different flavors of teams, and it sounds like this team maybe has gotten it together, but by in large most haven't.

While I was there I was working on some of their core games tooling and felt uneasy about my day-to-day. My teams tech debt was quite literally owning them. Constantly missing sprint scopes, spending countless hours arguing and debating about trivial stuff, it was all a mess. They ended up laying off a number of people from that team in a pretty shifty manner so maybe things have gotten better since then.