Fighting technical debt and feature creep, with data

In many big tech companies, very few engineers get to make meaningful decisions about the design and architecture of what they work on. This is even more true when it comes to deciding which features to implement, and is also often the case even in determining exactly how something is going to be implemented.

This is depressing, because it reduces many talented engineers from innovators to zombified feature implementors. What is even more insidious is when a new feature comes along for which the business demand trumps everything else, including proper research, technical design, and adequate testing.

This is one of the primary causes of technical debt. “We’ll make time to do it right later” ends up being a hollow promise in the face of yet more business requirements, with higher priorities. Obviously this is not sustainable.

I don’t have an answer for how to completely avoid tech debt. What I do have is one big suggestion for how you deal with the next big urgent product feature request — the one that is critical and must take priority over everything else (which, even in the best agile environments, I still see a lot of).

Know thine data

However you achieve it, you should know — or be able to find out — anything about your app, in under an hour.

If there’s something you can’t find out, then I would put it to you that you don’t have enough logging and instrumentation. If you can’t answer all of these questions, then this is probably the case:

  • What was the top browser / operating system that hit your app on the first of this month?
  • How many users visit your app from France?
  • How long does it take your first page to load, on average, for all users in North America, including loading all front-end assets?
  • What is the most common exception thrown on the back-end?
  • What is the most common exception thrown on the front-end?
  • How is your app converting overall?
  • If you were to add a new AB test, how would each feature be converting?

If you can answer all of these questions, then you’re already streets ahead, and you probably can’t learn anything new from me.

That said, it’s actually pretty rare that I encounter engineers who are fully capable of finding all of this out for themselves. Sometimes they rely on dedicated analytics teams. Other times they have a google analytics instance running. That’s a good start, but it’s really not enough.

Why is this important?

If a product owner asks you to do something — say, add a feature or change an experience, you should be able to find out very quickly how many of your users the change would impact. You should then be able to draw a comparison between what you feel it’s important to work on, and what they do. For example:

Product owner: We need to change this button’s text and roll out to production, as soon as possible.

You: 5000 people see that button per day, but right now I’m working on fixing an error which is affecting 10,000 per day. Still need me to do it right away?

Obviously, this exercise won’t always give you the result you’re hoping for — the data may simply tell a different story to what you’re expecting. But the important thing is, now you have a seat at the table. Beforehand, simply asserting “I think this is a better thing to do now” probably wouldn’t cut it, but now your opinion has some credibility behind it.

Tangentially, knowing how people actually use your app and what their experience is ends up making you a better overall engineer. You’ll begin to learn instinctively how important site speed is for your users, or what effect different stylesheets on your pages have to your customers, or how much your app’s reliability can affect conversion. Figuring out these things can have a pretty dramatic impact on your approach to building software in the first place.

Sometimes there’s no preexisting data which can help answer your questions

In which case, if you feel strongly enough about it, I encourage you to break out the humble AB test. If a feature that would take six weeks to build can be mocked and experimented with using a small proportion of your traffic in two days, the risk of wasting time building it becomes significantly reduced.

The most common question you may find yourself asking your product owner is, “If this feature takes X days and increases our revenue by Y dollars, do you still want me to go ahead with it?”. You should be fully capable of posing this kind of question.

Of course, it’s important to keep ego out of this process as much as possible. If the data doesn’t support the theory you hoped it would, and you don’t have convincing evidence as to why it’s incorrect, irrelevant, or superseded by some other concern, then you should probably accept the result and move on.

So how to I get to this state of zen, with all possible knowledge at my fingertips?

You know best how your product and instrumentation works — but, generally speaking:

  1. Come up with a complete list of all possible questions you might ever ask, or be asked, about your app, within reason.
  2. For each question you can’t find out the answer to now, add logs that enable you to. Keep your logging as general and reusable as possible. Don’t stop at the minimum instrumentation requirements that your company sets down. Go much, much further, until your app is reporting everything you think it needs to.
  3. If you work on a front-facing app, you absolutely need to log from the front-end too.
  4. Store your data somewhere you can quickly and easily access it and query it.
  5. Learn how to use Hadoop, or your data platform of choice. Graphs and charts are nice, but these will never really be able to fully answer every question you have, beyond the most basic. If you can write a Hive query, you can find out how long it took the average user to run through your web flow, for all Canadian users using Firefox between 2pm-5pm. Learn to join and correlate your logs to find out information like this.
  6. Build a common group of queries that you and your team-mates can use and adjust. Keep it in a git repo and share it.
  7. Ask your team-mates difficult questions about the code they’re pushing. If they can’t answer them, encourage them to add enough instrumentation that they’ll be able to.

Being a well rounded engineer in this day and age absolutely means you need to know how to read and use the data reported by your app. You may find this gives you the step up you need, from simply taking features and implementing them, to having a full say in what ought to be built or changed.

Data is power.

Addendum: if you’re interested in hearing some really fantastic thoughts about the role data plays in how we build software, in a grander sense than is covered this article, I can not recommend this talk by Greg Wilson enough:

works for PayPal, as a lead engineer in Checkout. Opinions expressed herein belong to him and not his employer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store