Making Sense of “Time to Value”

Last year I wrote a few words about the “Time to Value” metric mentioned by Adrian Cockcroft in his “Innovation at Speed” talk at Re:Invent last year. This talk is about how to get better at software-based innovation (obviously, this involves using the cloud heavily — that’s why AWS is talking about it). Since in OLX we’re also looking at how to get a bit better at measuring our product development performance, I decided to reflect a little bit on this time to value metric to better understand what it means, what type of behavior the metric incentivizes and how to evaluate if that’s something your company is ready for (or even interested in). As usual here, these thoughts are quite raw.

So, the first question to ask is: is “innovation” relevant to you? Do you need to, even want to, innovate at your company? Likely, the answer is going to be “yes.” Of course, not nearly all the work to be done would immediately be classified as innovative, but on the other hand, there aren’t that many well-treaded paths either.

As I mentioned in a previous post, we just came out of a rebranding exercise. Is rebranding innovation? It’s not like nobody has ever rebranded before. OLX has even done this. In many countries our product used to be called something completely different, and was then rebranded to OLX. Been there, done that. So, this rebranding should be a simple slam dunk, right? Not so much, because with every new look come opportunities to confuse and upset people, move some buttons around, remove some options, reshuffle some others. You may not think of this as revolutionary innovation, but it does involve diving into the unknown and therefore most practices we apply when “innovating” apply: experimenting, testing, researching, iterating.

So yes, chances are that in your company too, innovation is something you want to, and need to do.

“So,” Adrian Cockcroft says, “Time to Value is your one metric to optimize!” And Cockcroft is a VP-level important dude at Amazon, and previously at Netflix, so let’s not dismiss his suggestion out of the gate.

What is time to value then? Time to value is defined as the time between the moment you “do some work” and for that work to provide value to customers (live on production, ready for people to use). If you work in a company that has a yearly software release cycle, obviously this time to value is very high (up to a year). If we want to be able to compete in today’s world — Cockcroft’s argument goes — we have to move that number down from months to weeks, or ideally days or less.

How to do that?

The first step is to figure out a way to measure this time to value. Now this is where things get tricky already. What’s suggested during the talk is that the clock starts ticking once “somebody does some work,” but what does that mean?

Let’s say I’m an engineer in my company and during my morning shower I think: “You know what, if we reshuffle some of these buttons and make one of them much more prominent, I think more customers will actually discover and use this new feature we built!” So, I drive to the office (this hypothetical scenario takes place pre- or post-COVID times), open my laptop, adjust some HTML and CSS, commit my change and perhaps configure it to behind some A/B test, see it flow through our build, test and deploy timelines and — voila, it’s live to some percentage of customers, and I eagerly track the effects on conversion of my reshuffled buttons. Stop the clock. That’s your time to value.

We could at this stage get into an argument trying to quantify the amount of value brought compared to the amount of time invested, but that’s a topic for another day (maybe).

But, what if you’re not a “highly empowered engineer” that, without discussion with the rest of the team, just YOLO deploys his or her shower idea? Is that, in fact, even something we want to encourage? To me personally this is the ideal scenario, although not necessarily how everybody works today. I like it for a few reasons:

  1. Since this all happens in a time span of maybe a few hours, is run as an experiment and easily rolled back — cost and risk are low, but positive impact could be high.
  2. I used “highly empowered engineer” somewhat cynically here, but for me engineers that actively participate in deciding what to build is something I love seeing. This may not immediately need to translate to everybody in the team just doing whatever they think is best in some uncoordinated way, but some level of that — I’d encourage it.

This mode of working or measuring may not always be a good fit, though. It’s a fit once you have your product at a level where the core is solid and you can iterate. But, that’s not always the case.

Many company spend a significant amount of time on “big bet” style projects. It takes quite a lot of baseline work to get to something that provides value. Perhaps even six months or more. And six months is not the order of magnitude “time to value” we’re looking for.

Nevertheless, once over that first release bump, follow-up iterations can happen more quickly, and your time to value should drop significantly. One recent OLX example that would make Adrian Cockcroft proud would be the project we did adding video conferencing functionality to our light-weight applicant-tracking system that’s now integrated with OLX’s job category. The goal was to allow job interviews to take place online, which became quite relevant when COVID hit us hard. This project went from “hey, I have an idea” to customer value in about two weeks, because it was heavily built on off-the-shelf AWS components (which is why Adrian would be very proud). Two weeks for a feature like that is not bad at all. I have to admit here I wasn’t in support of developing this feature given the limited time available, partially because I didn’t think it could be done this quickly — I was proven wrong.

When innovation reaches that level of speed, you can experiment more freely. If such a feature would take 3-6 months to build, likely we’d first spend weeks or months doing design sprints, market research, build prototypes, test it in our UX labs with potential customers. But once you can go from “doing some work” to value in 2 weeks, why not just build it and see what happens?

Anyway, I think Time to Value is a great metric to look at at an organizational level, but at the team level, it’s tougher for teams that are in the stage of building huge things from the ground up, unless we find a way to launch such initiatives much more incrementally.

And the latter is in fact one of Cockcroft’s core points:

“There is no economy of scale in software — smaller changes are better.”

The fewer projects we have to do in such big-bang iterations, the better, and that fact is accurately reflected in time to value.

This Adrian Cockcroft guy, not so crazy!