The Innovator’s Challenge

Last week I published an article on picking battles. When I read comments on twitter, reddit and hacker news, the article was often summarised as “don’t jump on all the new stuff that people invent, stick with what you know and has been in use for years.” If that were true — wouldn’t that lead to a technological stand-still?

The intention of the article was slightly more subtle — I tried to point out the cost of adopting new tech. Adopting new, unproven technology might be much more costly than anticipated because of all the hidden problems that nobody ever found, because it hasn’t been stress tested before. Therefore, if the new language, runtime or database only slightly nicer, cooler, prettier, more concise or faster, it may not be worth the extra risk.

However, the pendulum swings to the other side too: if new technology makes a big difference in expressiveness, beauty or performance — and you’re sure those things matter to you — then choosing the new technology may be what makes your product a winner.

This does pose a challenge to the innovators among us — the people who work hard to push the status quo in software development:

Evolutionary change no longer cuts it, revolutionary change is required to be cost effective to its consumers.

As a reminder, here’s the two key reasons why using old technology is attractive:

  1. All edge cases have been hit and ironed out, bottlenecks are known — the tech is rock solid.
  2. There is a body of knowledge available:
    • You can read books about it
    • You can Google any problem and find a solution
    • You can find and hire experts

If you are an innovator building new tools, this is what you’re up against.

When you’re starting out, you won’t have any of these things. Your tool is going to be young and buggy, and documentation will likely be lacking and since nobody used or wrote about it, if you run into a problem, there’s probably only one person to help fix it: you.

This is a classic chicken and egg problem: your tool will only ever mature in this way when people actually use it, and people will only ever actually use it once it becomes mature.

Sigh.

Let’s say you developed a new programming language that is slightly better. It catches a few more errors at compile time; it allows you to solve problem X in 2 lines, instead of 10. Or you are developing a database system does most things Redis does, but has faster set operations, or allows you to write in-database scripts with JavaScript and not just Lua. I’m happy to play with a language like this, or a database system with these new features. However, there’s no documentation for your language, there’s likely bugs in the runtime and who knows what happens if I try to compile a 10,000 line codebase. Therefore, once I intend to build a production-quality system, I have to ask myself: does the slight improvement in expressiveness, or slight improvement in performance trade off well against all the risk I’m taking?

It does when your tool would be revolutionary: if your language is not only slightly better, but an order of magnitude better; if your database is not slightly faster, but orders of magnitude faster.

If you managed to build such a tool: Congratulations! Now it’s time for the second challenge you have as an innovator: sell your idea to the rest of the world. You have to market the tool in such a way that people can no longer ignore its benefits.

My favorite example of where this worked is Ruby on Rails. One can argue whether Rails is revolutionary, but personally I considered the combination of DRY (don’t repeat yourself), convention-over-configuration and internal DSLs in a web framework a leap forward for the state of web development at the time. Undoubtedly some of you will email me telling me that none of these ideas were novel, but with that you’ll only prove my point:

If you aren’t able to sell your idea to others, it may as well not exist at all — those who can sell the idea reap the benefits.

Ruby on Rails was originally invented to simplify building a specific product: Basecamp. As Basecamp became popular, people became interested in the tech is was built on. There was now a clear example of a popular application that was built using Ruby on Rails, and it had become clear it was up to the task of building a real application. Not only that, but the Ruby on Rails website looks pretty. It contained numerous tutorials and screencasts, one of which demonstrating how a simple blog application could be constructed in 10 minutes. In addition, there was David Heinemeier Hansson as outspoken spokesman. You may like his approach or not, it worked.

Today, Ruby on Rails is on my list of boring technology.

About a week or two ago I attended a pitch from a company working on what I have since come to believe could be the next big revolution in software development. When I heard the pitch, I got all excited. Not only about the idea itself, but also because there’s still a lot of work to be done in selling the idea to the larger public.

It looks like I will be joining this company next month. More on this topic later.