No More Arm Chairing
“That sounds like something an automated test should have caught.”
“If you cannot catch a basic bug like this, then what the hell is the QA department doing?”
“Classic business, to pull a stunt like that!”
Distance simplifies things. The larger the distance, the bigger this problem becomes. In some organizations I’ve seen it escalate to what I describe as the “hierarchy of disappointment.”
The hierarchy of disappointment
At the top of this hierarchy is the Business.
In the Business we identify opportunities in the market that are going to be huge (some would say yuge). And all we need to grab them is a tiny bit of product.
Sadly, Product in this company is a disappointment. Product says “no” to practically everything we ask, and the few things they do agree too, even though they're pretty basic, are delivered at a turtle's pace, and an unpredictable turtle at that. We’re not sure what Product is doing most of the time, except deliver some stripped-down-to-the-bone version of whatever we told them we needed, after the opportunity has already passed. Clearly they have somehow abstracted from the idea that money needs to be earned to keep this business running.
In Product we listen to the voice of the customer, and based on their feedback, behavior and other sources of data, figure out how to make our product even more awesome. Sometimes people from the Business descend from the stairs, and attempt to disrupt our roadmap with short term revenue boosting ideas to hit some arbitrary revenue target they set based on who-knows-what. These ideas are generally not particularly coherent from a product perspective, but hey — when it generates some cash, who cares, right? Luckily, we are good at saying “no” most of the time, and manage to pursue our own roadmap.
Sadly, our roadmap is progressing slowly, because Engineering cannot keep up. We're not very convinced our Engineering guys know what they're doing, because they keep complaining about all the technical debt that's there, and that we need to stop everything to pay it. “Who generated all that technical debt?” we are wondering in Product. Apparently, the Engineering guys cannot write code without having to fix it later, and as a result we cannot build the features that our users are clamoring for and tested so well in our UX lab. On top of that, Engineering generates bugs all over, generates technical debt, and then proceed to complain about how we're not prioritizing them cleaning up their mess. Engineering in this company is a big disappointment.
In Engineering, we do the real work. We're writing the code that keeps this company afloat. No code, no product; no product, no business. Simple as that. But we do not get the space to do our job properly. There's always product people chasing our ass, asking us to move faster, to cut corners, and asking what customer value unit tests really provide. We tried to explain technical debt, but we're not really sure they understand the cost of all the gymnastics and duck tape we apply to hit their arbitrary deadlines.
Part of the challenge in Engineering is that our users cannot use the cool features we build some of the time, because our Infrastructure guys cannot seem to keep a simple server running. We deliver them a nice package — and all they have to do is use off the shelf tools — tools that every other company in the universe is also using — and run our software. Yet somehow, this proves too difficult. One day the database goes down, another it turns out they seem to be too slow to provision new servers to handle all the users that want to use awesome new feature. Infrastructure in this company is a big disappointment.
We in Infrastructure are the real warriors in this company. While engineering checks out at 5pm, and is having a fun family time, we're on call trying to somehow keep this monstrous piece of “art” that they produced running. It's as inefficient as f***, executes like 200 database queries per request, and then people are surprised we incur so much infrastructure cost because we need the largest machines in the universe to even keep this application running for a few hours before it collapses under its enormous memory leaks. Then, they asks us to reduce infrastructure cost, even though 80% of that cost comes debug logs saturating the network, and all the petabytes of storage it requires to keep them even for a week. Are these people completely blind? This company is a dead end. Exit now.
At all levels we sit back on our comfortable arm chairs, and based on limited understanding pronounce our judgment. “Why is this hard? Just do this!”
On the abstract, everything is easy; when you get down to the detail, hardly anything is.
Yet, this is what we do. It seems human nature, so what do we do?
Two ideas.
Fight human nature
First, fight human nature. I’ve been on both sides of this issue. I’ve been the one on the outside being puzzled about how other departments and teams seem to struggle with seemingly basic things. I’ve also been up close, receiving judgmental feedback of the kind exemplified above.
As a result, I defined a basic assumption for myself whenever I feel tempted to do some arm chair business, infrastructure or product management.
Here goes:
People are smart. If I think they’re missing basic things, it’s probably me missing information or complexity.
Therefore, I set myself a basic rule whenever I’m tempted to make such comments or even think this way.
Here goes:
Whenever I’m tempted to “arm chair” something; don’t.
Why? Two reasons:
The first reason is that there is no possible scenario that such comments will be perceived as constructive or helpful. Violent communication! However, if I really believe I have a unique perspective to offer, I usually offer help along the lines of “hey, I have some ideas on how to avoid such situations in the future, if you’re interested we can discuss them face to face.” Then it’s up to the other party to decide if they see value in that or not.
The second reason is to attempt to keep the practice of judging people to a minimum. Judging people is super easy, but also very toxic. The moment you judge them, you label them in your head. “Incapable.” Then whenever they show up on your radar again, you will find that “incapable” label, perhaps not remembering what it was based on. And because of the distance trap, it’s likely to be invalid. Don’t do this.
Big tables
Second idea: invest in tables. Big tables.
As mentions, we’re dealing with a distance trap. And how do we reduce distance?
Get closer.
When you think about it, this is what DevOps is about to a large extent. The old model was to have separate engineering and operations teams. This naturally lead to distance, leading to lack of empathy, lack of mutual understanding, lack of productive collaboration and a lot of arm chairing. In DevOps, you bring both together in one team, sitting at the same table. Engineering becomes painfully aware of how their code operates in production, and the whole team owns the development and operational aspects of the software _together_.
Agile teams in turn, pull Product and Engineering together. Through mutual challenging of each other's ideas, and sitting at the same table, Engineering becomes more aware of the rationale behind the features to be built and what may possibly happen in the future. Product becomes more aware of bottlenecks, and gets visibility in trade-offs and their short and long term impact.
AWS takes the next natural step with BusProdDevOps, putting Business, Product, Engineering and Infrastructure together in one team, sitting at the same table. Business sees an opportunity, all functions analyze together what the options and the trade-offs between the options are at all levels, and well-considered decisions are made.
It's so obvious, yet so rare.
Big tables.
Big tables, and the right people sitting at them. People that want to understand the other perspectives. Business who wants to understand the complexity of Product, Engineering and Infrastructure. Product who wants to understand the urgency with which the Business operates and the financial state of the company. And wants to understand all other functions, and gives feedback on what things are technically easy, what things are hard, and what the consequences are of decisions long and short term. And Infrastructure who can estimate potential scaling issues that may come out of all of this.
They all sit at that table. Perhaps more than seems necessary, until the gap closes.