Markets are supposed to make corporations efficient, so if you do consulting, you have to wonder why so many organisations (both private and government) are so absurdly dysfunctional. The way I see it, there’s no real paradox: organisations are pushed by both forces of efficiency (like market forces) and forces of dysfunction (like political drama). Corporations are only efficient if the forces of efficiency are stronger.

There are many forces of dysfunction that can affect an organisation, but there’s one that’s particularly important for risk-averse organisations (like banks and large government departments). Whenever I see people in an organisation doing something that doesn’t make any sense, I always ask if this catch-22 can explain it:

Every risk-averse organisation needs someone to take the initiative to eliminate risks. But in a risk-averse organisation, that’s exactly what no one does.

Imagine you’re a software developer at a bank, and there’s a software project that’s in severe danger of failing, and you think you might be able to help save it. Do you A) volunteer to get involved, or B) keep yourself busy doing something else? Whatever you think the right answer is, many people would choose option B. Even if you chose A and did a great job, you risk becoming a scapegoat anyway if the project still fails.

In other words, there are two kinds of risk: risk to the organisation and risk to the individual in the organisation. Those who keep out of risky situations are simply optimising their level of exposure to individual risk, even if it increases the risk to the organisation.

Sure, there are heroes in the world who take the initiative against risk, but heroes like that aren’t permanent employees of risk-averse companies. Even if they call themselves permanent employees, it’s a matter of time before they take the blame for something and get fired. Minimising individual risk at the expense of organisational risk isn’t just an idea, but an evolutionary force that shapes organisations.

For example, why do many risk-averse corporations pay approximately double typical market rates for typical-quality workers from big “consulting” companies (that pay their workers the market rate and pocket the difference)? Surely, with a good interviewing process, they could get excellent workers for two-thirds the price. However, recognising good talent requires personal judgement, and personal judgement comes with the personal risk of being fired if the wrong choice is made. No one gets fired for hiring from Big Consultancy Inc. Joel Spolsky likened these firms to McDonalds: you know you won’t get the best burger in town, but you know exactly what you will get. There’s a similar story behind big companies using known-unreliable standards like “5 years experience with React”.

This kind of effect feeds back into the way organisation-level risk is handled. If large, risk averse organisations aren’t good at using individual employees’ personal judgement, then the lower-risk compromise is to outsource as many decisions like that to third parties with a different risk structure.

A Dysfunctional Corporation is an Efficient Corporation

Market forces make a corporation more efficient at making money, and risk exposure makes a corporation more efficient at avoiding risk. The problem is that corporations are better at avoiding individual-level risk than organisation-level risk.

I think of it like a mountain. Every now and then heavy rain falls and flows down the mountainside to the sea. Any rock that sits in the way gets washed down, so over time streams and rivers form that are optimised for carrying water away. Likewise, an organisation that’s exposed to risk will occasionally suffer a disaster. Someone in a leadership position will promise to “find who’s responsible” and that “heads will roll”, effectively disowning any responsibility for the disaster and pushing it down the organisational ladder. In this way, blame flows down the organisation, just like stormwater flows down the mountain. Sometimes the blame will land on someone, and that person will get fired, or get punished, or will simply quit in disgust. But however it happens, the barriers to the downward flow of blame disappear over time. Ideally, the organisational structure makes any blame disperse completely.

The bystander effect is a powerful way to avoid accountability. Actions require sign-offs from multiple people, decisions involve large meetings, projects have several middle managers with highly specific roles. There’s no single person who’s responsible by default. The buck stops nowhere.

To see how this works, suppose, for example, a big corporation loses a multi-million dollar contract with a third party because a large amount of cheese was accidentally delivered to a meeting room and it went mouldy. Nobody did anything about it because nobody has “cheese” in their job description, which also means nobody gets punished. Of course, that’s unacceptable to the corporation, so a new cheese manager role is created. The new cheese manager is mostly safe because accidents involving cheese are rare, but just to be sure, the cheese manager hires a cheese preservation consultant, allowing the blame for any cheese-related crises to be passed on to an external party. This creates a new problem: how can the cheese manager justify their job if a cheese consultant has been hired to take all responsibility for it? There are many solutions to this problem, but an effective one is micromanagement.

Once upon a time, I was working on a website that was essentially a version 2 of an existing website. The original website already had a functional login system, but the managers decided that it was important to design a new login system from scratch, to make absolutely sure we had one that met all our needs. So, one manager came up with a list of requirements (“As a user, I need textboxes for the username and password”, “As a user, I need an error page”), while a second manager wrote them onto a whiteboard, while a third manager read them out again to a UI designer. The designer drew sketches of a login system and passed them to the frontend developers. The new design was almost, but not quite, the same as the existing system, so the frontend developers came to me (the backend developer) with a list of change requests. I went back to the designer and showed him the existing login system. Apparently he’d never seen it before, but he said it met all the requirements the managers had come up with, and was okay as is.

For people like the cheese manager, this kind of micromanagement is an all-round win. They can be important by “helping” design the login system, but carry no risk of being blamed if anything goes wrong with it.

But Who’s Making the Money?

People only quit their jobs to start a new business if they care more about making something than about job security, so every new successful business starts with a small number of highly motivated individuals. In some highly competitive industries, like game development and high-frequency trading, many of these businesses never grow much and stay small and efficient. Other industries are natural monopolies, or have strong economies of scale or network effects, so that once the business is established in the market, it’s stable. In fact, stability becomes more of a priority, and creating new things becomes less of one.

New employees who join as a company grows might still be highly motivated, but they’re more likely to care about job security than the founders. This effect keeps getting stronger as the company gets bigger and more stable. There’s a version of the Dead Sea Effect at work, too. The flip side of “you never succeed if you never try” is that you can only fail (and get fired for failing) if you do try. Of course, this doesn’t mean that everyone who gets fired (or just quits) is a “rockstar” and everyone who stays is a “loser”; it’s just a common bias that people who prefer job security and stability are more likely to accumulate as a corporation gets older and bigger.

I once did a contract at a very large multinational where most of the productive new work was being done in small branch offices, while the main office was so dysfunctional it could be a Monty Python sketch. There were teams there that had been working on projects for up to ten years and still only had alpha-quality prototypes to show for it. The office survived through politics and its ownership of key cash cow projects.

At the start of the first week I was there, the middle managers demanded that all the engineers down tools and write detailed reports describing what they were working on. That plus various other meetings and stuff meant practically nothing useful got done. At the start of the next week, the middle managers said that it was great that they had all these detailed reports, but they needed everyone to write summaries of them, so very little progress was made on the projects for another week. Now, whenever I tell this story I’m scared that no one will believe me, but I swear it’s true: the following week the middle managers declared that the summaries were great, but lacked detail, and they really needed detailed reports. I immediately handed over the report I’d written the fortnight before, and their awkward response made it clear that wasn’t how the game was supposed to be played. So, they did the classic pointy-haired boss thing and simply pretended that meeting never happened and tried asking again a few hours later, hoping to get the “right” answer. I handed them the same report again. They tried a few more times before giving up and letting me get on with the job I was contracted to do.

Is there any Hope?

Please don’t think I wrote this blog post because I’m some misanthropic cynic. I wrote it because many people work for large organisations without being aware of this stuff, which makes them easy victims of those who game the system.

If you’re just an employee, I honestly don’t think there’s much you can do to fix a dysfunctional organisation. You can use knowledge and awareness to help defend yourself, but if you’re constantly frustrated by your work environment, maybe you should consider quitting. Just like it’s hard to imagine dysfunctional organisations exist if you’ve never seen one, if you spend every day inside one it’s hard to imagine working anywhere else. It’s easy to absorb the company “culture” and believe that working for a large company is the most important thing for your career, even though it’s often easier to make a meaningful impact in a smaller company. Look at the people around you on the way to work and remember that none of them care one bit about the politics inside Big Corp. I’m consciously aware of these things and try to remind myself of them, but I’m still always amazed at the feeling I get when I finish a contract at a bad place.

If you’re in a leadership position, there’s more you can do, but it’s a challenge. By far the most effective thing I’ve seen is the error budget used by Google’s Site Reliability Engineering. Counterintuitively, budgeting for a certain amount of failure can reduce the overall amount of failure because such a budget can create incentives for employees to reduce organisation-level risk. But it only works if it’s supported by the leadership of the organisation.