Amazon is a Jealous Boyfriend

CW: If your'e not a software engineer, this may bore you to tears.

I’m quitting Amazon after one year as a level 2 Software Engineer. Partly I’m quitting because living through a pandemic lockdown in a city where you don’t know a lot of people is duller than dirt. Mostly though I’m quitting because Amazon is a jealous boyfriend.

Amazon does not trust me. We hit it off well at first – I cleared that high bar to get the job. I watched a bunch of rah rah Bezos videos and something about how Amazon was a pirate ship with chaos but also plunder and rum and mermaids. Actually maybe I don’t remember that video too well.

One video I do remember was Bezos’s story about mechanisms vs good intentions.

A customer service rep was getting calls about a faulty product. The rep sympathized with the customer but could do nothing about it. So Amazon built a button for reps to immediately remove products from the website when customers complained. The theory is that we can assume everyone has good intentions but good intentions don’t cut it without mechanisms.

Sounds sensible. I bought it at first.

In practice this means that Amazon tries to deal with problems by adding automation. For every major software bug, Amazon will ask - what mechanism could we build to prevent this from happening again? And then we add another layer of automation to the system.

To me the idea that us Amazonians have good intentions but we need to be constantly adding mechanisms to protect us from ourselves is demeaning and backwards. The greatest asset Amazon has is human talent. The focus on building automation to restrict teams rather than empowering teams to make their own decisions makes working at Amazon inefficient and miserable.

Oh, and I’m probably going to abandon that jealous boyfriend metaphor. It’s just clickbait. Amazon has a trust problem. That’s about as far as the comparison goes.

Over-Automation

The Amazon philosophy is why use one code quality tool when 10 will do the same job 10 times slower and at 10 times the cost?

When I joined my team in advertising at Amazon our codebases were gross, but it wasn’t for lack of automation. We had one code coverage tool running in our builds and another running in our deployment pipelines. We had integration test and load test workflows that were broken and failing. One day when I was disgruntled I counted 19 separate “mechanisms” we used to ensure code quality. And the end result was garbage.

Some of these tools I don’t advocate in any context.

You can’t rely on tools to tell you if the unit tests in your code are valuable or not. That’s like outsourcing a newspaper editor role to Alexa. Sure Alexa can check for spelling errors. But you need a human to tell you that evacuating people doesn’t mean what you think it does.

Similarly you can add tools to your build to tell you about the existence of unit tests, but code coverage tools won’t tell you if your unit tests provide any value. For that you need a human pair of eyes.

I do like some of the tools – FindBugs and Checkstyle are aight. But all these tools bundled together make for bloated systems where we spend a fair amount of time maintaining the tools themselves. A coworker and I spent a day trying to figure out why code coverage was complaining about us lacking unit tests for our unit tests. More commonly you end up going down rabbit holes to fix integration tests running against brittle test environments.

In general this system bloat makes minor changes expensive and prevents us from failing fast. Minor improvements that should be easy become difficult and disincentivized. Sloppy code stays sloppy. The more we build on top of it, the more expensive it is to rewrite.

The most egregious result of all this system bloat is when we modify our design decisions in order to cut through the red tape.

Last winter we were building a new service with different scaling considerations than our other services. It made sense to deploy it as its own service so we could scale it separately. But a level 3 engineer on my team recommended we bundle it with an existing service to avoid the reviews and all the dev ops cost that comes with deploying new services at Amazon. Yuck.

Planning

Speaking of reviews, Operational Readiness Reviews were one of several inputs to our work queue that were external to the team. ORR action items are dev ops tasks assigned by a principal engineer after reviewing your system. We also were subject to Policy Engine action items, which were dev ops tasks assigned by an automated system based on organization-wide policies. Also SAS which was a software deprecation system that would assign us tasks to upgrade libraries with known vulnerabilities.

From the outside looking in, maybe this all sounds reasonable. It really isn’t.

Any software system is going to accrue tech debt. Dev teams should be responsible for prioritizing tech debt and ensuring that they’re working on some tech debt each sprint. The dev teams know what tasks will give us the most bang for our buck.

Instead at Amazon I sat through weekly meetings with 10 other engineers where we debated how we could implement automated rollback in a pipeline that was just deploying scripts to a server. Instead of adding authentication to an internal service that was wide open to anybody on the Amazon network, we prioritized upgrading our Jackson library because it contained a vulnerability.

Let me repeat that: We were maintaining a system that anybody with access to the network could call to delete our data. Rather than add authentication, we upgraded a software package that could have allowed a savvy hacker with access to the network to gain access to our already completely open system.

Allowing leadership to drive prioritization of tech debt is incredibly wasteful. Each system has its own issues and deserves its own consideration.

Imagine if the NBA commissioner made all teams practice the same set of drills. 3-point shooting has become more important in basketball, so the commissioner says all players must get 100 3-pointers up in each practice. In broad strokes it makes sense for basketball players to work on their 3-point shooting. But Dwight Howard doesn’t need to shoot 3s. Let Dwight Howard focus on his snake collection.

Blame

I’m not typically one to accept my working conditions. If a process is broken, I’ll try to fix it. At Amazon I did not have much success breaking anti-patterns.

Amazon is widely criticized for having a dog-eat-dog internal culture. I don’t think I agree with that exactly – to me it’s more of a blame game. I don’t see Amazonians trying to sabotage each other, but it does seem to me that the prime motivation is often to avoid blame.

Several times I found myself in a meeting saying that maybe the topic we were discussing would be appropriate for a retrospective, but could we please focus on solving the problem at hand? And the answer to that was basically NO. Messaging was more important. We needed to be able to tell management that missing this deadline was somebody else’s fault. Or maybe nobody’s fault. Definitely not our fault.

My manager literally asked me in a meeting once if I would be willing to take the blame for an outage if it happened over the weekend because we did not pause a feature launch on a Friday. My answer was a disappointed “sure.” I am not afraid of failure.

I have a theory that there’s not really anything I can do that somebody else can’t do too. So mostly I just try to do everything better and faster. In software development that means lean development practices and a willingness to fail fast. I assume that ultimately I’ll be evaluated on the quality of the software I produce and how fast I can build it.

Of course at Amazon it’s more about automated metrics, or so I’ve been told. Other engineers on my team are hesitant to submit multiple revisions of a code request, because for some reason high numbers of revisions are used against you in compensation reviews. So my colleagues merge code directly and skip the dry run build and code review. Because again, Amazon trusts numbers more than the opinions of their employees.

But I didn’t stick around long enough for that to much matter for me anyway. Working at Amazon was driving me crazy. So I’m waving my rights to all that lucrative AMZN stock and heading for greener pastures.

What I will miss most about Amazon is working with some really bright colleagues. My hope for them is that they all leave to other companies where they can take the training wheels off and really thrive. Or that they organize a revolt and stop disagreeing and committing to leadership mandates. Or that they breed snakes and leave Seattle on a pirate ship in search of a mermaid colony. Yeah maybe the last one.