Job 12: Amazon Prime Video
As my time at 11 Honoré was winding down, the company was relatively generous in allowing me ample time to interview and find a new role while doing my best to do as much work for them as I could. I expected that finding a job, even in LA, would be similar to what it was the year or two before when I had multiple interesting opportunities, but that didn't prove to be the case. I spoke with some very interesting companies, but wasn't finding anything that was working out with an offer. I had one very memorable experience where, while I was waiting to be escorted out of the building after an interview, I recognized the very distinctive voice of a close colleague from Hulu starting his interview for the same position. (He got the offer; I did not.)
Living in Santa Monica was expensive, though, and I didn't have a year's worth of salary in a savings account to cover expenses, so when an Amazon recruiter contacted me for a position at Prime Video, I knew better than to say no to a possibility, even though I thought that two separate times at Amazon was likely enough. I spent quite a bit of time talking with the recruiter and hiring managers on the Prime Video team to understand what sort of team I would be joining, and eventually decided to go through my third Amazon interview loop, satisfied that the local leadership for Prime Video was working to build a positive and supportive team culture. Despite a nearly fatal lapse in judgment during the Bar Raiser interview (Q: "Tell me about a time you were leading a project, and the project was going off the rails." A: "Projects I lead don't go off the rails."), I got an offer and, with no other possibilities in the LA area, agreed to accept it.
I joined Prime Video in October of 2019. After going through the standard new hire orientation rigamarole, I talked with the hiring manager and learned that they hadn't fully decided which team I would be leading. They were in the process of figuring out a major reorganization and would have a new team for me to lead within a month or two. In the meantime, they asked me to take charge of weekly operational reviews for the organization, and use whatever free time that allowed to familiarize myself more with the workings of Prime Video.
Prime Video's engineering organization was largely organized into video playback, application, and content catalog teams. Most of the teams in the Santa Monica office, including mine, were part of the content catalog, and we worked closely with partner teams in the Seattle office. Much of our work was related to getting raw video and content metadata (episode descriptions, subtitles, audio tracks, etc. all in multiple languages and formats) from the content studios, ingesting it into Amazon's databases, and preparing it for use in the app. After the reorganization I described above was announced, the team I ended up being put in charge of was a new team responsible for implementing a final quality control gate that would make sure all of the content metadata for a given piece of content was ready before the content would be made available to viewers.
Without such a quality control gate in place, it was apparently common that content would be available for viewing without having all of the different parts together. This was a particular problem internationally, where we would create poor viewer experiences by releasing only the English-language version of content into non-English-speaking countries before the appropriate translations were available. Enabling this quality control gate was part of a much larger rewrite of the entire content catalog system that both the Santa Monica and Seattle teams were neck deep in when I started.
The first deliverable expected from my team was due, IIRC, in about January of 2020. It was already late November, if not December, when I officially started leading the team, so figuring out whether my team was on track to meet this deadline was priority #1. I looked at the work required, reviewed all of the estimates, and confirmed that the best estimates showed my team needing well into February, if not March to do what they needed to do. As if nobody in the organization had ever heard of Fred Brooks, it was common practice within Prime Video for teams to borrow engineers from other teams to get back on track, so I was told to get one week's worth of time from this engineer, and two weeks from this new hire on another team, and so on to give my team enough developer-hours to make the math work out.
Despite the December holidays taking up a big chunk of time between when I took over the team and this first major deadline, our marching orders were to press on and get things done. Despite all of the engineer shufflling we did, I think we ended up completing our feature work more or less in accordance with our original estimates (an outcome which should surprise exactly noone), but somebody made a decision to "Launch" the overall project on time anyway, and the specific functionality my team was supporting ended up being considered a "Fast Follow".
Then, March of 2020 happened.
As it became clear to everybody that the pandemic closures would last more than a week and we started to get used to remote work, Prime Video made some organization-wide pivots to focus on pushing out some key new features in response to the pandemic. We built out support for theatrical release movies on Prime Video (since movie theaters were shut down) and also supported a number of live streaming events related to the pandemic. (I believe I was oncall for a live stream event from the Prime Minister of India.) For these features, we put pause on the larger rearchitecture project, and it generally felt like we were doing some small amount of good in the world with what we were pushing out.
And then it was back to business as usual. Almost as if nothing had changed, it was time to look at the next phase of our rearchitecture, figure out timelines, and make them hit a predetermined date, no matter what our estimates said. The most senior engineers in the larger organization spent several weeks going over requirements and system designs, and put together a list of work that each team would need to do. From the beginning, rough estimates of the work showed that my team would take longer to complete our part of the project than any other team in Prime Video, and simple projections of a completion date had us completing the project well into 2021. However, since completing the project was a committed goal for 2020 up to Amazon's senior leadership "S-Team", my marching orders were to figure out a way to make it work.
I seem to recall spending at least a month during the summer of 2020 trying to evaluate every possible approach that would allow our team to complete our work on time. Prime Video had a practice of shifting engineers across teams to push things over the finish line, so we looked at every other team to understand how the overall timeline would shift if engineers from their teams joined mine for this project. We challenged every requirement we could think of from the initial project spec to understand if any piece could be delayed until after the main launch — a move which would allow the larger organization to declare victory, leaving my team continuing to work on a project everybody else thought of as done. I trained my team on a lot of the same Scrum practices we had used to good success on the Alexa project, and added several sprints of measured team velocity data to back up those estimates.
We planned a meeting to discuss a revised schedule with one of the VPs that accounted for the work my team needed to do. It was still aggressive (a couple of months ahead of what my conservative project estimate recommended), but I did believe we could figure out some simplifications that would allow us to complete in that timeframe. Going into the meeting, I was told that nobody at Prime Video had ever created a project plan that was as detailed, thorough, and data-driven as what we had. I didn't have a lot of confidence that even this would be enough to change the plan — completing the project in 2020 remained an "S-Team Goal" — but I wanted to make sure I had done what I could to communicate the risk. At the last minute, not only was I disinvited from the meeting to review the plan, but the edits I had made to the document were reverted.
At some point around the time this was going on, I got an email from Amazon's patent office. Turns out, a patent I had applied for on the very first project I'd worked with at Amazon had been granted after over ten years. The patent itself wasn't even a full system design; it was basically a specification for logging HTTP request IDs in a way that would allow operations teams at Amazon to associate all internal requests that were part of fulfilling a single incoming external request, while also keeping all those internal requests separate and distinct from one another. The email about this went to all the co-inventors on the project, and our management chain. One of the co-inventors on the project, Brad Porter, had become a senior Vice President at Amazon in charge of their robotics program; his manager was Jeff Wilke. None of my managers noticed the email, but Jeff Wilke did, and he responded with a congratulations to Brad and the rest of the inventors.
Life is not a Christopher Nolan movie, so we don't get to know what would have happened if events had happened slightly differently. Getting a small bit of recognition for an engineering project from twelve years prior, while simultaneously having my experience ignored by my current leadership team, came across to me as a concrete example of how much Amazon's culture had changed in the intervening time. Not long after this email, Brad Porter announced he was leaving Amazon for an AI startup, and I started to wonder about a different future, one where I put back on my engineer hat and started building code again.
(In case you're wondering about that massive rearchitecture project; I caught up with one of the senior engineers on my team months after I left. Turns out those original estimates that we'd put so much effort into, only to be ignored, were pretty accurate. The artificial EOY goal was missed. But you already knew that.)
1: Lessons I'm Learning
At some point after the initial push of software feature launches for the COVID shutdowns, we had an organization-wide virtual all-hands. They highlighted all of the work the team had done to enable theatrical film releases and multiple livestreams and concerts, and expressed appreciation for the hard work the team had done. In closing, one of the Senior Vice Presidents of the organization said something to the effect of, “I hope that this work is being done in a way that is healthy and sustainable for the team.” … and that was it. Leadership acknowledged that the work the team had done may have required some sacrifices and long hours, but ‘hoped’ the team would find some healthy balance. Another of my earlier managers at Amazon liked to quip that “hope is not a strategy,” but hope was all they had. During the remainder of my time with Prime Video I saw no meaningful efforts to assess employees' stress levels or work life balance or otherwise put in place mechanisms to ensure that the work we were doing remained healthy and sustainable.
From the moment I joined Prime Video, I found the culture of the organization challenging to deal with. During the interview process, my hiring manager and the director of the Santa Monica office both indicated to me that they hoped my Amazon experience would help them to redirect the culture, since they also didn't think the culture was healthy or even reflected the best of what was typical at Amazon. I've seen in the past how important it is to establish a healthy culture for an organization, and also seen that culture is often a reflection of how the top leadership acts. During my time on this team, there was a ton of turnover in senior leadership ranks. In fact, of the six managers in the organizational hierarchy between me and Jeff Bezos when I started, only one was still in that chain by the time I left. Some of those leaders were people I'd known and worked closely with at Hulu, which was probably the best company culture I've experienced so far. Despite that change, the culture showed little sign of changing. It's possible that I didn't give the new leadership enough of a chance to change things, but in talking with them, I very much got the impression that they were resigned to this being "just the way it is". It seems to me that once a culture really gets entrenched in an organization, it takes very focused efforts to redirect the culture.