Job 7: Google DoubleClick Search
In the early 2000s, Google was considered one of the best employers for software engineers in the world. Their reputation for generous pay, a fair work-life balance, talented coworkers, and generous perks was pretty much unequaled. Shortly after the RDS launch, late in 2009, a recruiter from Google reached out to me about working at Google. I interviewed, and was offered a role, but the offer wasn't significantly better than what I was making at Amazon, and I wanted to see what happened after the RDS launch.
Due to apparently a significant number of declined offers, Google revisited their pay scales during 2010, and towards the end of the year, they reached out and extended a revised offer without me even needing to interview again. Despite the success of the RDS launch and my role in it, my pay at Amazon hadn't changed much that year, so Google's revised offer was more tempting than the initial offer the year before. After a bit of negotiation, we came to an agreement on a decent pay increase over what I was making at Amazon, and I decided to see if Google's reputation was all it was cut out to be.
The project I joined was a significant internal project to completely rewrite an advertising product that Google had acquired when they bought DoubleClick, and which my understanding was they had somewhat neglected since that acqusition. DoubleClick Search was a cross-platform search advertisting tool that allowed large advertisers to manage ads across the major search platforms — Google, Yahoo, and Microsoft at the time. Google's focus was, naturally, their own advertising products, so it's understandable this product would be neglected, but customers really valued it, and paid good money for it. At the time I joined, this product was still running in DoubleClick's original data centers on their platforms, and our first goal was to rewrite the whole thing on Google's stack and deploy on Google hardware.
In some of my first discussions with the team manager, he indicated that he liked that I had some front-end experience on my resume and wanted me to join the front end team. Despite not having done any significant front-end work during the 3 1/2 years at Amazon, I had enjoyed that type of work earlier in my career, and thought this would be a good chance to get back to that. The core frontend team for the product was already filled with some fairly experienced Google engineers, so they had me working on the front end for the bid optimization part of the product, and I joined the bid optimization team, which was mostly anchored by a couple of experts in optimization algorithms.
The initial rewrite of the product completed, if I remember correctly, 5–6 months after I started. The team earned several accolades at Google as one of the most significant engineering projects that year. I had done my part to unblock the launch, but mostly I remember copying a lot of what had already been done by the core frontend team to the bid optimization UI. Not too long before we were scheduled to launch, I was working on adding some charting components, and noticed that some of the data points had some really screwball values.
I was trying to chart some sort of efficiency ratio — something like advertising spend divided by attributed revenue for an ad campaign — and there were data points where this ratio shot to infinity, making the graph pretty useless for making visual comparisons. I confirmed that the common logic already handled zero values, but when I looked at the data, the values in the database for revenue earned were not zero, but something like $0.000001. That didn't make any sense. I went digging around the code and eventually the following:
private static final long ZERO_MICROS = 1L;
The system stored currency amounts in the database as micro-units (e.g. one millionth of a dollar) instead of whole units or cents to avoid dealing with floating point arithmetic and rounding errors. (This is a good idea, and I have followed the practice in subsequent projects dealing with money.) Apparently, at some point in the history of the product, a software engineer decided that single micro-dollars were small enough to not matter, and easier to deal with than divide-by-zero errors. What this meant was that anyplace where the database was supposed to say "0 dollars" (as in "the total revenue attributable to this campaign during this hour"), the database would say "1 micro-dollar" instead. Due to the magic of aggregation, this could be several micro-dollars off — still a rounding error that wouldn't show up if you were rounding to cents, but an error nonetheless. In the graphing library, while we did have a special case to avoid dividing by zero in that code, that special case didn't treat small values of micro-dollars as zero, leading to my near-infinite data points whenever there was zero dollars in the denominator.
There were two basic approaches to fixing this problem I considered. First, I considered correcting the data. There was a lot of historical data, and I wasn't sure if there was other code that would have started to crash if fed a bunch of zero values. (Presumably, divide-by-zero was a potential issue, otherwise the code wouldn't have been this way in the first place.) Second, I could correct our aggregation logic to avoid accumulating micro-dollars and treat any sufficiently small value of microdollars as zero. I chose the second option. I remember talking at least a little bit with other members of the team, and don't remember there being a ton of vocal resistance; more a reluctant acknowledgment that it probably made sense, and hopefully we would have time to fix it properly at some later point. So that's what I did. It was ugly, but it worked, and ended up being the most memorable code I wrote that year.
1: Lessons I'm learning
One of Google's perks that I worked pretty hard to take advantage of consistently during my time there was the employee gym. I had a lengthy commute up the 405 from Puyallup to Kirkland, and I took that commute pretty early in the day so I could get in a workout in a quiet gym. This meant that I was at my desk working before most engineers, and would, accordingly, leave earlier than most in the afternoon. Towards the end of my year at Google, I had a meeting with my manager and somebody from HR show up on my calendar. At the meeting, they told me they were putting me on a Performance Execution Plan because I was rarely at the office. I asked for an explanation, and it turned out that my manager would come by my desk around 6PM on a regular basis, never see me, and without taking any steps to resolve this issue on his own, decided I was a performance problem. (Google has a pretty good Calendar product for scheduling meetings.) I want to be clear that I don't look back at this year as a year where I did my best work. There are a lot of things that could have gone better, but this was definitely a vivid lesson in the importance of owning my own career, and making sure people see the work I do. At the time the meeting happened, I was already interviewing with my old teammates to go back to Amazon, which is what I ended up doing a few weeks later, but I often wonder if things would have been different if I had realized that I hadn't had a one-on-one with my manager and had simply put time on his calendar before things reached this point.
When many people hear I worked at Google, they ask about the perks. They had great snacks, free and delicious lunch every day, an onsite gym (which I used), ping pong tables, a climbing wall — really, too many perks to list. I worked at their location in Kirkland, WA, but visited the Mountain View headquarters a couple of times. There was a bit of an odd culture around actually using some of these perks, though. I remember a couple of times walking by one of the game rooms with other members of my team, and hearing snide comments about how the Googlers taking a break to play ping-pong or foosball clearly weren't working hard enough, and maybe didn't actually belong there. Opinions will also vary a lot on this, but I even found the free lunch on campus a little constraining compared to going out with my teammates for an offsite lunch from time to time. In the end — and I fault myself for this — I look back on my year as the year where I learned the least in my career, and ultimately the least fulfilling.