“There is an epidemic failure within the game to understand what is really happening…People who run ball clubs, they think in terms of buying players. Your goal shouldn't be to buy players, your goal should be to buy wins. ”
Peter Brand, “Moneyball”
Moneyball is a sports drama movie about how the 2002 Oakland A’s used Pete's approach to evaluate and acquire extremely undervalued players on a shoestring budget. The team would go on to break an all-time record 20 win streak record to win the American West League title. The reason why these players were so undervalued was because they were overlook for a bunch of superficial reasons ranging from looks to charisma to pitching technique. But the one thing that Pete did value that nobody else did was “getting on base.” Basically, were they efficient at scoring points in their role.
It’s funny because I rewatch the movie every year or so and there’s a different aspect of the movie that speaks to me every year. This quote happens to be the one this year. Yet I think it perfectly encapsulates the story of what I want to tell you today: a fundamental misunderstanding that is occuring in the industry today.
Actually, it captures 2 stories of misunderstanding: the engineers and the companies that hire them. Let me say in no uncertain terms that there is an epidemic failure to understand the skill level of engineers in the industry today. The industry knows how to hire profitably but not intelligently. And in that misunderstanding, they vastly overprice the salary of their engineers relative to the engineer’s competence. This presents a wonderful arbitrage opportunity for top tech companies. If companies want to reduce their costs, they can start by properly understanding who they are hiring and where these so-called “wins” really come from.
Consider this as if software engineering was a company and I’m writing to you as its CEO in an annual newsletter. And having been in the professional industry for close to 10 years now, I want to talk about the changes I’ve noticed and how we got to where we are today.
Honey I Shrunk The Kids Engineers
Let’s start with the most egregious offenders of the 2: the engineers. Strange that I’d start with my own customers. Not really.
This is just some tough love. And hopefully, by learning what the majority are misunderstanding, you engineers can take steps to becoming more valuable. Computer programming, which was once a niche hobby for nerds and paid very little, has become a big business. It’s almost akin to that of a bank manager in the 1970s. Because of the extremely high influx of CS graduates in the past 5 years, the engineering industry suffers from an extremely skinny long left side of the Dunning Kruger effect.
For those who do not know, the left side of the Dunning Kruger effect is the area of high confidence and low competence.
This influx is driven by extremely interesting market conditions. For one, software engineering salaries have never been higher because the number of companies at scale has never been richer. With high 6 figure salaries upwards of 200k right out of school, it is a very high reward for people of relatively low skill and moderate intelligence. It's almost like the frenzied days of the dot com bubble in 2000 or finance jobs pre-2008. Let’s hope we never get to repeat those again. This isn’t helped by the federal interest rate policy being set close to 0%. Unlike 2000, companies that should not exist because of extreme unprofitability are being kept afloat by cheap loans. They use those loans to finance their operations which include paying engineers FAANG-like 6 figure salaries and benefits in hopes of achieving profitability one day.
At least on a macroeconomic level, we have a surplus demand for engineers that shouldn’t actually exist. But I’m getting ahead of myself. I’ll address the companies later on. The engineer quality issue primarily stems from 2 factors. First is the relatively young age of the participants. Most people when they graduate have never accomplished anything significant in their lives and have never made a significant investment in themselves. As a result, they don’t know how to improve or don’t care to. Consider another profession where the starting salary is $100k: baseball. The athletes there spend years investing in themselves, training, studying other players, and investing in one’s own playing ability just to get a shot at a 6 figure contract that lasts, on average, less than 3 years. On the other hand, we hand kids 6 figures out the gate without this ritual. For most of their life, the path has been painted for them by someone else and all they had to do is stick to it. There is no experience or long-term vision or understanding of value with most of these people. They fundamentally lack an understanding of putting effort in to increase one’s value on one’s own volition. And this 6 figure job contract could potentially last the rest of their life.
Most believe that if they follow a prescribed path like they have up until this point (go to school, get good grades, get a job), they will get promoted and rewarded. Considering that most engineers are people who have done well in school, schooling and the “golden path” is all they know. Someone told them how to do the basics of coding but few have taken it upon themselves to improve beyond the basics.
How this translates into engineering prowess is that a lot of engineering is figuring things out on your own, both in terms of the codebase and your own skill. Lots of things are undocumented and not everyone has the answers. Not everyone knows how to approach every single problem and that’s okay. What matters is that you have the skills to figure it out.
Yet I don’t see that: I only see people who ask others to solve their problems for them and copy-paste other people’s code or get someone else to do the work for them.
One project I was working on, I had to actually tell a backend engineer how his own techstack worked and implemented his work for him, despite the fact that I am a mobile engineer and having been at the company for only ⅓ of the time he was, not even a year. It was my responsibility to do so to deliver a project so I did it. If I was in his shoes, I’d be embarrassed beyond belief and vow to work harder. 3 months later, he got promoted.
******
The second factor at work is the opinions of the most outspoken. The internet influencers, if you will. Utsav has done a wonderful video on why consuming endless Youtube content is ruining your career. I think it runs a bit shallow but the surface level reasons are there. But the takeaway is that most content on Youtube is shallow and people trick themselves into the left side of the Dunning Kruger curve by consuming nothing but surface level content like creating Instagram clone tutorials and influencers giving random advice. I want to dive into the core reason this issue even exists.
With tech influencers who say they’re the top percentile of engineers and with brand names like Google under their belt, marketing and perception becomes the badge of competency. Engineering is hard enough and by the time you become competent, it's already too late and these influencers move onto the next thing. And the entire industry is designed to produce consumable content, not actually useful content.
Even the credentials are only good enough to entitle most influencers out there to teach the basics to the average person.
One called himself a Google engineer who could land offers but only ever landed internship offers before launching a course on nailing technical interviews. One job hopped for 10 years. One only had 2 years of Google experience and 3 months of FB experience and now markets system design preparation. One wrote tutorials on how to write app clones and markets himself as a high quality programmer without having any professional experience. And one wrote a book on how mobile development at scale that is, at best, a surface level walkthrough of common practices and flow of productivity that he had no part in creating.
Would you trust these engineers to guide your career once you’re a year or 2 into your career? Probably not. In fact, chances are you can find someone even more qualified sitting right next to you at work than them to give you career advice. Now, when you consider them not as engineers but rather as marketers and salesmen, the actions and pictures make more sense. They peddle their credentials not to deliver quality content but rather get the initial viewer’s attention to pitch their product and content for their sales funnel. When they put out content, ask yourself if they are putting out the content for views and public perception or because they truly want to create value in the world? Where are the incentives? Do they have a vested interest in turning basic engineers into elite engineers? Or just to incrementally improve everyone else’s lives by a tiny amount with information and milk that profit? Would they even stake their life savings on their beliefs? And most importantly, do they have integrity and are they consistent? Consider that when you consume their content, you are implicitly entering into a business transaction. You are trading your time and attention for information and value. And one thing that I’ve had the benefit of learning is that you can never make a good deal with a person with the wrong motivations. The way the incentives are aligned, these influencers don’t put out quality content: just disposable content that’s marginally profitable and holds your attention for just a second longer. It's why many of these influencers move into non-tech: because their timeline and content farm runs out and/or doesn’t sell well. I dare you to name one significant influencer who has stuck with engineering content for more than 3 years and hasn’t branched out into entrepreneurship, crypto, or lifestyle branding.
But I honestly don’t blame them either. In fact, I actually respect these influencers. It takes a certain level of hard work, dedication, and brains to optimize viewership and attention and guts to leverage your entire life into that opportunity. Something I wholly admit I do not ever want to do.
The problem is with the customers. You. If the customers demanded a higher standard for content, these influencers would disappear overnight. But if the customer just wants consumable, easy, and disposable content, who are these people to deny them that? If you can get away with creating something that is valuable but boring and surface level for easy money, then why not?
That’s why I don’t play that game. It's not interesting and it's something that I don’t believe is valuable, even if the market thinks it is. That content is already free in abundance in school lectures, textbooks, and even your coworkers once you’ve put your foot in the door. Anyone can tell you what a load balancer is. Few people can actually use one intelligently in a system design interview to get a job. And even fewer can actually understand and build out Instagram at scale. I’d rather be of the latter populace. The reality is the influencers’ audience tend to consume these videos because they are looking for easy shortcuts to success when the reality is that it takes a long time of self-investment and making a lot of mistakes. They have no desire to improve and rather grift, just like the people they spend time watching. The willingness to be wrong a thousand times before being right once is not there. I dare to call this “an audience that matures out of the content.” Even if they move from engineering to something more mature like finance, the delivery system and emotional appeals will still remain the same. Alas, the folly of human nature. But it works. What tends to hold people back is that most people don’t own their failures (“Its the industry that’s keeping me down! I’m being underpaid even though I only have 6 months of experience!”). They don’t take responsibility and nobody tries to consume quality content. Or demand better answers. They are literally paying for the answer they want with their attention. Because that’s just too fucking hard?
******
Generational Failure
I may sound like a boomer complaining about how bad things are with young professionals. But keep in mind, I’m only 28. I belong to this generation. But I don’t betray it either. If anything, it's the generation of engineers that have become the older managers that bear a great deal of this responsibility. Let’s talk about promotions. In its current form, the ladder from L3-L5 doesn’t help either. This is not just at Facebook where there is a 24-36 month promo expectation but everywhere else. Most promotions at this level just require the engineer to stick to the promotion guidelines and check off the rubric.
In fact, most managers are very hands-on in guiding the engineers from L3 - L5, laying out projects for them for promotion and telling them what to do. It's almost like a game tutorial. As long as they finish the task, they all but guaranteed a promotion.
But what happens after that? What happens after the L5 promotion? Consider the fact that if people have never taken control of their careers up until then, they would certainly have more to lose to try and learn how to now. You’ve spent 8 years copy-pasting code and getting promoted for that. Are you going to risk your annual performance and 6 figure salary on trying to implement new ideas to progress your career when you have a wife, kids, and a mortgage? Probably not. That’s also why L5 is the terminal level: because beyond that, the career is in the person’s hands and not very well defined. It’s up to the individual to set that path for themselves and to figure it out.
In fact, most managers are only L5’s themselves before transitioning. And this is why so many L5’s become managers. At some point, they realize they just cannot cut it without someone else telling them what to do and repeating it blindly. They’ve only ever survived off the fact they were on the team longer than anyone else. Simply put, they only knew more as a result of survivorship and tenure. They never bothered to reinvest in one’s self and grow on their own or improve their mental models. As a result, they because they never actually developed the technical skills. It’s simply easier to teach someone else the same blind path they followed of copy-pasting code and shipping the product. It's the equivalent to following a Youtube tutorial and then teaching that Youtube tutorial to someone else and labeling yourself as a guru. Almost like what the influencers are doing now.
And it's why there is a natural inclination to thinking that software engineering is only ever a young person’s game or that the natural path is into management. The old get phased out because they aren’t any better than the younger generation who will work faster, put in more time, and for less money.
The reality is that most people cannot keep up the façade of competence for a long time. It’s simply easier to tell someone else what to do than it is to do it yourself and manage them. Managers handhold junior engineers through the same path they themselves have walked.
Because of this, I think it's ridiculous for people to switch so often between L3-L5. Someone is already basically telling you what to do and spoon feeding you projects for promotions. If you have to switch to get promoted at below the senior level, you’re actually trash. No seriously, you have the actual skill of garbage. Ask these people what they’re doing at work to make themselves a better engineer and you’ll hear crickets. Or some stupid answer that focuses on what they can’t control like “I need to play the office politics game” when that’s a much smaller part of the job than they realize.
The truth of why people aren’t successful in engineering (or anything in life) is almost always “you’re not good enough.” But people assume they are good enough and neglect the technical mechanics. They over optimize and focus on the wrong things (should I study x, should I switch companies after 2 weeks, etc.), play the short-term game, and don’t have a long term vision for themselves. They grift from area to area and this partially explains why people jump from company to company: they cannot cut it there, make no effort to improve one’s self, and think the answer lies in a higher paycheck elsewhere that they’ve tricked that company to pay. Skills are irrelevant to the sales pitch they give at interviews.
******
Adult Daycares
There is also a systematic failure to understand what a company really needs in an engineer. Its an abject failure perpetuated by a desire to push the revenue numbers up that is wholly necessary. It's a pervasively sick game that is extremely rational that naturally encourages a bimodal distribution of skill.
Let me clarify this.
For these big tech companies that can dole out these 6 figure salaries, they often are playing with a significant amount of leverage. For maybe 200 lines of code and 2 weeks of planning, you could deliver a product to 100 million users that will generate the company 15 million dollars. To prevent these individual products from breaking the entire platform, the architecture must be well defined and rigidly separate from the products. Individual sandboxes, if you will.
Therefore, what you end up with is an army of product developers who try to eek out a bit more revenue through disposable products. And then you have the infrastructure developers who have to maintain the codebase where these disposable products don’t blow up the entire pipeline or efficiency of the codebase. The former tends to be very rinse-and-repeat mindlessly as all these products really boil down to the same tired copy-paste formula as defined by a standard architecture recommendation.
This is where we get to where the wins are actually won in engineering. The company thinks its in the engineers who write the disposable products. This is wrong. The wins are won long before the product engineer writes his first line of code. Its in the infrastructure. A good efficient infrastructure embraces the practices of its corporate culture and the realities of engineering. In most cases, this means that a good architecture is one that can have engineers be interchangeable and replaceable. The most efficient way to accomplish this is to reduce the learning curve and to simplify everything so that anyone can use it.
Once that happens, all the product engineer has to do is copy paste the least amount of code possible to get the job done. And the more efficient the architecture, the less skill your product engineers need to have.
That’s not to say products aren’t important. Rather, some of the products are quite complex and fascinating. Payment information processing and banking are amazing. It's just that a majority of product engineers are glorified code monkeys. You cannot give a code monkey the job of managing the developer infrastructure but you can have a code monkey generate advertising products. Not to mention, is this really what you want to spend the most brilliant minds on?
And they are wholly necessary for the company. If an ad or product can generate 1-2% more engagement, that’s millions of dollars for the company. So it's in the company’s best interest to keep them sated with a high salary and to just grind the same meaningless disposable code over and over again, irrespective of the individual’s skill. Throw more people at the problem. The only thing you’re limited by is the salary you pay them.
This is why the over-standardization of the interview is completely wrong. You’re trying to test for a financial analyst or a logician when you really need a code monkey. What you have is a company that’s interested in grinding out more products because it’s profitable. You have a market full of people ready to copy-paste code. And you have an engineering braintrust who are trying to find intelligent people who are rational logical problem solvers. No wonder the vast majority of the criticism is that it does not test what a person does on the job and demand a practical exam. Because the vast majority of people just copy and paste code they don’t understand. And ironically, they’re right: because that’s probably what they will end up doing. What you end up with are people who luck into these companies simply by the law of large numbers. Only a minority are actually able to carry the burden of a bunch of developers who don’t know what they’re doing. Yet that is what is holding up the ship and is necessary for the company to remain and grow in profitability.
For all the harsh remarks I’ve written, I also want to make it clear that this is not a significant problem that needs a remedy either. For every bridge that is planned, built, maintained, managed, and repaired, there are millions of cars that are driving over it to get to some place. The bridge-maker doesn’t know where they are going. Nor should it be. His job is to create an efficient roadway for others and his job would not exist if there was nobody driving on the roads.
A company that overpays for code monkeys and a engineering braintrust that tries to create a filter so only the best pass. But who they’re effectively hiring are code monkeys who can also count to 10. A remarkable code monkey nonetheless.
******
COVID Special
In the wake of the “great resignation”, companies have actually seen a significant increase in their rejection rates. Anecdotally, this has ranged from 7-15%.
How can that be? With so many great developers now on the market, you should see companies frothing at the mouth to extend the many talented engineers a handsome offer?
If you paid attention to the past few pages, you’ll begin to understand why.
The top people who are paid handsomely are burnt out from carrying the people at the bottom who can't be bothered to be a proper engineers. They have been compensated highly and with the insane runup in stock price in the past year (Google alone went up 69% in 2021), they have enough to retire on.
Seeing the high number of resignations and the news of a boosted salary, these subpar engineers begin to apply to look for a higher salary elsewhere. However, they cannot make the mark and as a result, we see a high number of rejects across the board. These skew the numbers for everyone.
If this is not the case, then you must believe that the lower quality engineers are quitting in droves. In which case, this shouldn’t warrant a higher salary increase at all.
With a large number of engineers quitting, this forces the next highest performers to carry more load. They quit as well from carrying the people at the bottom and the cycle repeats while the company tries to figure out how to hire more people with the same inefficient logic as before.
Again, you don’t need geniuses. You need geniuses for the infrastructure and code monkeys to drive the products.
Some will defend themselves and say they only want to work for pre-IPO status companies for the money. But that should tell you where the incentives lie. For most people, its not enough that they should receive a living wage. Its to become rich off of a lucky IPO. The problem is when you’re paid grossly disproportionately to your skill.
It's the same as gambling. You’re playing a game where you have no influence on the outcome hoping to strike it big. A majority of engineers are gambling with their careers. They are not working on something they firmly believe in nor have any intention of improving their value.
I would be remiss to talk to someone who can only talk about engineering as far as the latest IPO. And I would hope he learns what Discounted Cash Flow is.
******
Can This Continue?
If a majority of engineers are gambling with their careers and running the clock until they become irrelevant, can this trend of throwing engineers to produce disposable products continue? For the moment, yes. And I believe that this will continue. Even 10 years from now, a 200k starting salary will still be a top tier salary. What is more likely to happen is that engineering companies will wise up to the fact that anyone can do the job and it doesn’t require passing an exam.
The most efficient course of action for companies is to cut costs and create 2 tracks of salaries: the product engineers and the infrastructure engineers. Or the people who write disposable products and those who don’t. The only difference is that the former will have 70% of the latter’s pay grade. You don’t need to pay a code monkey the salary of a thoughtful competent hard worker. Voila, you’ve saved 30% of your engineering salary costs while still getting the same quality and value from your codebase and workforce. Whether or not this goes over well with engineers is another question. But market supply and demand seems to dictate that this would be the most rational course of action. Moreover, this speaks to a more fundamental issue in business: how to hire the right people and this is something that I think a lot of MBAs get wrong. There are a lot of silly ways that people are taught to value a business or human resources ranging from employee output, return on invested capital, hours worked, etc. But ultimately it boils down to one simple human factor: how much a person cares. Having coached well over 100+ engineers and seeing their career trajectory, I can tell you one thing for certain.
That a person who cares is one of the best investments you can make. That person is worth their weight in gold alone.
If you hire someone who doesn't care, you are hiring a mercenary. Which is essentially what these product engineers are. But unskilled mercenaries are worthless while a skilled mercenary cannot be trusted to suppress his own ambitions. What we’ve done today is treated the skilled loyal troops equal to the unskilled mercenaries in the pursuit of efficient profits and let a meager bonus system attempt to distinguish the two. It's extremely disheartening to see but a consequence of corporate America’s desire for profits. It permeates in the company’s culture and if you want to really ask yourself what a leadership’s priorities are, just look at a company’s Slack channel and see whether they’re talking about the stock price or the quality of the product. Despite this, for those who truly want to advance in their career and become a wonderful engineer, it’s actually never been easier. Mercenaries get shown the door or leave on their own in pursuit of the next big paycheck. Simply staying, showing dedication, and most importantly investing in one’s self gets rewarded in the long run. It may not be a year or 2 years down the road but as long as you keep learning new things and growing in responsibility, you will become bigger than anyone can imagine. I hope my own career track and the track I’ve put others on proves this. In 2011, I was working for free in a small sweaty 300 sqft office in Menlo Park for no pay. But I spent every moment of every day for the past 10 years investing in myself, learning new skills outside of coding, and sticking with one tech stack and one primary language. I have always been paid on the lower 25% end of the market (according to levels.fyi).
Simply by following that formula of asking how I could become more efficient and better at delivering projects, I have become one of the youngest engineers responsible for the infrastructure of not just 1 but 2 apps belonging to multi-billion (multi-trillion if you count Apple). I have done very well for myself financially as well. Without investing in myself and seeking to improve my skills as an engineer, I never would have been afforded these opportunities and I am grateful that I did. Luck and timing did have a big part to play but I would like to think that it is my skill that prepared me and gave me these opportunities to begin with. I would encourage those who read this letter to do the same. Invest in yourself and always strive to improve. Because the greatest return on investment isn’t in the latest IPO. It's you. It's your skills. It's how irreplaceable you become. No matter how much value the market puts on the wrong ideas and practices, it will always be willing to pay top dollar for rare and wonderful talent. No matter how high a salary a company pays, people will always protect and defend the competent who they can rely on when shit hits the fan. So why not become that?
January 4th, 2022