Why I left academia, and post-experience views on industry
Also, some theory of what in life is meaningful
This post was originally titled “why I’m leaving academia”. I originally started writing it six years ago, just before I graduated from my PhD and went off to work in tech. I’ve managed to become disillusioned with a whole new industry since then, but I should keep this post in order.
1: Why academia?
Take a step back and think about what matters in the world.
Most of the time, our day-to-day is some form of maintenance. We cook meals and eat them, we do laundry, we take out the trash. A lot of things (including most jobs) can be seen as longer-term maintenance - farmers, construction workers, drivers and so forth are upholding the basic infrastructure society needs to survive. Then there’s education (on both the learning and teaching side), which is a meta form of that.
Some things are longer-term. Building a new train line or a factory isn’t just maintaining society, it’s adding something new that should make society work better in the long term. From the real long-term view this is still temporary - all infrastructure breaks down eventually - but that takes longer than the time between eating one meal and having to eat the next one. Having and raising children is an even longer-term thing, but again, there’s a sense in which it’s maintenance.
But if we take a step back, we start thinking - what are we even doing here? Surviving is nice, but shouldn’t we be doing some kind of long-term project1. Figuring out new math - understanding the basic building blocks of reality and idea-space - is a fundamentally meaningful thing in a way almost nothing else is. Physics and arguably computer science are a bit more contingent on our physical world, but researching them and getting new deep understanding is, in a sense, the one kind of job that’s true meaningful advancement of humanity2.
2. Why not academia?
The first answer to this is obvious: Terrible job market, low pay relative to the skill and effort involved, frequently toxic social environment. The environment is both highly isolating and entangles your sense of self-worth with your productivity, which leads to some pretty bad depression traps (especially since even the most brilliant and productive researchers I know all spent at least half of grad school feeling terrible about being unproductive)3. I had pretty bad depression for most of grad school and gradually started getting better once I left4.
But if we said working in academic research is good because of uncovering fundamental truths, shouldn’t that still be good, and enough for most people to push past the difficulties and do what is important in life?
So this is where the philosophical part comes in. What does it mean for humanity to know something?
The most obvious failure mode: You write a bunch of papers, they get well reviewed and maybe get into some journals, the ten people in the world who work in your field all think you’re smart. But anyone outside the field who wants to understand it still has to spend years studying the background to even understand the questions you’re researching, and by that point they spend more time on understanding your results than you did on discovering them5.
This wasn’t always true. A lot of the basic math results of the 20th century - measure theory, the basics of modern abstract algebra, information and computability theory - were very hard to discover, apply to a wide range of things, and are actually pretty easy to teach new students. In Gauss’s time, discovering the Fundamental Theorem of Algebra6 was so hard that he became famous for solving it (and, over his life, found four different proofs until he found one he felt was sufficiently rigorous). With the approach of modern math, I had a fully formal proof of it as a straightforward homework exercise in a third-semester course.
But it is true for most modern stuff. They say David Hilbert was the last mathematician to really understand all of math (presumably not every detail, but the main results in all the fields). He died in 1943. Since then, most additions are marginal. It’s not just that advanced math rarely influences applied uses, it almost never even influences other areas of math - I’m slightly weird because I was in combinatorics, but still, my actual PhD research rarely used anything beyond basic undergrad-level algebra and combinatorics.
So if we don’t really transmit information or add it to our universe, are we really discovering new things? I’m not a physicist but my impression is that physics is the same way - string theory is cool but so abstract and flexible it’s functionally incapable of making real predictions. And ML research is fun, but all the actual results of ML come from improvements in hardware capabilities and tinkering we don’t really understand.
3. Why tech?
So given that we despair on research, where to next? For most mathematicians, the next answer is tech7. It’s famous for having a lot of nice things going for it - low pressure work8, nice conditions9, good money10. But we want an ideological reason, and there is a convenient one: You get to build things. Not just research results in obscure corners no one will read - you build real things that people use, so they need to be good and work whether or not anyone understands them. Isn’t that cool? How well does it hold up?
There’s also another thing here: Organizing information is really cool and important. Having the internet means the academic research that used to just go in an obscure library archive where it’s never read now goes on the ArXiv and gets read by dozens of people. With GPT, this might get even better, as LLMs learn to incorporate new data and bring it up whenever necessary, so that instead of a fallible human reading your research then forgetting it, we’ll have a grand central repository of information able to apply it when needed. Like hardware is the bottleneck to AI, maybe AI will be the bottleneck to universal research access.
4. Why not tech?
In practice, not very well, for a few reasons.
First, the same kind of novelty pressure that applies in academia applies in tech. Tech people care, and derive status from, looking smart. Which means a lot of silicon valley is about deliberately making things unnecessarily complicated, using advanced or fashionable tools that aren’t needed, and trying to pose about how cool and unexplained your tech stack is. This often makes working at a tech company frustrating - some of the omnipresent lack of documentation and legibility is because maintaining legibility is hard, but a lot of it is because everyone wants to look smart and that produces strong incentives against both producing and consuming legibility in your products. This gets worse the further you are from the bottom line - it’s very bad at bigtech and somewhat better at startups. It’s generally worse in tech than academia, since academia is so isolating that everyone loves explaining their research while techies, well… there’s a reason Stack Overflow has the reputation it does.
Second, something like 95% of tech work doesn’t need to be done (Elon Musk may have been a jackass in how he went about firing 90% of Twitter employees, but there’s a reason the site didn’t immediately collapse). A lot of it is pointlessly duplicative (finding a new cool way to replace an existing product with a newer and worse one, or creating a shitty Twitter clone). A lot of it is writing unit tests for obscure features a customer asked for - this one probably does need to be done, but isn’t super interesting or useful to do (and is also very impermanent, once the customer migrates away from this feature in six months). This is true both within companies (for bigtech) and between them (for startups). Most tech companies most of the time will have one person doing work that actually needs to be done for every ten engineers.
And finally - it’s worth looking at the marginal contribution of an engineer. I’d guess the average engineer is worth about what they’re paid - a fairly high amount, but not amount commensurate with pushing forwards the very boundaries of human ability. And I think that’s actually pretty reasonable.
5. And so…
I ended up working in hedge funds, first as a dev and then as a quant researcher. This doesn’t solve all the previous problems, but it solves a few. People in Hedge Funds generally care enough about results that there’s less making things pointlessly complicated11. On a personal level, I like the work in terms of matching my skills and natural work rhythm. There is some objective metric - you’re here to make money, ideally through improving the market and providing value in an abstract sort of way - and it’s less duplicative than tech (a wasteful hedge fund generally doesn’t survive). While there’s a lot to criticize, I think the average hedge fund worker provides value commensurate or somewhat in excess of their (slightly higher than tech) salaries - you’re not saving the world singlehandedly, but you’re doing alright12.
It doesn’t solve the original problem, which was about finding meaning in what you do.
So here’s where being a bit older and more experienced pays in - I’m a bit more okay with that. I’m a bit more prosocial now than I used to be too, and more okay with doing my part in a larger society where different people do different valuable things instead of having to be at the edge myself. And as another effect of being more prosocial, I find more meaning in other people, which means I need it less in my work.
This isn’t my full answer to the fundamental problem of meaning. I have a longer and more complicated answer to that, that I should write someday in a different post if I ever get around to it.
and just to conclude things back from the top… academia isn’t something you should do if you think it might be good. It’s something you should try if you can’t imagine doing anything else. And even then, you might end up changing your mind about it - it’s a necessary condition, but sadly not a sufficient one.
Having children is the same way, and having children also reduces pressure to find meaning in other things in life. I’ve spent a lot of time in Israel lately, where people (unlike in New York) actually have kids, and they really are a lot more chill about meaning and achievement13. I recently talked to someone who said they were okay with having kids, but would only do it through surrogacy, since they didn’t want to put up with that kind of potential health damage. And I compare that to the way actual pronatalist people talk about children - the way my mom talks about having children - and itputs this general thing in scope: People who actually value a goal just do it, and pay whatever the price is. They don’t worry about what’s worth what.
My actual answers to this sort of problem are longer and belong in a different post. This one is about justifying research as a thing humans should do, but that doesn’t mean it’s the one fundamental goal.
I guess other physical sciences are okay too.
Richard Feynman talks about feeling this way in his first real university job in Surely You’re Joking, Mt Feynman. And this was after both finishing a PhD and being a a rising young star in the Manhattan project.
Although this was far from uniform and had good and bad periods (including the last year). Leaving a bad environment is a good start, but then you have to actually find a good environment that has both social support and a sense of meaning, and neither of those are easy to find.
This isn’t only true in super-abstract algebraic geometry style things. My PhD was in combinatorics and I still have trouble explaining it to people with a general math background.
Any polynomial P(z) over the complex numbers has a root, that is, a complex value u such that P(u)=0.
Or finance, which is were I eventually ended up. But that’s all the way down in section 5.
This reputation is mostly just a marketing trick by employers. Tech is highly competitive and pretty high-pressure even at companies that try to sell themselves as chill.
This one is mostly true and quite nice. Sometimes you get free strawberries.
This one is… ambiguous. On the one hand yes, a lot of people in tech make decent money, but so do nurses, therapists and AC repairmen - after you adjust for tech disproportionately employing extraordinarily talented people, the tech wage premium isn’t as sharp as you might think, although it does exist.
Less, but far from none. Humility and simplicity are severely undervalued in finance.
Well, except for crypto, which is an unholy abomination that joins everything bad about tech with everything bad about finance.
If still pretty stressed and miserable about everything else, especially the weather. It’s kind of a terrible place.