This was interesting from DeepMind’s Sholto Douglas about the reality of working in AI labs. They have billions of dollars flooding into them but they’re also scaling rapidly in a slightly chaotic way, working in ways that constantly throw up more things to explore than their existing capacity allows:
I also think that it’s underappreciated just how far from a perfect machine these labs are. It’s not like you have a thousand people optimizing the hell out of computer use and they’ve been trying as hard as they possibly can.
Everything at these labs, every single part of the model generation pipeline is the best effort pulled together under incredible time pressure, incredible constraints as these companies are rapidly growing, trying desperately to pull and upskill enough people to do the things that they need to do. I think it is best understood as with incredibly difficult prioritization problems.
https://www.dwarkesh.com/p/sholto-trenton-2
It connects to something Mark Zuckerberg observed here:
What we basically found was that we were bottlenecked on compute to run tests, based on the number of hypotheses. It turns out, even with just the humans we have right now on the ads team, we already have more good ideas to test than we actually have either compute or, really, cohorts of people to test them with.
Even if you have three and a half billion people using your products, you still want each test to be statistically significant. It needs to have hundreds of thousands or millions of people. There’s only so much throughput you can get on testing through that. So we’re already at the point, even with just the people we have, that we can’t really test everything that we want.
Mark Carrigan 