From the Bitter Lesson of AI scaling to the unsolved mysteries of protein folding, Google DeepMind’s Pushmeet Kohli and Biohub’s Sal Candido are rethinking what it actually takes to build AI that truly understands biology. In this special roundtable hosted by Brandon Anderson, they explore why AlphaFold’s breakthrough was just the beginning, why simply scaling compute and data won’t solve biology, and how next-generation AI models will fundamentally transform our understanding of proteins, cells, and human disease.
We dive deep into the future of AI-driven biology: uncovering scaling laws in biological data, balancing scientific intuition against general-purpose architectures, why protein structure prediction is far from fully solved, and what is required to build predictive models of living systems. Pushmeet reflects on the lessons learned from AlphaFold, the limits of human interpretability, and whether future frontier models could understand other AI systems better than humans ever could. Sal explains why protein language models may already hold untapped scientific knowledge, how biological modeling must expand beyond individual proteins, and why fulfilling Biohub’s mission to cure all diseases demands thinking in terms of 10x quantum leaps rather than incremental tweaks.
Key Discussion Points:
- The “Bitter Lesson” in biology: Why scaling compute and data alone is not enough
- Why discovering the right scaling laws matters more than blindly increasing model scale
- How low-quality metagenomic data can actually improve protein language models
- Why AI researchers often optimize for readily available data rather than the most critical scientific questions
- Lessons from DeepMind: Striking the right balance among modeling, data generation, and domain expertise
- Why building a virtual cell requires fundamentally different datasets
- AlphaFold’s hand-crafted architecture and the role of scientific intuition
- Why high-quality data matters far more than simply having more data
- Inductive biases, scaling laws, and the future of specialized AI architectures
- Why we haven’t yet entered the post-Transformer era, and how architectures continue to evolve
- Why AlphaFold has not truly solved the protein folding problem in full
- Protein dynamics, intrinsic disorder, and the limitations of static structure prediction
- How cryo-EM micrographs can unlock richer biological representations
- Moving from single-protein models toward models of entire biological systems
- Feynman’s famous principle, and why AI can now create things we do not yet understand
- The latent biological knowledge hidden within protein language models
- Why trust and uncertainty calibration matter more than full interpretability
- Whether frontier AI models can explain other AI systems better than humans can
- When AI will deliver a 10x to 100x acceleration in drug discovery
- Why curing all diseases requires aiming for 10x leaps rather than 10% gains
Pushmeet Kohli — Google DeepMind
Sal Candido — Biohub
- Biohub: https://biohub.org/team/salvatore-candido/
- X: https://x.com/salcandido
- LinkedIn: https://www.linkedin.com/in/salcandido/
Brandon Anderson — Host
Timeline
00:00:00 Introduction: The “Bitter Lesson” of Biological Data
00:01:00 Finding Scaling Laws and the Right Data for Biology
00:04:54 DeepMind’s Bitter Lesson: Problem-Solving vs. Model Scaling
00:06:50 AlphaFold, Data Constraints, and Building the Virtual Cell
00:09:52 Hand-Crafted Architectures vs. Scaling Compute
00:11:53 High-Quality Data, Inductive Biases, and Model Design
00:13:39 Beyond Transformers: The Future of AI Architectures
00:14:38 Why AlphaFold Hasn’t Fully Solved Protein Folding
00:17:39 Protein Dynamics, Design, and Cryo-EM
00:19:09 From Single Proteins to Whole Biological Systems
00:20:43 The Feynman Principle: Creating What We Do Not Understand
00:22:18 Hidden Knowledge in Protein Language Models
00:23:48 AlphaFold, Trust, and Interpretability
00:25:56 Can AI Understand Other AI Models Better Than Humans?
00:27:43 When Will AI Truly Revolutionize Drug Discovery?
00:29:47 Why Curing All Diseases Demands 10x Leaps
Transcript
Introduction: Is There a “Bitter Lesson” in Data?
Brandon Anderson [00:00:04]: Great to be here. This morning has been genuinely inspiring, with so many incredible announcements. I truly believe the future of biological science is unfolding right here. This is the modeling breakout session. We are all modelers, so talking about data comes naturally.
When it comes to data, one thing I like to reflect on is how to scale it properly. That brings to mind “The Bitter Lesson,” but framed through the lens of data. For those outside the AI sphere, “The Bitter Lesson” refers to the realization that general methods that leverage computation and can scale will ultimately win out. As long as you can scale them far enough, they prevail. So my first question goes to Sal: Is there a “Bitter Lesson” in data?
Sal Candido [00:01:00]: Absolutely. Clearly, for it to work, you have to have the right data. One common misconception about scaling laws is the assumption that they are ubiquitous and always present. In reality, a large part of the work lies precisely in finding that scaling law in the first place. Much of what we do is trying to figure out: Under what circumstances does pouring in more compute and more data actually yield better results?
It’s amazing when you find yourself in that regime, because once you do, you can just turn the crank. It turns into an engineering problem, which is what I love. It depends on model architecture, but to a very large extent, it comes down to data…