Benchmarking Generalization: How AI Learns Beyond Training Data
Understanding how large models generalize beyond their training distributions through inference-time computation.
A new episode of Inference Time Tactics is live.
Rob and Cooper from Neurometric talk with Yash Sharma (PhD, Max Planck Institute) about how modern models really generalize—and what happens when they move beyond training data.
They discuss:
• Compositional generalization vs. memorization
• Why scaling compute alone won’t produce breakthroughs like cancer prediction
• Benchmarking real-world generalization with Let It Wag
• How inference-time compute helps measure model understanding
• What these findings mean for builders of agentic and multimodel systems
If you’re building or evaluating AI systems, this conversation clarifies how generalization really works—and where today’s models still fall short.
Listen 👉 inferencetimetactics.podbean.com