5harathFounding Growth Lead, Willow Voice

The Future of AI

19 Mar 2024 · 1 min

Watch on YouTube

1:00

In this dialogue, Speaker A outlines the current paradigm of large language model development: expensive pre‑training followed by reinforcement learning from human feedback (RLHF). They note that after deployment, models are largely static; developers can fine‑tune them but cannot easily teach them new knowledge, so they must rely on clever prompt engineering to inject context. Speaker A contrasts this with how the human brain learns, arguing that future AI systems should evolve continuously—where inference and interaction dynamically influence the model’s output—rather than remaining frozen at a fixed training cutoff. Speaker B offers brief acknowledgments.

Chapters

So right now you pre-train a model, it costs a lot of money to build a competitive, big large language model, and then you can do reinforcement learning from human feedback on it. And that kind of gives it a particular flavor, right? Mm-hmm. And then kind of it just gets put in developers' hands and then there's not a lot that can happen. You can sort of fine-tune it, but you can't really teach it new things through fine-tuning. So you basically have to inject context in the prompt in some sort of smart way and just have it apply its reasoning to that context that you injected. But, um, that's not how the brain works, right?
Mm-hmm.
I'm both being changed by this conversation and also you're running inference at the exact same time on like the process of running inference changes the output. And so it feels inevitable that the best models of the future will kind of feel, will behave more like that probably. Um, mm-hmm. And we won't have these like frozen in time, oh, here's where the training dates cut off.