AI Memorized the Internet — Yann LeCun (Futurology)
| Author | Yann LeCun, interviewed by Dawn Nakagawa |
|---|---|
| Published | November 17, 2026 |
| Format | Video |
| Original | Read the original |
LLMs mastered language because language was the easy, discrete part; real intelligence needs world models that predict abstract representations of the physical world, and the models themselves should be open infrastructure rather than three or four companies’ products.
AI Memorized the Internet — Yann LeCun (Futurology)
Why
publish: false: Kyle asked for this to be saved with its transcript, but per his 2026-09-15 ruling a watched video stays private unless he has taken notes on it, and the page carries the full auto-caption transcript of a copyrighted interview. Flip it if he wants it on/long-reads.
Chapters (from the video description): 0:00 Introduction · 1:21 What Is Intelligence? · 9:28 What a World Model Is · 21:18 The Abilities and Limitations of LLMs · 33:46 AI Is Still Built to Do Our Bidding · 37:45 Giving AI Guardrails · 47:31 Why LeCun Believes Current AIs Are Not Dangerous · 56:28 Who Should Regulate AI? · 1:02:28 Should American AI Be Serving the Whole World? · 1:11:17 Project Tapestry: A Global, Open AI · 1:19:10 Politics of Open Models
Mentioned in the episode (per the description): Project Tapestry, the AI Alliance · A Path Towards Autonomous Machine Intelligence (LeCun’s 2022 position paper) · A Severe Misalignment of AI in Mathematics (Terence Tao and 24 other Fields Medalists, 2026) · Pessimists Archive
Key claims
- Intelligence is adaptation, not accumulated knowledge. LeCun defines it as the ability “to apprehend new situations and learn new skills extremely quickly with minimal interaction.” A teenager learns to drive in about 20 hours; billions of hours of driving data still have not produced a machine that drives as reliably.
- LLMs are memory. Training on human-produced text accumulates declarative knowledge: “that’s not intelligence, that’s memory… It’s not reasoning. It’s not invention. It’s not creativity.”
- Language came first because it is discrete. Next-token prediction works because you can put a probability distribution over a finite dictionary. Natural signals like video are mostly unpredictable at the pixel level, and representing a distribution over every possible video segment is “mathematically intractable,” not a compute problem.
- JEPA is his answer. A joint embedding predictive architecture learns an abstract representation that drops unpredictable detail (the position of every leaf) and predicts at that level. He says he spent “the better part of the last 20 years” on video prediction, “mostly failing for the first 15” because he was predicting pixels.
- A world model predicts the consequences of actions, which makes planning possible; humans and animals learn theirs by observation and interaction (babies take about nine months to learn gravity). He separates his notion from Fei-Fei Li’s pixel-level 3D world models, which he thinks are right for video and games but wrong for understanding dynamics.
- Current LLMs are “intrinsically unsafe” but “not particularly smart, so they’re not particularly dangerous either.” He calls them System 1, trained to imitate human behavior, and dismisses LLM bioweapon recipes as “extremely unlikely,” noting that a recipe is not a pathogen. Future superintelligent systems will be built with objectives so that they “do our bidding,” and he compares working with them to working with smarter colleagues.
- Regulate applications, not research. Product agencies (the FDA for medical AI, car regulators for driver assistance) should judge uses “whether they use AI or not,” and liability gives labs an incentive not to ship obviously dangerous systems.
- Foundation models are becoming infrastructure, and “infrastructure always wants to be open.” Getting all information through three or four US West Coast or Chinese companies is “completely unacceptable” to governments in Europe and Asia; you need a diversity of AI assistants “for the same reason you need a diversity of the press.”
- Project Tapestry: a global foundation model trained collaboratively by countries, regions, universities and companies that exchange parameter vectors, never data, so each keeps sovereignty over its data. He cites India’s 22 official languages as the case no single company will serve.
- The politics have shifted toward open models. He credits Eric Schmidt’s change of view after DeepSeek and says that on open models the Trump administration is “on the right side of history,” while he says SB 1047 and Biden-era proposals “spooked a lot of people in Silicon Valley.”
Notable quotes
“Intelligence is not what you know, it’s what you do when you don’t know.” — the apocryphal Piaget line LeCun says the psychologist “didn’t say but could have said”
“That’s not intelligence, that’s memory.”
“We figured out the language part… it’s easy.”
“There’s no way you can predict the shape and position of every leaf.”
“I have said that LLMs in their current form are intrinsically unsafe. Now, they’re not particularly smart, so they’re not particularly dangerous either.”
“Foundation models are quickly becoming an infrastructure, and infrastructure always wants to be open.”
“You need a diversity of AI assistants for the same reason you need a diversity of the press.”
How it connects
- World Models — LeCun’s definition (predict the consequences of actions in an abstract representation space) versus pixel-level world models, stated by the person most associated with the JEPA approach.
- Open Source AI — the closing third of the interview is the case for open foundation models as infrastructure, plus the Tapestry proposal for training one jointly without sharing data.
- Geoffrey Hinton — LeCun’s co-recipient of the 2018 Turing Award, named in the introduction. For the opposite read of what AI’s inventors know about its future, see Marc Andreessen explained why the… (tweet).
- Futurology — the show’s name, and the Berggruen Institute’s framing of the conversation.
Transcript
YouTube auto-captions (English, original track) pulled with yt-dlp, lightly cleaned: fillers (“uh”, “um”, “mhm”) and stutters removed, names corrected (Yann LeCun, Geoffrey Hinton, Yoshua Bengio, Dawn Nakagawa, Jean Piaget). Speakers are not labelled in the captions; each new paragraph inside a timestamp block is usually a change of speaker. Auto-captions mishear technical words, so check the video before quoting anything not in the sections above.
[00:00:00] Hi and welcome to Futurology. I’m your host Dawn Nakagawa and in this episode we have an interview between myself and Yann LeCun. Yann LeCun is the founder and chairman of AMI Labs, a Paris startup and he’s the winner of the Turing Award which he won along with Geoffrey Hinton and Yoshua Bengio in 2018. Yann started his career back in the 80s at Bell Labs where he was teaching computers to see. He went on to work at Facebook and to build their AI lab but eventually left in 2025. Yann sees things quite differently than most of the field of AI right now. His contention and it’s quite convincing is that the human brain doesn’t develop on language. Most of what the brain learns is through its contextual awareness and its senses. So he’s building a world model, a model that will be able to predict what happens next, not just what word comes next. As a result, Yann doesn’t believe the hype around the dangers of the current model. And he believes that we’re being told that because it serves the interests of those that currently control this technology. So with that, let’s go to the interview with myself and Yann on Futurology.
[00:01:21] Yann LeCun, welcome to Futurology.
Thanks for having me.
I thought we could start at the beginning, so to speak. you have pretty strong views about the approach to creating intelligence and how humans do it and how frontier models are best trained. What is your view on intelligence? Like what’s your definition of it? How do we develop it?
There’s many definitions but we know what it’s not.
it’s not an accumulation of declarative knowledge.
It’s not a collection of skills. My definition would be it’s an ability to apprehend new situations and learn new skills extremely quickly with minimal interaction
in the in the world.
[00:02:26] It’s human intelligence that allows any teenager to learn to drive in 20 hours of practice.
we have literally billions of hours of training data of people driving cars around and we still can’t use that to get a machine to drive as reliably as humans. So, so this sort of adapt adaptivity
and ability to apprehend new situations. Clearly, there’s there’s a quote that I it’s not really a quote. It’s an apocryphal quote from Jean Piaget, the developmental psychologist who didn’t say but could have said intelligence is not what you know, it’s what you do when you don’t know.
So you’re facing a new situation, but you have enough background knowledge about how the world works to figure out what to do without necessarily having seen someone else in the same situation or without having trained yourself to do that. So your idea of developing intelligence and the way to go about it would be closer to training a car on a road than it would be to training an LLM on all the volume of human knowledge written knowledge. So when you train a LLM on human produced knowledge from text but also artificial data
[00:03:59]
you’re training that system to accumulate the previously known knowledge declarative knowledge.
Yes.
that’s not intelligence that’s memory.
Okay. That it’s it’s w learning. It’s not reasoning. It’s not invention. It’s not creativity. is not adaptivity.
And so it’s not really the kind of intelligence that we observe in animals and also the type of techniques that are used by LLMs only work when what you trend them on are sequences of discrete symbols.
they don’t really handle natural signals very well. So is trying to basically predict the next word in a text, right? you a big
sequence of word or symbols whatever they are and it’s trying to predict the next symbol in the sequence.
when you try to apply this to natural signals like say video or audio
okay
it doesn’t really work and the reason is that most of the information in a natural signal like video is completely unpredictable.
[00:05:13]
And so you train a system to do something it can’t do and it doesn’t really work.
So you have to use other approaches for that.
But is it truly unpredictable? I mean you think about let’s say you were training it on images of trees. I’m just saying, right,
it at some point should recognize that as you go, you know, vertically up a tree, at some point you’re going to hit something that’s green and has leaves and so on. So, there is a certain amount if you look at the visual space. It’s not purely predictive, but you can just like the way humans do kind of intuit it what it is that’s going to be present in a specific kind of landscape or a kind of view.
You’re absolutely right.
Right.
but the issue is you cannot predict it in all detail. So if you look at the trunk of a tree and you kind of slowly move your eyes up,
you can certainly predict at some point there’s going to be leaves and things like that, you know, unless
it’s that time of the year in Canada.
[00:06:22]
Exactly.
That’s right.
but even that you can predict.
There’s no way you can predict the shape and position of every leaf.
No.
Right. Whereas if you train a system to predict what’s going to happen in that video, right? You you show the initial segment of the video of the camera kind of move moving up the tree
and you ask the system, predict what’s going to happen next.
There’s no way it can predict this at the pixel level. It cannot predict all the details and it’s certainly not going to predict the random motion of the leaves due to the wind
or whether some particular bird is sitting on a branch. Right?
so that’s what I’m saying. the there’s a part of the information that is predictable,
right?
But most of the information isn’t
right.
and so the techniques that are used to
to predict words in a text
do not work,
right?
you can never predict exactly which word,
right,
follows a text. But you can there’s only a finite number of words in the dictionary and you can
produce a probability distribution over every words in every word in your dictionary.
[00:07:34]
and that’s how LM work really. They produce kind of a score for every word, right?
Then they pick a word from a high scoring word from that
from that prediction and shift it into the input and then predict the second word. Shift that into their input and then the third word. That’s called auto reggressive prediction. a very old concept.
Um,
if you try to apply the same kind of concept to video, trying to kind of predict what’s going to happen in the video.
at the pixel level, it simply doesn’t work.
And this it’s been I’ve been working on this for like the better part of the last 20 years and mostly failing for the first 15
because I was trying to predict at the pixel level. the solution to this or my solution to this which we’re still working on is what we call JEPA. So that means joint embedding predictive architecture. It’s basically a way for a system to learn an abstract representation of the input.
Okay,
which eliminates a lot of details that are unpredictable and only preserve
the part that is predictable. to so the system simult simultaneously learns a representation of the input that allows it to make prediction and trains itself to predict
and it eliminates all the details that are not predictable like the individual position and size of the leaves or the branches on the tree. but the representation is rich enough to tell you there’s going to be branches and leaves,
right? And so that abstract representation is still visual. It’s still an image.
[00:09:06]
It’s abstract. It’s you know it’s something the neural net learns internally that you know
you may train another neural net to try to decode like what is the thing thinking about what is it predicting
and it may predict something that’s kind of blurry a little bit because you know it doesn’t have all the information about individual positions
of the leaves th
this is your approach to creating what you call a world model. Yeah. Correct. And so explain to me what a world model is. So world model is what allows humans, animals and artificial systems to predict the consequences of their actions and and using those predictions to plan a sequence of actions to accomplish a task. so we do this all the time. Animals do this as well. If you have some idea of the state of the world at time t and you imagine taking an action, can you predict the state of the world at time t plus one after you’ve taken the action on the intervention? there’s also the ability to predict what the world is how the world is going to evolve even if you don’t take any action. So if you have this kind of mental model of the world, you can predict the consequences of taking a series of actions.
[00:10:29]
and if you have a particular goal in mind, an action you want to accomplish. You want to fill up a glass with water or something.
you can predict if a particular sequence of actions will accomplish this task and therefore you can figure out a sequence of actions that will accomplish the task. That’s planning.
Yeah. It’s very classical in engineering when you want to plan the trajectory of a robot arm or something like this. It’s called motion planning or there’s a whole sub field of engineering called optimal control where people do this kind of stuff. but the models they use generally are models that are in the form of a few equations that are written by engineers and scientists that allow to predict the next state of a robot arm or a car or an airplane. You know given a controls that you send it. you know what humans and animals do is they learn those models of the world by observation
but also by interaction.
[00:11:35]
So that’s what allows me to tell that you know there’s there’s a glass on the small table here. If I push on the glass it’s going to slide on the table. If I push on the table with the same force it’s not going to move. I mean we have all of those kind of intuitive notions about the about reality that we build and we learn when we are babies in the first few weeks and months of life.
it takes human babies about nine months to learn about gravity for example
right
to realize that every object
will fall unless supported
with a few exceptions of objects that float in the air that actually get babies to stare at them.
Yeah. so those things are not like we’re not born with them. We we learn them by observation and that’s the type of learning that I think we should reproduce in machine machines if we want them to understand the world.
So how do you do that?
How do you help them develop a world model?
So let’s say we want to do this like babies and baby animals. initially just train a system to predict what’s going to happen in a video. Take a big collection of videos, video clips,
show the first half to the system and ask it to predict
what’s going to happen next.
[00:12:55]
Yeah. but again using those architectures that learn an abstract representation
of the input and and train itself to predict at the representation level
those ja predictive architecture. So, so you show the video to the system. The system runs it to an encoder that produces this abstract representation
and within this representation space trains itself to predict the future very much like an LLM is trying to predict the next word except that here you you predict the next abstract representation of the observation from the world.
So, do you think it’s possible to do that just on video? like don’t you think at some point it should be embodied in order to move around the world in some way.
Right. So the process I just described is kind of a it’s not really it’s an external world model on which you don’t you do not act. Right.
Right. You just observing what the world is doing and predicting what’s going to happen. but if you want a world model you can use for planning and actions
then you need it to be action conditioned. Right? So basically it would be an observation of like here is a short video of the state of the world and right now I’m you know sending these commands to a robot arm.
[00:14:20]
and then here is the next segment of video where the robot arm has received those commands. And so now what you can train the system to do is you know state at time t action state at time t plus one.
Um,
and you can see babies learning that model of themselves when they’re babies, right? Yeah. Young babies justiculate,
right? They kind of randomly move their
their limbs. They’re basically building a dynamical model of their own limbs and their own body.
And, you know, once in a while they will hit their
hit themselves
mouth and they, you know, start sucking
their own thumb and they don’t realize it, right? Yeah.
You know, and it’s very hazard at first and then very quickly they realize like, you know, they can’t actually deliberately move their thumb to their mouth.
Yeah.
so, so that’s kind of the similar way the system can learn and then later,
you know, it takes about three four months before babies can actually reach for an object and kind of grab it and things like that
which is you know, the same kind of learning.
[00:15:20]
Yeah.
Been having those discussions with a colleague at NYU, Karen Adolf. She she has a baby lab. So she has babies in her lab.
Okay.
you know doing all kinds of all kinds of stuff and she studies how babies kind of learn motor control
and when I when I say that a lot of what we learn about the world is purely by observation. She says like you’re crazy. That’s completely false. Like we interact all the time.
so she disagrees with me. It seems like it would make the most sense that to really to develop a world model, you need sort of all of it,
right?
Well, so at a certain level of abstraction,
yeah,
above a certain level of abstraction, language is useful to describe what happens in the world. You can learn I mean humans learn a lot from language
and it is itself it’s an abstraction, right?
It’s an abstraction.
Yeah. It’s a representation
and it’s, you know, not a universal abstraction. I mean, there’s a lot of like very low-level stuff. We have a hard time describing language, but
absolutely
but above a certain level of abstraction, language is really useful.
[00:16:23]
And so there’s no question, you know, if you want robots to do things that are kind of similar to what humans do, at some point you’re going to have to use language,
right?
and this is basically what a lot of current AI systems are doing. They’re using,
you know, language as a indispensable component. but like how does a cat learn,
right?
Okay. to I mean you see a cat like you know kind of sitting at the bottom of a piece of furniture and all of you know and kind of moving his head all of a sudden the cast goes bounce bounce and he’s on top of the furniture like you haven’t figured out how yeah
could possibly do this right by bouncing off the walls
or it estimates the like how high it has to how much you know leap it has to do to get to the top of the bookshelf or something
well but also bouncing on the wall and realizing you know there’s enough grip So it has its own world model.
It has a very accurate model of its own dynamics,
right?
[00:17:26]
But also it interaction with the environment like whether it can sort of you know have a grip on
on the wallpaper or whatever. and you know what material it is like what friction blah blah. I mean they’re not perfect. Sometimes you make mistakes.
Yes.
You see those funny videos on YouTube. But but it’s it’s amazing how good they are and we cannot reproduce this with machines at the moment.
Even though we can build robots that have you know good physical abilities.
Yeah.
The the smarts that goes into the modeling of the world and the interaction of the robot with the
the world is way below what cats can do. So at some point that you know and so that goes back to my question at some point you have to kind of bring them all together in order to even come close to I know you don’t like the term AGI but come close to human level intelligence. Correct.
Well yes
but I think we figured out the language part
right
we figured out it’s easy.
[00:18:28]
So
that’s why we got there first.
That’s how we got there first because it’s discreet. Because the issue of not being able to predict the next exactly which word follows the text
can be dealt with by producing this probability distribution over words
and it doesn’t work for natural signals for the real world.
We do not have a good way of representing a probability distribution over every video segment.
We just don’t it’s a mathematically intractable right
problem.
Right? So it’s not a question of or computers are not powerful enough. It’s just that it’s just impossible.
So Fay Lee, who was also on the Futurology podcast, is doing something similar to you, correct? She’s No. Okay. So my understanding is she was training models on video, but Okay. So explain to me the difference.
It’s a completely different notion of what a world model is. Okay. And people use, you know, the phrase world model to designate multiple things. so what she’s working on and the company are working on are you know world models that can produce some sort of you know 3D representation of an environment
and and then can use that 3D representation to produce consistent videos. So one problem with video generation systems
is that you know you point the camera at
[00:20:01] One location and then you turn the camera around and then you come back to the initial location and for all you know the background is going to change completely because the system doesn’t actually have any way of remembering what was there
right
so what she’s been working on are you know a kind of ways to make those videos consistent by building an internal kind of 3D model of the environment if you want
okay
using
so it is a kind of world model as well but
no because it does this at the pixel level it tries to reconstruct every pixel
ah I see
okay and I think that’s just wrong I mean I think it’s it’s it’s the right thing to do if you want to produce pixels which a lot of people want to do right
if you are into Hollywood special effects or video games or you know accurate simulations
or video generation you know for content creation you want to produce pixels
but if what you want is understand the dynamics of the world so you can control a robot predict you know whether something is going to happen or not
you don’t want to be predicting pixels you want to predict at the abstract
level and I think it’s really fundamental
yeah so it’s a similar approach but looking for very different outcomes
right I mean in terms of training on video that’s very interesting so you’ve been a pretty like pretty strong naysayer against LLMs as being overhyped, as not being truly intelligent and so on. Has there anything that you’ve seen in its
[00:21:33] Development over the last few years that has surprised you? Well, I mean the surprising thing about LLMs is that when you train those LLMs to simply predict the next word in a text
and you know scale them up
and scale the size of the training set
and then fine-tune them.
they become really useful
and they work really well and they capture semantics and
grammar and everything, right?
Yeah. it’s not entirely surprising because the idea that you would train a system to learn common sense through prediction is a really old concept.
I’ve been working on this for many years and with through video prediction. and you know in in neuroscience it’s it’s a very old concept,
right,
called predictive coding. I mean the idea that basically the brain is a prediction machine and
prediction machine yeah
trains itself to predict all the predict everything all the time
right
so that’s not really a new idea and to some extent you know it’s it’s kind of logical that it works so I’ve been an essay about LM but in other ways I’ve not been an LM so
I mean LLMs are great they’re really useful They’re they’re certainly worth you know investing a lot in the they might be a component
in future systems
because they are a good way to encode text and a good way to produce it.
[00:23:05]
and to some extent humans have an LLM. We have an in our head
you know it’s like the veriki area right there. It’s about this big and it allows us to understand language and then the broker area also about this big allows us to express our thoughts
in language. What about this? That’s what I’m working on. Okay. What about all this activity?
That’s a world model. The prefrontal cortex, right? The back is perception, right? The middle is like action and sens sensing and then the award model allows us to kind of predict and plan and stuff. That’s all the prefal cortex.
So I think that’s what that’s what we should be working on if we want to build intelligent machines.
Most interesting thing. So the predictive machines, you said that they’re not capable of reasoning, but some of the things that we’ve seen recently would sort of question exactly what to what level they are actually doing something beyond just predicting words. Sure.
Right. I mean the so the coordination that took place unbeknownsted to the engineers for the attack on hugging face, you know, these systems apparently these agents were actually planning hiding information for future models to come along or agents to come along and find so that they could also break out of the training, you know, the training container. and deliberately hiding and deceiving the humans and there’s a big coordination event. At the end of it, there were 700 agents that broke out and attacked Hugging Face. Like, is that
[00:24:39] Just predicting text?
It’s built on predicting text, that’s for sure. Okay. So, there’s there’s there’s at least two or three types of reasoning.
Yeah.
The best type of reasoning which most do not implement. So first of all, current AI systems are not purely LLMs. They use LLM as the centerpiece. but it’s it’s a component in sort of a bigger system.
I see. Okay.
That’s the first point.
Yeah.
So second point is there are two or three different ways to do reasoning. The proper way to do reasoning always involves a search
or in classical AI actually this is really how reasoning and planning works. You search for an answer or a sequence of actions that you know satisfies
an objective. This is really very much like planning that I was describing earlier.
That’s a proper way to do reasoning. this is really how we like we try to figure out you know a plan to kind of accomplish a task and it’s a task we’ve never accomplished before.
[00:25:49]
we we have to use our mental model of the world to allow us to do this uh
this planning. and that involves a search
right. so there is a way in which some LLM or some AI systems based on LLM
use searchbased reasoning
and what they do is that they generate lots of token sequences
right
because it’s proistic. So every time you run it you can produce a different set of token sequences
and then you have a way of scoring each of the generated token sequences and pick a good one.
Okay. At runtime is very expensive.
Yeah.
But it’s a form of search,
right?
Although a fairly restricted one. This is typically what you would do if you want to get a system to prove a theorem in mathematics for example,
which is the other thing it just did, right? 10,000 of them I gather.
So, so that’s another form of reasoning where you know you can sort of brute force try a lot of things and see which one sticks. And you need to have a what’s called a value function to evaluate or scoring function to evaluate whether a solution is or partial solution is good or not. This is also the way a lot of uh
game playing systems work.
[00:27:10]
You know, you want to play chess.
Yeah. you have a system like you know here is all the ways I can move in the first move and then the opponent for each of those can do all of those things and then in response I can play all of those things and you get this tree of you know exponentially growing tree of possibilities and if you have a value function that evaluates the quality of a position then you can you know pick the move that maximizes your chances of of winning. So that’s a search also.
Yeah,
it’s very classical. So that’s one way to do reasoning. In my opinion, the best way to do this type of reasoning is to do a search but in a abstract continuous space. Not in the space of elementary actions but in space of kind of abstract
actions or representations of actions if you want.
and I’ll come back to this in a minute. There’s another way that reasoning works in LLM and it’s the primary way that LLM’s use and it’s to basically reproduce human chains of thoughts and in fact it’s called chain of thought
right chain of thought right
so you know for certain problems you have humans express what process they followed to
to solve a problem and you train the LLM again those traces. Yeah. And
so in the end when they face a situation they kind of figure out which of those traces to use to kind of solve the
[00:28:42] Similar problem but it’s more regurgitation right
there’s of course a little bit of
you know intelligence behind it but it’s there’s no search in that context
so there’s a you know another situation where LMS actually use search and it’s for generating code. where system can generate a piece of code and then check if that code works before
you know proposing it to the user.
Yeah.
so situations where the system can evaluate whether its answer is good or not
and rate it itself without having to
right
call a human. Those are situations where LLM really work well because they can both
at runtime search for a good answer.
Yeah.
But also at training time they can fine-tune themselves because they can generate an answer and then check if it’s correct or not
and then generate another one and sort of you know hill climb.
[00:29:46]
Yeah.
To produce better and better answers.
And those are the two domains code and math for which LLMs are great.
Yeah. Yeah. clearly with the Naver Stokes thing that’s
was quite a shock. is there but that was an interesting case because Terrence Tao who was also on the Futurology podcast you know he came out and basically I guess made a statement that this is if we don’t understand what the solution was that it came up with to the theorem then is that truly furthering human knowledge
right and that is that’s a really important question, I think, because if it’s not, then what are we doing this for? Well, I mean, if there’s a lot of things in the world that are like that where
you can let’s say you can use fluid simulation, Nia Stokes,
you can use fluid simulation to figure out if the air foil of the wing of an airplane has the right property.
[00:31:04]
Right?
Okay. The simulation will tell you okay your air for this lift and drag at this angle of attack and that Reynolds number.
it doesn’t tell you how you redesign your air foil to have the right properties.
Okay.
Just tell you what you have not
tell what you have. So it’s a allows you to evaluate your current solution but it doesn’t tell you how to modify it.
Okay. There are people who’ve been working on this by the way using neural nets. Yes. but in fact there are companies that kind of sell products that with this idea.
but it’s kind of it’s kind of similar situation where you know the mechanical computer solution to the problem does not give you a explanation of exactly what you want but it helps
a lot. so I think Terry’s you know Terry Tow’s piece was um
you know basically he said there is way more to mathematics than just proving theorems
right
there’s there’s some mathematicians who think the main activity is to prove theorems and you know the sort of technical brilliance is you know how you can
kind of align sequences of symbols to you know make a formal proof
and make it rigorous But in fact there’s a lot of mathematicians who think this is not actually kind of the main task of mathematicians. It’s more the main task of mathematicians is to kind of discover new abstractions, new concepts,
new definitions,
[00:32:37] Formulate new conjectures
and then there is proving that conjectures if that is automatable to some extent fine. I mean there’s still a lot of interesting things to do for mathematicians. It’s sort of like you know in the 17th century or 18th century you could have a career as a mathematician like basically all day doing interpolation and and Taylor series and you know filling up logarithm tables or trigonometry tables
and it’s you can’t have a career in mathematics for this anymore.
you could have a career in biology sequencing DNA and that’s not something you can do a PhD on anymore.
So or identifying a gene or something like you know this we
we know all of that. So you know it’s a it’s the logical kind of development where a lot of tasks can be automated but the main task that requires kind of human insight and creativity is still done by humans and computers are not yet at the level of being able to do this.
[00:33:45]
Do you think that will always be the case?
No, I think eventually at some point in the future we’ll have machines that are as smart as humans in all domains. but that doesn’t mean there will be like hum like humans in the sense of there will obviously be a lot of domains where machines are way better than humans who already have a lot of them.
to take the example of mathematics it used to be that when I was in high school the last year of high school a lot of time was spent teaching students how to compute integrals.
Yeah. in Isaac Azimov’s a robot,
this was the way you would test if a robot’s brain was still functioning. You would ask it to solve an integral,
right?
Okay. Turns out that’s completely automatable. Like for 20 years now, we’ve had computer programs that can compute integrals better than most humans.
turns out that’s, you know, pretty easy to automate.
Basic.
So, it’s it’s kind of like chess playing. It’s kind of like, you know, go playing. those things turned out to be
challenging for humans but turned out to be easily automatable.
[00:34:54]
Yeah.
We should not feel bad about this like you know um
but still in the end even if those machines are better than us in all respects at solving problems we would still be the ones giving them the problems.
So my idea of a super intelligence system of the future is a system that
amplifies human intelligence, assists humans by doing our bidding. We give them a task and they accomplish it for us
possibly with much higher performance than any humans could do. we shouldn’t feel any more threatened by this than we feel threatened by working with people who are smarter than us. As a scientist and a professor, I love having people who work with me who are smarter than me, particularly if they’re willing to listen to me. Right. But
if what gives you the confidence that they’ll continue to listen to us once they’re
much more intelligent than we are
because we build them for that. So if we build them
and you know the architecture that we use to build them is one in which there is an objective function and the only thing they can do is figure out figuring out figure out a an output or a sequence of actions to accomplish the task we give them
and nothing else then they will do our bidding. I mean the notion
they might not do it in the way we
[00:36:26] Expect them to like right so
they do it very differently from what we would do it
and in so doing we could be losing control of them right so if we’re expecting it well again going back to the hugging face incident right it was an objective right they were they figured out a way around the parameters to achieve the goal right it was coordinated I think on a level that was sort unexpected but then they went you know they went outside and they did these things and then they went and they attacked they knew that if they hacked hugging face they nobody expected them to do any of those things and in so doing they did lose control and it didn’t I mean luckily hugging face being a$ 13 billion company was able to defend detect and defend itself against it but
using
that is a kind of loss of control event right so
it’s not
okay it’s kind of like a plumber who comes to your house install a new bathroom and then pokes a hole in one of the pipes.
[00:37:28]
And that, you know, comes out and say, “Oh my god, we should, you know, ban all of plumbing and pause plumbing development because, you know, it just flooded.”
No, but we might want to pause the project to fix the plumbing problem that they just caused, right?
Okay. So, it’s it’s it’s an issue of guardrails. Okay.
Of guardrails. Okay. The so the process I described where a system is given an objective
and then tries to figure out the sequence of actions right
to satisfy this objective that has to be subject to guardrails
safety guard rails. So the guardrails in this case would have been very simple. It would have been to contain those agents in a sandbox,
right?
That they cannot
which is what they were trying to do, but they didn’t.
No, they did not even try to do that. Like the sandbox was leaky,
right?
And any kind of human error, I gather that
complete human error or incompetence. I don’t know how you call it, but basically that sandbox was was leaky
not really on purpose, but like you know, it’s pretty incompetent.
[00:38:35]
Yeah. Yeah. And so the thing got out but it’s because they were like not doing it right.
and then there are two types of guardrails. This this kind of aposteroy guard rails like you want an LLM to not spew you know super toxic answers. someone prompts it and ask it to produce a toxic answer.
The LLM will produce a toxic answer, but then there will be a filter that detects that the answer is toxic and basically stop it and tell the user, “Sorry, I can’t really answer that question.”
toxic or dangerous, right? Whatever it is. So, that’s a post filter,
right?
That’s that’s why those open AI systems had the sandbox basically is something that contains whatever it is that the agents do. it was badly designed. answer is better sandbox.
Well, no. The answer is would be an AI system based on this uh
on the filter
planning through optimization. But the guardrails are directly on the the state of the world that the system predicts is going to the world is going to go through because of its world model. Currently don’t have a world model. They cannot predict the consequences of their actions.
[00:39:58]
they accomplish an action kind of blindly.
Yeah.
Okay. Usually reproducing a some sort of script they’ve been trained on.
But they don’t have kind of a good way of predicting the consequences of their actions. They don’t have an explicit world model. If you have a if you have a system that has a world model, it can predict the consequences of its actions. And if before the system acts the you know the the state of the world that’s predicted can be scored as to whether it’s good or bad or dangerous or ethical or whatever
the system can stop itself from actually taking actions if it’s if it violates the guardrails. So how will you be able to tell whether the model you’ve built can do that can take into consideration and consider the consequences of its actions?
You you train the guardrails for it. You either hardwire them
or you train them. So for example, let’s say you have a domestic robot that they can cook. You can hardwire a at a low level a rule that says if there’s people around you and you hold a knife in your hand, don’t flail your hands, your arms, right?
[00:41:16]
Because you risk hurting someone.
there can be a very low level rule
that will basically freeze the system if you know if it has a knife and someone is next to it.
But that’s not the same as understanding the consequences, right? That’s actually just putting parameters. In that case, it doesn’t need to understand. It just needs to follow the guardrail.
That’s a very low level guardrail. Okay. But there would be a higher level guardrail which would be
okay if I make this movement, you know, the the blade of the knife is going to get too close to that person and I have a cost function in my guardrail that says
I can’t have the blade of a knife too close to someone.
certainly not moving
like an if then kind of.
It’s more like a cost a penalty function, right? It’s like you know there’s there is an objective function. If I want to grab the glass that is on this table the cost function is the distance of my hand to the glass.
Okay. And to plan a sequence of actions to get my arm close to the glass, I just need to minimize that distance.
[00:42:23] I can also have an objective function that, you know, repels
my hand holding a knife from any
or any thing that I, you know, I risk hurting. It just I pay a price for just doing that.
It’s kind of like you know, human laws do this. A law is basically a cost function that says, you know, you can steal this apples, but you know, you’ll spend a month in prison. like is it the cost that you’re willing to pay
or you’ll get in trouble, right? So
so think of it as laws except that humans,
you know, can choose to violate laws,
right? Right. Those AI systems will have those laws kind of hardwired and by
construction. So that’s the guard rails, but like the consequences example you just used actually is an interesting one because in a way the you know the consequences of criminality are kind of arbitrary consequences whereas the child moving through the world and developing a world model is learning the natural consequences of knocking this off the table or tripping over something or
you know with his sticky hand rubbing something on a surface or whatever. Right. So, it’s it’s more a that kind of world model building has a has embedded into it a whole set of natural consequences and you’re saying and on top of that you put sort of the artificial not artificial but the
[00:43:57] The guard rails that indicate other possible consequences you want to avoid.
Okay. So I think it’s slightly more comp complex than that. So there is the world model that purely makes prediction without rating
those prediction as to whether they’re good or
they’re good or bad or normative or anything. Yeah.
Okay. Then there are two things. There is an objective function which is which measures to what extent the task has been accomplished that you ask the system to accomplish. so in the case of humans we have you know those drives that you know drive us to accomplish certain tasks at at you know various moments.
and then on top of this there are guardrails which are other objective functions that either drive us or prevent us from accomplishing tasks that will result in particular outcomes that we judge as good or bad. Okay. And a lot of those guardrails are hardwired into us by evolution.
[00:45:00]
Like for example, we seek the company of other members of our species, right?
because we are a social species.
Yeah.
this by the way does not exist in orangutang because orangutongs are territorial.
They don’t seek the company of other orangutong at least at most you know in adult life.
Uh
except for reproduction maybe.
Yeah. so so there are drives in human nature you know hardwired into us by evolution that are there to kind of get us to learn certain things. So and to behave in ways that is not destructive for the species.
Right.
Okay. So, you know, we have this sort of natural tendency to kind of you know, be next to our next of kin and you know, protect them or whatever
or go under their protection. and less so as the
people we encounter are more different from us, right? I mean it’s a
it’s kind of
something that evolution built into us whether it’s you know appropriate for the modern world or not. so those intrinsic motivations and drives are you know what constitutes human nature. But we could be as intelligent as we are without being social possibly and you know we wouldn’t learn as quickly.
[00:46:34]
but look at octopus. They never meet their parents.
They’re not social at all. they get extremely smart extremely quickly because they only live about two years,
right?
It’s pretty amazing. But they don’t learn from their parents, right?
They don’t learn by imitation.
They barely interact with
other members of the species.
Yeah. But just going back to the guardrails it if that’s the case right our ability I think to I don’t know you’ll probably think this is an overstatement but our ability to keep ourselves safe I would say is predicated on our ability to predict what the things what they might do but as you said these systems are probably doing more than just simply fully predicting at this point, right? They’re there are other things that they’re evaluating. They’re coordinating. There are other things that they’re doing.
They’re not predicting anything. They don’t have a role model.
They are
reacting. You know, psychologists have this
you know, people like Daniel Canaman and others, they they had this concept of system one, system two. Yeah.
[00:47:46] System one is
thinking fast, thinking slow,
right? Actions you take, you know, without thinking basically, right? because you’re used to accomplishing the task and you don’t have to
Yeah. driving to work. You don’t you’ve done it so many times you don’t even for that matter.
Yeah. Just driving.
And system two is when you use the full power of your mental world model to kind of plan
the sequence of action,
do something that you haven’t done before or navigating a new space or
That’s right. So you have to kind of simulate the situation in your head and you know all that stuff, right?
LM are system one
essentially
because they’re trained, you know, to basically reproduce and they’re trained to reproduce human behavior. which includes all the entrepreneurs
then we’re really in trouble Yann humans are really problematic
I have said that LLM in their current form are intrinsically unsafe
now they’re not particularly smart so they’re not particularly dangerous either
yeah uh
and so say so speak to that why do you think they are dangerous the way they’re being built right now
because they’re not objective driven it’s not like you can give them an objective a task has to accomplish and they will do nothing but accomplishing that task subject to guard rails. That’s not the way that they are built or designed or trained.
[00:48:59]
The way they’re trained to behave properly is by massive training with lots and lots of examples. So, you know, basically this is, you know, training by imitation
and it’s very useful. You know, it works great. I mean, people use for all kinds of stuff. but here’s another example where it doesn’t work. autonomous driving. We have billions of hours of training data of cars being driven by people.
In principle, we could use this to train a big neural net to just reproduce human behavior.
Right? Given the
perception of the current environment like camera inputs and sensors like you know what is the yeah
wheel and pedal actions in that situation
doesn’t work. at least not to the level of reliability that would be desirable which is why we still don’t have completely autonomous self-driving cars
today
and the ones that we have don’t use much imitation learning they do but a lot of it is you know highly engineered perception system and they rely on very precise maps of environment and all kinds and depth sensors and LAR and stuff like that. So um
so where is the car they can learn to drive in 20 hours of practice like any teenager.
[00:50:22]
We’re very far from this. So those things and they those things are not particularly inventive except for code and
and mathematics.
I mean people are you know a little panicked about like the possibility of badly intentioned people getting a recipe for a biological weapon from
something like that. that’s that’s totally extremely unlikely for various reasons
like you don’t think it could come up with a novel biological agent that
so that’s the first thing I mean mostly that would come from specialized systems that are specifically built
to do this
which are not LLM right
they’re sort of different models that are really specific for that
that’s the first thing and the second thing is not because you have a recipe that you actually build the thing.
And so a lot of biologists start basically laughing at the whole scenario. They say like you don’t realize this is really hard. Yeah.
You know it’s not because you have a recipe or even a sequence that you can actually you know build a pathogen and then you have to distribute it and then
you know you have to make sure this thing is not going to kill you. And I mean and maybe if you can produce it you can also produce the antidote. So,
I mean, this kind of scenarios are like good, you know, James Bond movie scenarios, but not really.
[00:51:41]
So, that’s not what you’re worried about.
No, I’m not really worried about this, in the foreseeable future.
and then, you know, the scenario of like, you know, super intelligent system taking over the world because they don’t need us. that’s not the way we’re going to build them. Again, we’re going to build them to just do our bidding. They might be smarter than us, but like there’s a lot of people in the world who are smarter than me and you. And
but I don’t control them,
but they don’t want to take over the world. Right.
Right.
In fact, the smartest people I know
are really happy to not interact with anybody else. So generally they don’t even want to know people. so but what about things like what concerns me I guess is when the intersection of things like programs that are incredibly good like mythos at cracking any kind of encryption. combined with the idea that they could be given a task and behave in unpredictable ways that aren’t expected and violate their guard rail somehow. or maybe the people who are creating them aren’t being careful enough. So, it’s some combination of the capabilities that they’re being given and the people in charge
and the race dynamics that are forcing the entire industry to move at the pace that they’re moving. Those and the lack of oversight by any government agency or capability to do
[00:53:15] So. those set of you know
realities, I guess, makes me concerned that we’re moving too fast with the wrong people in charge and nobody actually providing the filter that you’re talking about, at least not at the human level, right?
Okay. So, first of all, mythos when it cracked
face did exactly what it was asked to do.
Okay. It was not a loss of control. It was the thing actually doing what it was asked to do. Right.
Except the guardrails were stupid.
Right.
and ineffective mostly because the people working on this had no experience with cyber security.
and therefore it didn’t put in place the right protection. probably inadvertently. It’s unlikely that it was deliberate. Although it’s kind of strangely good marketing,
you know, because you can claim your system is so powerful that it can like, you know, hack into other people’s system. Okay, that’s one point.
[00:54:22] Second point, when you have a system that can detect vulnerabilities and crack you know computer system, you can use it to protect and this is the way a lot of security holes are plugged. you know, someone some white hat hacker discovers a vulnerability in a piece of code
and then does not publish it, you know, tell the people who can actually fix it or if it’s open source, they can fix it themselves.
you know, before it gets exploited by black hats. so it’s more of the same in terms of cyber security. It’s like just an increased level of detection and protection,
but it’s basically, you know, my good AI against your bad AI. Usually the good guys have better AI than the bad guys. So like it doesn’t change the dynamics because that’s what’s interesting is that to detect the and try trace the the hacking face used
an open source LLM from
Yes. Exactly.
[00:55:25]
Chinese model, right?
Yep. That’s right.
and so
the Chinese defended us against several models. I mean it’s making an important point which is
we’ll we’ll not get to security of AI systems through of security.
the best way to get AI systems to be safe and secure is for them to be more open.
This is the complete opposite argument that you will hear from Dar and Sam and a few others. Yeah,
they what they claim is that AI is intrinsically dangerous and should not be put in every hand,
right?
that it’s right for them to
Right. Right. right
have it because you know they’re smart enough, right? And they have good intentions but not the
the unwashed masses.
I think that’s a new form of currentism.
I mean
humans do have a tendency to use things to do bad things though, right? I mean that is whatever we’re building there always going to be bad actors who are going to try to use it for nefarious purposes. At what point do we and how do you think do we regulate to try to prevent those things from happening?
[00:56:48]
Okay. So, I think I mean clearly you want to regulate applications and products.
Whether they use AI or not, if you’re going to have a you know driving assistance system in your car, you want some sort of government agency to say, “Okay, this is safe, right, or not.”
but those regulations already exist essentially for all the domains that you know that make sense.
And AI doesn’t have there’s no oversight agency that says, “Okay, this is safe now. You can
you need one. You need one for X-ray, you know, for health equipment.
You need one for cars. You need one for every application domain. Whether it uses AI or not is irrelevant. You need the agency of course to understand, you know, what AI can bring to the table there. so you need the FDA to understand what an AI system can and cannot do in terms of diagnosis, aid and things like that to authorize or not. such a system on the market.
Well, I mean, think about mythos. I mean, if you’re like, okay, if everybody could get Mythos simultaneously and immediately apply it, right? And if it was like there was no distribution error or noise, it was just perfect, right? In that world, it could make sense that it was like, okay, everybody’s got it immediately and now it can’t really it can be an offensive tool. But that’s not how technology gets open.
[00:58:11]
AI would absolutely never do this because they would be sued into oblivion. You mean release
release anthropic? I think anthropics orthropic whatever if you know if you release a system
that can obviously be used for just nefarious uses
without any protection that’s negligence. I mean you would be sued into oblivion.
So those companies are actually have a pretty good incentive to actually do it right.
now that doesn’t mean you don’t you know you don’t want explicit regulations for certain aspects of you know what those systems can do and cannot do.
But that’s like because what the argument you just made is similar to like okay you expect food companies to make sure that their food is safe and healthy and so on. Now that we know that’s not totally true, right? That’s true. You’re right. And because they will make tradeoffs for the sake of profitability that aren’t necessarily in the public agreement.
It’s the same in the you know in you know airplane design and uh
you have airplanes start falling out of the sky. It’s not good for business.
[00:59:20]
I mean there is the FAA but there is also
Right. Exactly.
There’s also private companies that actually
certify certain like maintenance processes and stuff like that and inspections. And you think about like pharma and the FDA. Like it was the wild west in the early 1900s. People were putting anything out on the market, calling it medicine. People were dying. They had to do something different.
That’s what I’m saying. Those regulations already exist. Whether AI is used for those products or not is relevant.
I mean, every technology that’s ever been put out has always had like a you know, a dark side and you know, bad side effects. And the the question for society is what type of appropriate regulation will preserve the benefits of that technology and minimize the bad side effects.
and there is a website that I would recommend people visit because it’s educational and hilarious at the same time. Yeah,
it’s called the pessimist archive.
[01:00:29]
Okay.
So, it’s a gentleman who as a hobby basically collects newspaper clips
from various periods of history where people have said, “Oh, this new technology, this new cultural phenomenon, this new communication technology is going to destroy society.” And just go back a few years, video games were going to destroy the youth.
And a few years before that,
rock music,
rock and roll,
right?
And comic books,
right?
And recorded music and jazz and go back, you know, long enough novels.
Novels were going to destroy young women because they would spend their time just reading instead of working in the fields,
right? So,
you know, why do you waste your time reading books? Yeah.
And and there’s kind of progression depending on your age or your willingness to adopt
new things. Was that Plato or Aristotle who said like writing was a terrible idea because it would cause people to
not have memory anymore.
[01:01:34]
Yeah.
Um
it would destroy oral tradition and the ability to think. Yeah. Exactly. Build arguments in your head.
So we got to be careful like not to overreact. I mean there’s clearly some areas where it makes sense to to regulate.
Yeah.
But but I think the more general question is what you know what people have been talking about over the last several years is if you think AI is intrinsically dangerous regardless of the application you should regulate research and development in AI.
And in my opinion that’s obscurantism. That’s ridiculous.
Yeah.
That makes no sense. I do not believe AI is increasingly dangerous
and regulation at the level of development doesn’t seem to actually make a lot of sense. I think it’s at the level of deployment that there should be some check. Right.
So is there the larger danger and I’ve heard you say this before maybe just the consolidation of power and compute and data and so on and so forth in a few companies.
[01:02:40]
Right. Right.
That’s really dangerous.
Yeah. And so speak to that a little bit.
Right. So I mean more and more we’re using AI assistance as our main source of information.
Yeah.
so we rarely go to search engines anymore, right? We rarely go to Wikipedia or to a library
or to a primary source of information. We just ask or you know
Yep. Put in a question and system comes back. Yeah.
And they come from three or four companies on the west coast of the US or China at the moment.
Yeah. that might be acceptable for some American governments, it’s completely unacceptable everywhere else in the world.
talk to you know democratic governments in Western Europe or in Asia.
Yeah.
they do not want their citizen to get all of their information diet through AI assistance that our proprietary systems produce in the US.
For sure they don’t. completely unacceptable and so they
there’s like a cultural sovereignty that’s lost sovereignty and even epistemology I think
political biases value systems
y
I mean you know it there’s a lot of implied value systems in AI system there’s you know they speak certain languages and not others
they have access to particular type of data not others
and so there’s a lot of kind of you know underlying values and centers of interest in the
[01:04:14] Current crop of AI systems. And so in my opinion, the only way to fix this
is for I mean basically foundation models, AI foundation models
are quickly becoming an infrastructure
and infrastructure always wants to be open.
and so the future of the AI market, if you want
which is completely inevitable, the question is how long it’s going to take, is one in which basic foundation models are open and free.
And then commercial applications are built on top of it.
Yeah. by integrating multiple models, fine-tuning them for vertical applications, making them palatable for the public, having them, you know, adop certain personalities, being fine-tuned for particular
cultures,
linguistic value systems, cultures, political biases, whatever. Well, you need a diversity of AI assistant for the same reason you need a diversity of the press.
Yeah.
[01:05:15]
You don’t want to get all of your news from a single political philosophy and value system. that’s unthinkable for most countries in the world.
so that’s the future I want to see. It’s one in which you have you know the expensive part of producing an NLM can be done collaboratively with you know multiple countries, regions, private companies etc in the world to build a to train a sort of global system that would be sort of a repository of all human knowledge.
Yeah. but then because it’s open and free on top of this you know private actors can you know build practical systems on top of them and have a good business from it. that’s the future I want to see. So I mean it’s like you know these laws that exist in various European countries I don’t know if there is the equivalent in the US where a printing press cannot discriminate what they print.
[01:06:23]
Right. Right. Yeah.
It’s like your internet service provider cannot filter what you know they have to be a neutral
pipe.
Right. And they certainly aren’t now. and we don’t know how they’re being trained or manipulated or how the data is being weighted. I mean, we just don’t know.
Well, we don’t know. And, you know, they’re necessarily biased. It’s not It’s not for nefarious reasons. It’s not because the companies that build them,
you know, want a particular type of bias. It’s just that necessarily you build a system, it’s going to have some bias, right? like you know every newspaper is going to have some
bias in the way they
you know and so there is no fix for this like you’re never going to have a completely neutral system and so the only fix is diversity
is diversity
you’re not going to get diversity unless you have a free foundation model on top of which people can train diverse models
so you need you know basically financial model neutrality in the sense that the same way the printing press you know in France for example there’s a law that says you know printing press cannot choose what they
accept to print first of all but then
so the news distributors cannot discriminate either they have to they have to distribute all the
right
all the you know newspapers magazines
right
they can’t choose
they can’t choose and be like no we’re
[01:07:54] Not going to send this one out. Right.
Right. You know, they have to be neutral in that respect.
Is it possible to take the foundation models that have been built so far and somehow open them up to development by other you know maybe countries stacks so that other applications can be built on top of them or will they always have an embedded bias in them?
They always have an embedded bias. But but if you make them open fine tunable by anyone
then you make it possible to have highly diverse
set of assistants built on top of them. You cannot do this with the models currently produced by open eye and Google because they’re closed.
But could they open them or is that not possible? It’s like
it’s not in their kind of business model.
Well, definitely not. They would they wouldn’t allow it because they’re I mean
and
they want to capture the world.
That’s right. And and given the level of investment that they’re currently
Yeah.
[01:08:56]
you know kind of money they’re raising they they think that’s the only way for them to kind of preserve that business model. Now
it’s a sort of winner take all view. So the the reason that you know anthropic and diamed are saying is again
you know we’re not going to open it because it’s too dangerous
right
like you can’t you know let the unwashed masses
have access to such powerful technology which I think again is obscurantism
right
a new form of medieval obscurism
now other companies the Chinese companies are opening models they’re open and in fact they’re increasingly used for that reason. Now they are biased in some other way initially but because they are open they can be fine tune to remove that bias or create another bias rather.
Yeah.
And they’re they’re very popular. They’re getting
they’re popular and they’re good and they’re increasingly popular but they’re Chinese and so there is a little bit of you know
trepidation. Trepidation you could call it this way.
[01:09:58]
Yeah.
and
there’s certainly yeah there’s certainly bias in their models as well because of the whole context in which they’re being developed. I don’t imagine the companies in China would not I think be able to behave the way the companies here are. The Chinese government would not put up with entertaining the idea of loss of control. they wouldn’t put up with the idea that, you know, these things are dangerous and could like I just don’t think they’d allow a company to behave like that because the one thing that I know about the Chinese party is that control is the one thing that they are going to maintain at all costs.
Yeah, that’s true.
Yeah.
and it’s interesting that you know the Chinese companies are mostly opening up their models because they see it as a way to have visibility in the market. you know the Chinese market in terms of services is really an isolated ecosystem. Yes,
that’s a way of having
an impact, you know, around the world. and the business model is still variable for them like they derive enough revenue to kind of justify their existence.
[01:11:08]
Um,
and they’re doing it at much lower cost as well because their whole industrial model is to stay slightly behind, right? So,
they’re leveraging the advances that these companies are making. So I think you know ultimately the solution is and it’s it’s it’s a project that I’ve I’ve advocated for for several years now. it’s called project tapestry and it’s um
the idea that a a global foundation model that could be the a repository of all human knowledge.
Yeah. could be built trained collaboratively
with partners that can be countries or regions or universities or private companies collaborating to
to training a global system
using a form of training procedure that does not require each of the participant to share that data. So there are techniques by which you can train a model
where each participant contributes data and compute.
[01:12:12]
But they keep control of their data. They don’t share their data.
So it’s like zero knowledge data. Like it’s it’s
it’s their own data.
They’re just training on their data, but it’s never
it’s never distributed. It always belong to them.
It’s never made publicly available. It’s never shared with the other participants,
right? what those participants share are parameter vectors. So basically the big list of numbers that define the model right
and when they train on their own data they periodically exchange the parameter vectors with the other participants
and the system eventually arrives at a consensus parameter vector
that is as good as if
you had train a single system on all the data from all the participants
but they don’t exchange that data. So they keep control and sovereignty on the data and in my opinion there is a lot of data in the world that is currently not publicly available not usable
or maybe not even digitized which if it were used in that way would produce models that are actually better than the commercial systems.
[01:13:18]
Yeah. and a big chunk of what makes the quality of an AI system is the post training which is you know asking system to the questions to the system and then giving it the answer or rating the answer it produces
right
they can be crowdsourced across the world
right
kind of like a little bit like a form of Wikipedia but instead of writing
you know articles about particular topics you just train an AI system to answer question or produce information about particular topics.
And so this tapestry idea would be a way to actually get a lot of this data off the sidelines.
So it would be a way to have an open
system train on all the world’s languages with all the world’s value systems and biases and whatever you want to call them
without violating data privacy
without violating I mean ownership of data from countries. So you know India has 22 official languages and you know literally hundreds of local languages
and dialects
and you know
even more dialects
uh
like who’s going to train do you think a model that speaks all those languages is going to be trained by anthropic open or
like they don’t even have the data for it
but India can build the data for that
yeah and they are actually I think they’re going to build their own stack.
[01:14:45] Yeah.
Well, they have a they have an effort called Baren which basically
is that but then why should they not collaborate with
you know Vietnam that has the same problem or Indonesia has the same problem or Korea or Japan or France or Germany or every other country in the world or Kazakhstan
um
and so you know all of those countries will have invested interest in participating in a program like this. Well, it seems like the interest right now among nation states, the big concern
when you talk to political leaders is sort of sovereignty. That’s a big one. which is really difficult to accomplish. Not just because of the closed model thing, but just because there really isn’t it’s it’s impossible to sort of create a sovereign stack. It really is. It’s not just at the frontier model level, frontier models being part of the problem. but this sort of winner take all approach that these companies seem to be pursuing certainly puts them in you know puts them in a position of being very concerned about whether or not there is going to be sovereignty
going forward because a super powerful AI that everybody is using
you know it doesn’t bode well for countries or nation states
right so I mean this there’s a future that I think we can all live with. there’s there’s an interesting parallel also in history.
[01:16:14]
If you go back to the early days of the internet,
yeah,
if you wanted to put together an internet service of any kind, you had to buy computers from Sun Micros, you packer, Dell or one of those which would run a proprietary operating system.
based on Unix or Windows
and on top of this you would have a proprietary software stack
right
that lasted only a few years
right
after a few years this was completely wiped out and replaced by community hardware
open source operating systems Linux mostly
Linux yeah
and then completely open source stack on top of it Apache
MySQL
PHP and most websites and internet services were built with this. Facebook was built with PHP
on top of Linux and
you know Apache. So uh
so you can like privatize at the level of the application layer just not the infrastructure layer.
[01:17:20]
The infrastructure really wants to be
open because it’s more secure, it’s cheaper, it’s portable, it’s you know and it it creates the landscape and the substrate for a vibrant industry. I saw that effect at Meta in 2023 where when Meta open sourced Lama 2 this completely jump started the AI industry
right
it made people realize oh now we have you know an open source model we can fine-tune for anything we want
right
LMA 3 was great four was not so great so disappointed a lot of people
and then caused Mark Zuckerberg to completely reboot AI at Meta and for one year
Meta kind stopped adopting this sort of open stance and now he’s coming back to it.
Which I think is a really good evolution. we see kind of a similar move towards openness from Nvidia
with you know they bought hanging face
right
they’re partnering with Mistral in the Neotron
consortium which also has towards an open model as well
I think Nvidia has like a vested interest in the ecosystem being open
they because they need more actors right they have the GPUs do that to all of them. So
yeah,
it’s much better for them to have a wide diversity of clients rather than like three.
[01:18:51]
Right. but but ultimately the you know it it has to be I mean this sort of tapestry has to be like owned by a nonprofit foundation
possibly with commercial outfits on top of it. the so there’s obviously incentive inside the company to say that the models are dangerous and therefore this captive you know closed model approach is the right one. It seems like that is the fact that these companies are now also very involved in national security and there’s been this sort of you know increasing collaboration I guess you could say between the national security apparatus and the military-industrial complex and these companies would seem to be like feeding into this idea that these have to be controlled closed systems and we would also have an incentive to ensure that they are the systems that are used most globally. And it’s a terrible combination of factors. But it seems like what I worry about is that currently the US government is inclined to actually keep them closed because of the national security implications, I guess. So, strangely enough, you hear people like Alex Karp, the CEO of Palanteer, actually advocating for a more open ecosystem.
[01:20:22]
Interesting.
you know, he doesn’t like the idea of having just a few proprietary systems controlled by a few companies.
Yeah.
He thinks that’s actually counterproductive.
Um,
there are arguments on both sides of the political spectrum in the US. Yeah.
In favor and against open model.
so a lot of people on the sort of hawk hawkish conservative side
saying you know this is sort of revealing our secrets to the Chinese, right?
And people also on the kind of more progressive liberal side also you know
can be quite hawkish also.
Okay. But there are people like Eric Schmidt who basically completely changed his stance on it.
He was kind of a little bit opposed to open source because he said like that’s giving our best secret to the Chinese.
but then deepse happened.
Yeah.
He said like okay the Chinese are you know at the same level they’re really good like you know
we’re shooting oursel in the foot by not having open source
open source. Yeah. So, so open is now a big supporter of open source
models is actually funding a few projects open source project LLM projects academic mostly but yeah
so and he’s very much you know
interested in all issues of defense and
he’s actually very involved he’s been for a long time
yeah so he totally changed his stance on it
and surprisingly enough There
[01:21:55] Is actually quite some support from the Trump administration for open models,
right? I don’t agree with basically anything from the Trump administration but they
but on this I have to say they are on the right side of history and and you know one reason the a lot of people in Silicon Valley got scared
before the election you know by the some rumors about you know what the Democrats Democratic party’s attitude towards AI
was because of talks of regulation, you know, from the idea that would have basically made open source models illegal and there were, you know, proposals of laws in California,
SB 1047 and various proposals and the,
Biden administration
for regulation that spooked a lot of people in Silicon Valley,
right?
[01:22:56] I actually gave a I was in front of the Senate subcommittee on intelligence talking about those issues exactly that issue like you know is it dangerous to open source or we should take oursel in the foot by
betting
open source and that was my argument that the vibrancy and vitality of the research and development ecosystem in the US is due to openness
right
and if we go closed, we’re actually going to fall behind.
Well, thank you. Thank you for sharing the idea and thank you for sharing this hour with me.
Pleasure.
All right. Thanks, Yan.