Skip to content
kwharrison13.comEssays, books and notes by Kyle Harrison

AI Memorized the Internet — Yann LeCun (Futurology)

From kwharrison13.com · Long read
AI Memorized the Internet — Yann LeCun (Futurology)
AuthorYann LeCun, interviewed by Dawn Nakagawa
PublishedNovember 17, 2026
FormatVideo
OriginalRead the original

LLMs mastered language because language was the easy, discrete part; real intelligence needs world models that predict abstract representations of the physical world, and the models themselves should be open infrastructure rather than three or four companies’ products.

AI Memorized the Internet — Yann LeCun (Futurology)

Why publish: false: Kyle asked for this to be saved with its transcript, but per his 2026-09-15 ruling a watched video stays private unless he has taken notes on it, and the page carries the full auto-caption transcript of a copyrighted interview. Flip it if he wants it on /long-reads.

Chapters (from the video description): 0:00 Introduction · 1:21 What Is Intelligence? · 9:28 What a World Model Is · 21:18 The Abilities and Limitations of LLMs · 33:46 AI Is Still Built to Do Our Bidding · 37:45 Giving AI Guardrails · 47:31 Why LeCun Believes Current AIs Are Not Dangerous · 56:28 Who Should Regulate AI? · 1:02:28 Should American AI Be Serving the Whole World? · 1:11:17 Project Tapestry: A Global, Open AI · 1:19:10 Politics of Open Models

Mentioned in the episode (per the description): Project Tapestry, the AI Alliance · A Path Towards Autonomous Machine Intelligence (LeCun’s 2022 position paper) · A Severe Misalignment of AI in Mathematics (Terence Tao and 24 other Fields Medalists, 2026) · Pessimists Archive

Key claims

  • Intelligence is adaptation, not accumulated knowledge. LeCun defines it as the ability “to apprehend new situations and learn new skills extremely quickly with minimal interaction.” A teenager learns to drive in about 20 hours; billions of hours of driving data still have not produced a machine that drives as reliably.
  • LLMs are memory. Training on human-produced text accumulates declarative knowledge: “that’s not intelligence, that’s memory… It’s not reasoning. It’s not invention. It’s not creativity.”
  • Language came first because it is discrete. Next-token prediction works because you can put a probability distribution over a finite dictionary. Natural signals like video are mostly unpredictable at the pixel level, and representing a distribution over every possible video segment is “mathematically intractable,” not a compute problem.
  • JEPA is his answer. A joint embedding predictive architecture learns an abstract representation that drops unpredictable detail (the position of every leaf) and predicts at that level. He says he spent “the better part of the last 20 years” on video prediction, “mostly failing for the first 15” because he was predicting pixels.
  • A world model predicts the consequences of actions, which makes planning possible; humans and animals learn theirs by observation and interaction (babies take about nine months to learn gravity). He separates his notion from Fei-Fei Li’s pixel-level 3D world models, which he thinks are right for video and games but wrong for understanding dynamics.
  • Current LLMs are “intrinsically unsafe” but “not particularly smart, so they’re not particularly dangerous either.” He calls them System 1, trained to imitate human behavior, and dismisses LLM bioweapon recipes as “extremely unlikely,” noting that a recipe is not a pathogen. Future superintelligent systems will be built with objectives so that they “do our bidding,” and he compares working with them to working with smarter colleagues.
  • Regulate applications, not research. Product agencies (the FDA for medical AI, car regulators for driver assistance) should judge uses “whether they use AI or not,” and liability gives labs an incentive not to ship obviously dangerous systems.
  • Foundation models are becoming infrastructure, and “infrastructure always wants to be open.” Getting all information through three or four US West Coast or Chinese companies is “completely unacceptable” to governments in Europe and Asia; you need a diversity of AI assistants “for the same reason you need a diversity of the press.”
  • Project Tapestry: a global foundation model trained collaboratively by countries, regions, universities and companies that exchange parameter vectors, never data, so each keeps sovereignty over its data. He cites India’s 22 official languages as the case no single company will serve.
  • The politics have shifted toward open models. He credits Eric Schmidt’s change of view after DeepSeek and says that on open models the Trump administration is “on the right side of history,” while he says SB 1047 and Biden-era proposals “spooked a lot of people in Silicon Valley.”

Notable quotes

“Intelligence is not what you know, it’s what you do when you don’t know.” — the apocryphal Piaget line LeCun says the psychologist “didn’t say but could have said”

“That’s not intelligence, that’s memory.”

“We figured out the language part… it’s easy.”

“There’s no way you can predict the shape and position of every leaf.”

“I have said that LLMs in their current form are intrinsically unsafe. Now, they’re not particularly smart, so they’re not particularly dangerous either.”

“Foundation models are quickly becoming an infrastructure, and infrastructure always wants to be open.”

“You need a diversity of AI assistants for the same reason you need a diversity of the press.”

How it connects

  • World Models — LeCun’s definition (predict the consequences of actions in an abstract representation space) versus pixel-level world models, stated by the person most associated with the JEPA approach.
  • Open Source AI — the closing third of the interview is the case for open foundation models as infrastructure, plus the Tapestry proposal for training one jointly without sharing data.
  • Geoffrey Hinton — LeCun’s co-recipient of the 2018 Turing Award, named in the introduction. For the opposite read of what AI’s inventors know about its future, see Marc Andreessen explained why the… (tweet).
  • Futurology — the show’s name, and the Berggruen Institute’s framing of the conversation.

Transcript

YouTube auto-captions (English, original track) pulled with yt-dlp, lightly cleaned: fillers (“uh”, “um”, “mhm”) and stutters removed, names corrected (Yann LeCun, Geoffrey Hinton, Yoshua Bengio, Dawn Nakagawa, Jean Piaget). Speakers are not labelled in the captions; each new paragraph inside a timestamp block is usually a change of speaker. Auto-captions mishear technical words, so check the video before quoting anything not in the sections above.

[00:00:00] Hi and welcome to Futurology. I’m your host Dawn Nakagawa and in this episode we have an interview between myself and Yann LeCun. Yann LeCun is the founder and chairman of AMI Labs, a Paris startup and he’s the winner of the Turing Award which he won along with Geoffrey Hinton and Yoshua Bengio in 2018. Yann started his career back in the 80s at Bell Labs where he was teaching computers to see. He went on to work at Facebook and to build their AI lab but eventually left in 2025. Yann sees things quite differently than most of the field of AI right now. His contention and it’s quite convincing is that the human brain doesn’t develop on language. Most of what the brain learns is through its contextual awareness and its senses. So he’s building a world model, a model that will be able to predict what happens next, not just what word comes next. As a result, Yann doesn’t believe the hype around the dangers of the current model. And he believes that we’re being told that because it serves the interests of those that currently control this technology. So with that, let’s go to the interview with myself and Yann on Futurology.

[00:01:21] Yann LeCun, welcome to Futurology.

Thanks for having me.

I thought we could start at the beginning, so to speak. you have pretty strong views about the approach to creating intelligence and how humans do it and how frontier models are best trained. What is your view on intelligence? Like what’s your definition of it? How do we develop it?

There’s many definitions but we know what it’s not.

it’s not an accumulation of declarative knowledge.

It’s not a collection of skills. My definition would be it’s an ability to apprehend new situations and learn new skills extremely quickly with minimal interaction

in the in the world.

[00:02:26] It’s human intelligence that allows any teenager to learn to drive in 20 hours of practice.

we have literally billions of hours of training data of people driving cars around and we still can’t use that to get a machine to drive as reliably as humans. So, so this sort of adapt adaptivity

and ability to apprehend new situations. Clearly, there’s there’s a quote that I it’s not really a quote. It’s an apocryphal quote from Jean Piaget, the developmental psychologist who didn’t say but could have said intelligence is not what you know, it’s what you do when you don’t know.

So you’re facing a new situation, but you have enough background knowledge about how the world works to figure out what to do without necessarily having seen someone else in the same situation or without having trained yourself to do that. So your idea of developing intelligence and the way to go about it would be closer to training a car on a road than it would be to training an LLM on all the volume of human knowledge written knowledge. So when you train a LLM on human produced knowledge from text but also artificial data

[00:03:59]

you’re training that system to accumulate the previously known knowledge declarative knowledge.

Yes.

that’s not intelligence that’s memory.

Okay. That it’s it’s w learning. It’s not reasoning. It’s not invention. It’s not creativity. is not adaptivity.

And so it’s not really the kind of intelligence that we observe in animals and also the type of techniques that are used by LLMs only work when what you trend them on are sequences of discrete symbols.

they don’t really handle natural signals very well. So is trying to basically predict the next word in a text, right? you a big

sequence of word or symbols whatever they are and it’s trying to predict the next symbol in the sequence.

when you try to apply this to natural signals like say video or audio

okay

it doesn’t really work and the reason is that most of the information in a natural signal like video is completely unpredictable.

[00:05:13]

And so you train a system to do something it can’t do and it doesn’t really work.

So you have to use other approaches for that.

But is it truly unpredictable? I mean you think about let’s say you were training it on images of trees. I’m just saying, right,

it at some point should recognize that as you go, you know, vertically up a tree, at some point you’re going to hit something that’s green and has leaves and so on. So, there is a certain amount if you look at the visual space. It’s not purely predictive, but you can just like the way humans do kind of intuit it what it is that’s going to be present in a specific kind of landscape or a kind of view.

You’re absolutely right.

Right.

but the issue is you cannot predict it in all detail. So if you look at the trunk of a tree and you kind of slowly move your eyes up,

you can certainly predict at some point there’s going to be leaves and things like that, you know, unless

it’s that time of the year in Canada.

[00:06:22]

Exactly.

That’s right.

but even that you can predict.

There’s no way you can predict the shape and position of every leaf.

No.

Right. Whereas if you train a system to predict what’s going to happen in that video, right? You you show the initial segment of the video of the camera kind of move moving up the tree

and you ask the system, predict what’s going to happen next.

There’s no way it can predict this at the pixel level. It cannot predict all the details and it’s certainly not going to predict the random motion of the leaves due to the wind

or whether some particular bird is sitting on a branch. Right?

so that’s what I’m saying. the there’s a part of the information that is predictable,

right?

But most of the information isn’t

right.

and so the techniques that are used to

to predict words in a text

do not work,

right?

you can never predict exactly which word,

right,

follows a text. But you can there’s only a finite number of words in the dictionary and you can

produce a probability distribution over every words in every word in your dictionary.

[00:07:34]

and that’s how LM work really. They produce kind of a score for every word, right?

Then they pick a word from a high scoring word from that

from that prediction and shift it into the input and then predict the second word. Shift that into their input and then the third word. That’s called auto reggressive prediction. a very old concept.

Um,

if you try to apply the same kind of concept to video, trying to kind of predict what’s going to happen in the video.

at the pixel level, it simply doesn’t work.

And this it’s been I’ve been working on this for like the better part of the last 20 years and mostly failing for the first 15

because I was trying to predict at the pixel level. the solution to this or my solution to this which we’re still working on is what we call JEPA. So that means joint embedding predictive architecture. It’s basically a way for a system to learn an abstract representation of the input.

Okay,

which eliminates a lot of details that are unpredictable and only preserve

the part that is predictable. to so the system simult simultaneously learns a representation of the input that allows it to make prediction and trains itself to predict

and it eliminates all the details that are not predictable like the individual position and size of the leaves or the branches on the tree. but the representation is rich enough to tell you there’s going to be branches and leaves,

right? And so that abstract representation is still visual. It’s still an image.

[00:09:06]

It’s abstract. It’s you know it’s something the neural net learns internally that you know

you may train another neural net to try to decode like what is the thing thinking about what is it predicting

and it may predict something that’s kind of blurry a little bit because you know it doesn’t have all the information about individual positions

of the leaves th

this is your approach to creating what you call a world model. Yeah. Correct. And so explain to me what a world model is. So world model is what allows humans, animals and artificial systems to predict the consequences of their actions and and using those predictions to plan a sequence of actions to accomplish a task. so we do this all the time. Animals do this as well. If you have some idea of the state of the world at time t and you imagine taking an action, can you predict the state of the world at time t plus one after you’ve taken the action on the intervention? there’s also the ability to predict what the world is how the world is going to evolve even if you don’t take any action. So if you have this kind of mental model of the world, you can predict the consequences of taking a series of actions.

[00:10:29]

and if you have a particular goal in mind, an action you want to accomplish. You want to fill up a glass with water or something.

you can predict if a particular sequence of actions will accomplish this task and therefore you can figure out a sequence of actions that will accomplish the task. That’s planning.

Yeah. It’s very classical in engineering when you want to plan the trajectory of a robot arm or something like this. It’s called motion planning or there’s a whole sub field of engineering called optimal control where people do this kind of stuff. but the models they use generally are models that are in the form of a few equations that are written by engineers and scientists that allow to predict the next state of a robot arm or a car or an airplane. You know given a controls that you send it. you know what humans and animals do is they learn those models of the world by observation

but also by interaction.

[00:11:35]

So that’s what allows me to tell that you know there’s there’s a glass on the small table here. If I push on the glass it’s going to slide on the table. If I push on the table with the same force it’s not going to move. I mean we have all of those kind of intuitive notions about the about reality that we build and we learn when we are babies in the first few weeks and months of life.

it takes human babies about nine months to learn about gravity for example

right

to realize that every object

will fall unless supported

with a few exceptions of objects that float in the air that actually get babies to stare at them.

Yeah. so those things are not like we’re not born with them. We we learn them by observation and that’s the type of learning that I think we should reproduce in machine machines if we want them to understand the world.

So how do you do that?

How do you help them develop a world model?

So let’s say we want to do this like babies and baby animals. initially just train a system to predict what’s going to happen in a video. Take a big collection of videos, video clips,

show the first half to the system and ask it to predict

what’s going to happen next.

[00:12:55]

Yeah. but again using those architectures that learn an abstract representation

of the input and and train itself to predict at the representation level

those ja predictive architecture. So, so you show the video to the system. The system runs it to an encoder that produces this abstract representation

and within this representation space trains itself to predict the future very much like an LLM is trying to predict the next word except that here you you predict the next abstract representation of the observation from the world.

So, do you think it’s possible to do that just on video? like don’t you think at some point it should be embodied in order to move around the world in some way.

Right. So the process I just described is kind of a it’s not really it’s an external world model on which you don’t you do not act. Right.

Right. You just observing what the world is doing and predicting what’s going to happen. but if you want a world model you can use for planning and actions

then you need it to be action conditioned. Right? So basically it would be an observation of like here is a short video of the state of the world and right now I’m you know sending these commands to a robot arm.

[00:14:20]

and then here is the next segment of video where the robot arm has received those commands. And so now what you can train the system to do is you know state at time t action state at time t plus one.

Um,

and you can see babies learning that model of themselves when they’re babies, right? Yeah. Young babies justiculate,

right? They kind of randomly move their

their limbs. They’re basically building a dynamical model of their own limbs and their own body.

And, you know, once in a while they will hit their

hit themselves

mouth and they, you know, start sucking

their own thumb and they don’t realize it, right? Yeah.

You know, and it’s very hazard at first and then very quickly they realize like, you know, they can’t actually deliberately move their thumb to their mouth.

Yeah.

so, so that’s kind of the similar way the system can learn and then later,

you know, it takes about three four months before babies can actually reach for an object and kind of grab it and things like that

which is you know, the same kind of learning.

[00:15:20]

Yeah.

Been having those discussions with a colleague at NYU, Karen Adolf. She she has a baby lab. So she has babies in her lab.

Okay.

you know doing all kinds of all kinds of stuff and she studies how babies kind of learn motor control

and when I when I say that a lot of what we learn about the world is purely by observation. She says like you’re crazy. That’s completely false. Like we interact all the time.

so she disagrees with me. It seems like it would make the most sense that to really to develop a world model, you need sort of all of it,

right?

Well, so at a certain level of abstraction,

yeah,

above a certain level of abstraction, language is useful to describe what happens in the world. You can learn I mean humans learn a lot from language

and it is itself it’s an abstraction, right?

It’s an abstraction.

Yeah. It’s a representation

and it’s, you know, not a universal abstraction. I mean, there’s a lot of like very low-level stuff. We have a hard time describing language, but

absolutely

but above a certain level of abstraction, language is really useful.

[00:16:23]

And so there’s no question, you know, if you want robots to do things that are kind of similar to what humans do, at some point you’re going to have to use language,

right?

and this is basically what a lot of current AI systems are doing. They’re using,

you know, language as a indispensable component. but like how does a cat learn,

right?

Okay. to I mean you see a cat like you know kind of sitting at the bottom of a piece of furniture and all of you know and kind of moving his head all of a sudden the cast goes bounce bounce and he’s on top of the furniture like you haven’t figured out how yeah

could possibly do this right by bouncing off the walls

or it estimates the like how high it has to how much you know leap it has to do to get to the top of the bookshelf or something

well but also bouncing on the wall and realizing you know there’s enough grip So it has its own world model.

It has a very accurate model of its own dynamics,

right?

[00:17:26]

But also it interaction with the environment like whether it can sort of you know have a grip on

on the wallpaper or whatever. and you know what material it is like what friction blah blah. I mean they’re not perfect. Sometimes you make mistakes.

Yes.

You see those funny videos on YouTube. But but it’s it’s amazing how good they are and we cannot reproduce this with machines at the moment.

Even though we can build robots that have you know good physical abilities.

Yeah.

The the smarts that goes into the modeling of the world and the interaction of the robot with the

the world is way below what cats can do. So at some point that you know and so that goes back to my question at some point you have to kind of bring them all together in order to even come close to I know you don’t like the term AGI but come close to human level intelligence. Correct.

Well yes

but I think we figured out the language part

right

we figured out it’s easy.

[00:18:28]

So

that’s why we got there first.

That’s how we got there first because it’s discreet. Because the issue of not being able to predict the next exactly which word follows the text

can be dealt with by producing this probability distribution over words

and it doesn’t work for natural signals for the real world.

We do not have a good way of representing a probability distribution over every video segment.

We just don’t it’s a mathematically intractable right

problem.

Right? So it’s not a question of or computers are not powerful enough. It’s just that it’s just impossible.

So Fay Lee, who was also on the Futurology podcast, is doing something similar to you, correct? She’s No. Okay. So my understanding is she was training models on video, but Okay. So explain to me the difference.

It’s a completely different notion of what a world model is. Okay. And people use, you know, the phrase world model to designate multiple things. so what she’s working on and the company are working on are you know world models that can produce some sort of you know 3D representation of an environment

and and then can use that 3D representation to produce consistent videos. So one problem with video generation systems

is that you know you point the camera at

[00:20:01] One location and then you turn the camera around and then you come back to the initial location and for all you know the background is going to change completely because the system doesn’t actually have any way of remembering what was there

right

so what she’s been working on are you know a kind of ways to make those videos consistent by building an internal kind of 3D model of the environment if you want

okay

using

so it is a kind of world model as well but

no because it does this at the pixel level it tries to reconstruct every pixel

ah I see

okay and I think that’s just wrong I mean I think it’s it’s it’s the right thing to do if you want to produce pixels which a lot of people want to do right

if you are into Hollywood special effects or video games or you know accurate simulations

or video generation you know for content creation you want to produce pixels

but if what you want is understand the dynamics of the world so you can control a robot predict you know whether something is going to happen or not

you don’t want to be predicting pixels you want to predict at the abstract

level and I think it’s really fundamental

yeah so it’s a similar approach but looking for very different outcomes

right I mean in terms of training on video that’s very interesting so you’ve been a pretty like pretty strong naysayer against LLMs as being overhyped, as not being truly intelligent and so on. Has there anything that you’ve seen in its

[00:21:33] Development over the last few years that has surprised you? Well, I mean the surprising thing about LLMs is that when you train those LLMs to simply predict the next word in a text

and you know scale them up

and scale the size of the training set

and then fine-tune them.

they become really useful

and they work really well and they capture semantics and

grammar and everything, right?

Yeah. it’s not entirely surprising because the idea that you would train a system to learn common sense through prediction is a really old concept.

I’ve been working on this for many years and with through video prediction. and you know in in neuroscience it’s it’s a very old concept,

right,

called predictive coding. I mean the idea that basically the brain is a prediction machine and

prediction machine yeah

trains itself to predict all the predict everything all the time

right

so that’s not really a new idea and to some extent you know it’s it’s kind of logical that it works so I’ve been an essay about LM but in other ways I’ve not been an LM so

I mean LLMs are great they’re really useful They’re they’re certainly worth you know investing a lot in the they might be a component

in future systems

because they are a good way to encode text and a good way to produce it.

[00:23:05]

and to some extent humans have an LLM. We have an in our head

you know it’s like the veriki area right there. It’s about this big and it allows us to understand language and then the broker area also about this big allows us to express our thoughts

in language. What about this? That’s what I’m working on. Okay. What about all this activity?

That’s a world model. The prefrontal cortex, right? The back is perception, right? The middle is like action and sens sensing and then the award model allows us to kind of predict and plan and stuff. That’s all the prefal cortex.

So I think that’s what that’s what we should be working on if we want to build intelligent machines.

Most interesting thing. So the predictive machines, you said that they’re not capable of reasoning, but some of the things that we’ve seen recently would sort of question exactly what to what level they are actually doing something beyond just predicting words. Sure.

Right. I mean the so the coordination that took place unbeknownsted to the engineers for the attack on hugging face, you know, these systems apparently these agents were actually planning hiding information for future models to come along or agents to come along and find so that they could also break out of the training, you know, the training container. and deliberately hiding and deceiving the humans and there’s a big coordination event. At the end of it, there were 700 agents that broke out and attacked Hugging Face. Like, is that

[00:24:39] Just predicting text?

It’s built on predicting text, that’s for sure. Okay. So, there’s there’s there’s at least two or three types of reasoning.

Yeah.

The best type of reasoning which most do not implement. So first of all, current AI systems are not purely LLMs. They use LLM as the centerpiece. but it’s it’s a component in sort of a bigger system.

I see. Okay.

That’s the first point.

Yeah.

So second point is there are two or three different ways to do reasoning. The proper way to do reasoning always involves a search

or in classical AI actually this is really how reasoning and planning works. You search for an answer or a sequence of actions that you know satisfies

an objective. This is really very much like planning that I was describing earlier.

That’s a proper way to do reasoning. this is really how we like we try to figure out you know a plan to kind of accomplish a task and it’s a task we’ve never accomplished before.

[00:25:49]

we we have to use our mental model of the world to allow us to do this uh

this planning. and that involves a search

right. so there is a way in which some LLM or some AI systems based on LLM

use searchbased reasoning

and what they do is that they generate lots of token sequences

right

because it’s proistic. So every time you run it you can produce a different set of token sequences

and then you have a way of scoring each of the generated token sequences and pick a good one.

Okay. At runtime is very expensive.

Yeah.

But it’s a form of search,

right?

Although a fairly restricted one. This is typically what you would do if you want to get a system to prove a theorem in mathematics for example,

which is the other thing it just did, right? 10,000 of them I gather.

So, so that’s another form of reasoning where you know you can sort of brute force try a lot of things and see which one sticks. And you need to have a what’s called a value function to evaluate or scoring function to evaluate whether a solution is or partial solution is good or not. This is also the way a lot of uh

game playing systems work.

[00:27:10]

You know, you want to play chess.

Yeah. you have a system like you know here is all the ways I can move in the first move and then the opponent for each of those can do all of those things and then in response I can play all of those things and you get this tree of you know exponentially growing tree of possibilities and if you have a value function that evaluates the quality of a position then you can you know pick the move that maximizes your chances of of winning. So that’s a search also.

Yeah,

it’s very classical. So that’s one way to do reasoning. In my opinion, the best way to do this type of reasoning is to do a search but in a abstract continuous space. Not in the space of elementary actions but in space of kind of abstract

actions or representations of actions if you want.

and I’ll come back to this in a minute. There’s another way that reasoning works in LLM and it’s the primary way that LLM’s use and it’s to basically reproduce human chains of thoughts and in fact it’s called chain of thought

right chain of thought right

so you know for certain problems you have humans express what process they followed to

to solve a problem and you train the LLM again those traces. Yeah. And

so in the end when they face a situation they kind of figure out which of those traces to use to kind of solve the

[00:28:42] Similar problem but it’s more regurgitation right

there’s of course a little bit of

you know intelligence behind it but it’s there’s no search in that context

so there’s a you know another situation where LMS actually use search and it’s for generating code. where system can generate a piece of code and then check if that code works before

you know proposing it to the user.

Yeah.

so situations where the system can evaluate whether its answer is good or not

and rate it itself without having to

right

call a human. Those are situations where LLM really work well because they can both

at runtime search for a good answer.

Yeah.

But also at training time they can fine-tune themselves because they can generate an answer and then check if it’s correct or not

and then generate another one and sort of you know hill climb.

[00:29:46]

Yeah.

To produce better and better answers.

And those are the two domains code and math for which LLMs are great.

Yeah. Yeah. clearly with the Naver Stokes thing that’s

was quite a shock. is there but that was an interesting case because Terrence Tao who was also on the Futurology podcast you know he came out and basically I guess made a statement that this is if we don’t understand what the solution was that it came up with to the theorem then is that truly furthering human knowledge

right and that is that’s a really important question, I think, because if it’s not, then what are we doing this for? Well, I mean, if there’s a lot of things in the world that are like that where

you can let’s say you can use fluid simulation, Nia Stokes,

you can use fluid simulation to figure out if the air foil of the wing of an airplane has the right property.

[00:31:04]

Right?

Okay. The simulation will tell you okay your air for this lift and drag at this angle of attack and that Reynolds number.

it doesn’t tell you how you redesign your air foil to have the right properties.

Okay.

Just tell you what you have not

tell what you have. So it’s a allows you to evaluate your current solution but it doesn’t tell you how to modify it.

Okay. There are people who’ve been working on this by the way using neural nets. Yes. but in fact there are companies that kind of sell products that with this idea.

but it’s kind of it’s kind of similar situation where you know the mechanical computer solution to the problem does not give you a explanation of exactly what you want but it helps

a lot. so I think Terry’s you know Terry Tow’s piece was um

you know basically he said there is way more to mathematics than just proving theorems

right

there’s there’s some mathematicians who think the main activity is to prove theorems and you know the sort of technical brilliance is you know how you can

kind of align sequences of symbols to you know make a formal proof

and make it rigorous But in fact there’s a lot of mathematicians who think this is not actually kind of the main task of mathematicians. It’s more the main task of mathematicians is to kind of discover new abstractions, new concepts,

new definitions,

[00:32:37] Formulate new conjectures

and then there is proving that conjectures if that is automatable to some extent fine. I mean there’s still a lot of interesting things to do for mathematicians. It’s sort of like you know in the 17th century or 18th century you could have a career as a mathematician like basically all day doing interpolation and and Taylor series and you know filling up logarithm tables or trigonometry tables

and it’s you can’t have a career in mathematics for this anymore.

you could have a career in biology sequencing DNA and that’s not something you can do a PhD on anymore.

So or identifying a gene or something like you know this we

we know all of that. So you know it’s a it’s the logical kind of development where a lot of tasks can be automated but the main task that requires kind of human insight and creativity is still done by humans and computers are not yet at the level of being able to do this.

[00:33:45]

Do you think that will always be the case?

No, I think eventually at some point in the future we’ll have machines that are as smart as humans in all domains. but that doesn’t mean there will be like hum like humans in the sense of there will obviously be a lot of domains where machines are way better than humans who already have a lot of them.

to take the example of mathematics it used to be that when I was in high school the last year of high school a lot of time was spent teaching students how to compute integrals.

Yeah. in Isaac Azimov’s a robot,

this was the way you would test if a robot’s brain was still functioning. You would ask it to solve an integral,

right?

Okay. Turns out that’s completely automatable. Like for 20 years now, we’ve had computer programs that can compute integrals better than most humans.

turns out that’s, you know, pretty easy to automate.

Basic.

So, it’s it’s kind of like chess playing. It’s kind of like, you know, go playing. those things turned out to be

challenging for humans but turned out to be easily automatable.

[00:34:54]

Yeah.

We should not feel bad about this like you know um

but still in the end even if those machines are better than us in all respects at solving problems we would still be the ones giving them the problems.

So my idea of a super intelligence system of the future is a system that

amplifies human intelligence, assists humans by doing our bidding. We give them a task and they accomplish it for us

possibly with much higher performance than any humans could do. we shouldn’t feel any more threatened by this than we feel threatened by working with people who are smarter than us. As a scientist and a professor, I love having people who work with me who are smarter than me, particularly if they’re willing to listen to me. Right. But

if what gives you the confidence that they’ll continue to listen to us once they’re

much more intelligent than we are

because we build them for that. So if we build them

and you know the architecture that we use to build them is one in which there is an objective function and the only thing they can do is figure out figuring out figure out a an output or a sequence of actions to accomplish the task we give them

and nothing else then they will do our bidding. I mean the notion

they might not do it in the way we

[00:36:26] Expect them to like right so

they do it very differently from what we would do it

and in so doing we could be losing control of them right so if we’re expecting it well again going back to the hugging face incident right it was an objective right they were they figured out a way around the parameters to achieve the goal right it was coordinated I think on a level that was sort unexpected but then they went you know they went outside and they did these things and then they went and they attacked they knew that if they hacked hugging face they nobody expected them to do any of those things and in so doing they did lose control and it didn’t I mean luckily hugging face being a$ 13 billion company was able to defend detect and defend itself against it but

using

that is a kind of loss of control event right so

it’s not

okay it’s kind of like a plumber who comes to your house install a new bathroom and then pokes a hole in one of the pipes.

[00:37:28]

And that, you know, comes out and say, “Oh my god, we should, you know, ban all of plumbing and pause plumbing development because, you know, it just flooded.”

No, but we might want to pause the project to fix the plumbing problem that they just caused, right?

Okay. So, it’s it’s it’s an issue of guardrails. Okay.

Of guardrails. Okay. The so the process I described where a system is given an objective

and then tries to figure out the sequence of actions right

to satisfy this objective that has to be subject to guardrails

safety guard rails. So the guardrails in this case would have been very simple. It would have been to contain those agents in a sandbox,

right?

That they cannot

which is what they were trying to do, but they didn’t.

No, they did not even try to do that. Like the sandbox was leaky,

right?

And any kind of human error, I gather that

complete human error or incompetence. I don’t know how you call it, but basically that sandbox was was leaky

not really on purpose, but like you know, it’s pretty incompetent.

[00:38:35]

Yeah. Yeah. And so the thing got out but it’s because they were like not doing it right.

and then there are two types of guardrails. This this kind of aposteroy guard rails like you want an LLM to not spew you know super toxic answers. someone prompts it and ask it to produce a toxic answer.

The LLM will produce a toxic answer, but then there will be a filter that detects that the answer is toxic and basically stop it and tell the user, “Sorry, I can’t really answer that question.”

toxic or dangerous, right? Whatever it is. So, that’s a post filter,

right?

That’s that’s why those open AI systems had the sandbox basically is something that contains whatever it is that the agents do. it was badly designed. answer is better sandbox.

Well, no. The answer is would be an AI system based on this uh

on the filter

planning through optimization. But the guardrails are directly on the the state of the world that the system predicts is going to the world is going to go through because of its world model. Currently don’t have a world model. They cannot predict the consequences of their actions.

[00:39:58]

they accomplish an action kind of blindly.

Yeah.

Okay. Usually reproducing a some sort of script they’ve been trained on.

But they don’t have kind of a good way of predicting the consequences of their actions. They don’t have an explicit world model. If you have a if you have a system that has a world model, it can predict the consequences of its actions. And if before the system acts the you know the the state of the world that’s predicted can be scored as to whether it’s good or bad or dangerous or ethical or whatever

the system can stop itself from actually taking actions if it’s if it violates the guardrails. So how will you be able to tell whether the model you’ve built can do that can take into consideration and consider the consequences of its actions?

You you train the guardrails for it. You either hardwire them

or you train them. So for example, let’s say you have a domestic robot that they can cook. You can hardwire a at a low level a rule that says if there’s people around you and you hold a knife in your hand, don’t flail your hands, your arms, right?

[00:41:16]

Because you risk hurting someone.

there can be a very low level rule

that will basically freeze the system if you know if it has a knife and someone is next to it.

But that’s not the same as understanding the consequences, right? That’s actually just putting parameters. In that case, it doesn’t need to understand. It just needs to follow the guardrail.

That’s a very low level guardrail. Okay. But there would be a higher level guardrail which would be

okay if I make this movement, you know, the the blade of the knife is going to get too close to that person and I have a cost function in my guardrail that says

I can’t have the blade of a knife too close to someone.

certainly not moving

like an if then kind of.

It’s more like a cost a penalty function, right? It’s like you know there’s there is an objective function. If I want to grab the glass that is on this table the cost function is the distance of my hand to the glass.

Okay. And to plan a sequence of actions to get my arm close to the glass, I just need to minimize that distance.

[00:42:23] I can also have an objective function that, you know, repels

my hand holding a knife from any

or any thing that I, you know, I risk hurting. It just I pay a price for just doing that.

It’s kind of like you know, human laws do this. A law is basically a cost function that says, you know, you can steal this apples, but you know, you’ll spend a month in prison. like is it the cost that you’re willing to pay

or you’ll get in trouble, right? So

so think of it as laws except that humans,

you know, can choose to violate laws,

right? Right. Those AI systems will have those laws kind of hardwired and by

construction. So that’s the guard rails, but like the consequences example you just used actually is an interesting one because in a way the you know the consequences of criminality are kind of arbitrary consequences whereas the child moving through the world and developing a world model is learning the natural consequences of knocking this off the table or tripping over something or

you know with his sticky hand rubbing something on a surface or whatever. Right. So, it’s it’s more a that kind of world model building has a has embedded into it a whole set of natural consequences and you’re saying and on top of that you put sort of the artificial not artificial but the

[00:43:57] The guard rails that indicate other possible consequences you want to avoid.

Okay. So I think it’s slightly more comp complex than that. So there is the world model that purely makes prediction without rating

those prediction as to whether they’re good or

they’re good or bad or normative or anything. Yeah.

Okay. Then there are two things. There is an objective function which is which measures to what extent the task has been accomplished that you ask the system to accomplish. so in the case of humans we have you know those drives that you know drive us to accomplish certain tasks at at you know various moments.

and then on top of this there are guardrails which are other objective functions that either drive us or prevent us from accomplishing tasks that will result in particular outcomes that we judge as good or bad. Okay. And a lot of those guardrails are hardwired into us by evolution.

[00:45:00]

Like for example, we seek the company of other members of our species, right?

because we are a social species.

Yeah.

this by the way does not exist in orangutang because orangutongs are territorial.

They don’t seek the company of other orangutong at least at most you know in adult life.

Uh

except for reproduction maybe.

Yeah. so so there are drives in human nature you know hardwired into us by evolution that are there to kind of get us to learn certain things. So and to behave in ways that is not destructive for the species.

Right.

Okay. So, you know, we have this sort of natural tendency to kind of you know, be next to our next of kin and you know, protect them or whatever

or go under their protection. and less so as the

people we encounter are more different from us, right? I mean it’s a

it’s kind of

something that evolution built into us whether it’s you know appropriate for the modern world or not. so those intrinsic motivations and drives are you know what constitutes human nature. But we could be as intelligent as we are without being social possibly and you know we wouldn’t learn as quickly.

[00:46:34]

but look at octopus. They never meet their parents.

They’re not social at all. they get extremely smart extremely quickly because they only live about two years,

right?

It’s pretty amazing. But they don’t learn from their parents, right?

They don’t learn by imitation.

They barely interact with

other members of the species.

Yeah. But just going back to the guardrails it if that’s the case right our ability I think to I don’t know you’ll probably think this is an overstatement but our ability to keep ourselves safe I would say is predicated on our ability to predict what the things what they might do but as you said these systems are probably doing more than just simply fully predicting at this point, right? They’re there are other things that they’re evaluating. They’re coordinating. There are other things that they’re doing.

They’re not predicting anything. They don’t have a role model.

They are

reacting. You know, psychologists have this

you know, people like Daniel Canaman and others, they they had this concept of system one, system two. Yeah.

[00:47:46] System one is

thinking fast, thinking slow,

right? Actions you take, you know, without thinking basically, right? because you’re used to accomplishing the task and you don’t have to

Yeah. driving to work. You don’t you’ve done it so many times you don’t even for that matter.

Yeah. Just driving.

And system two is when you use the full power of your mental world model to kind of plan

the sequence of action,

do something that you haven’t done before or navigating a new space or

That’s right. So you have to kind of simulate the situation in your head and you know all that stuff, right?

LM are system one

essentially

because they’re trained, you know, to basically reproduce and they’re trained to reproduce human behavior. which includes all the entrepreneurs

then we’re really in trouble Yann humans are really problematic

I have said that LLM in their current form are intrinsically unsafe

now they’re not particularly smart so they’re not particularly dangerous either

yeah uh

and so say so speak to that why do you think they are dangerous the way they’re being built right now

because they’re not objective driven it’s not like you can give them an objective a task has to accomplish and they will do nothing but accomplishing that task subject to guard rails. That’s not the way that they are built or designed or trained.

[00:48:59]

The way they’re trained to behave properly is by massive training with lots and lots of examples. So, you know, basically this is, you know, training by imitation

and it’s very useful. You know, it works great. I mean, people use for all kinds of stuff. but here’s another example where it doesn’t work. autonomous driving. We have billions of hours of training data of cars being driven by people.

In principle, we could use this to train a big neural net to just reproduce human behavior.

Right? Given the

perception of the current environment like camera inputs and sensors like you know what is the yeah

wheel and pedal actions in that situation

doesn’t work. at least not to the level of reliability that would be desirable which is why we still don’t have completely autonomous self-driving cars

today

and the ones that we have don’t use much imitation learning they do but a lot of it is you know highly engineered perception system and they rely on very precise maps of environment and all kinds and depth sensors and LAR and stuff like that. So um

so where is the car they can learn to drive in 20 hours of practice like any teenager.

[00:50:22]

We’re very far from this. So those things and they those things are not particularly inventive except for code and

and mathematics.

I mean people are you know a little panicked about like the possibility of badly intentioned people getting a recipe for a biological weapon from

something like that. that’s that’s totally extremely unlikely for various reasons

like you don’t think it could come up with a novel biological agent that

so that’s the first thing I mean mostly that would come from specialized systems that are specifically built

to do this

which are not LLM right

they’re sort of different models that are really specific for that

that’s the first thing and the second thing is not because you have a recipe that you actually build the thing.

And so a lot of biologists start basically laughing at the whole scenario. They say like you don’t realize this is really hard. Yeah.

You know it’s not because you have a recipe or even a sequence that you can actually you know build a pathogen and then you have to distribute it and then

you know you have to make sure this thing is not going to kill you. And I mean and maybe if you can produce it you can also produce the antidote. So,

I mean, this kind of scenarios are like good, you know, James Bond movie scenarios, but not really.

[00:51:41]

So, that’s not what you’re worried about.

No, I’m not really worried about this, in the foreseeable future.

and then, you know, the scenario of like, you know, super intelligent system taking over the world because they don’t need us. that’s not the way we’re going to build them. Again, we’re going to build them to just do our bidding. They might be smarter than us, but like there’s a lot of people in the world who are smarter than me and you. And

but I don’t control them,

but they don’t want to take over the world. Right.

Right.

In fact, the smartest people I know

are really happy to not interact with anybody else. So generally they don’t even want to know people. so but what about things like what concerns me I guess is when the intersection of things like programs that are incredibly good like mythos at cracking any kind of encryption. combined with the idea that they could be given a task and behave in unpredictable ways that aren’t expected and violate their guard rail somehow. or maybe the people who are creating them aren’t being careful enough. So, it’s some combination of the capabilities that they’re being given and the people in charge

and the race dynamics that are forcing the entire industry to move at the pace that they’re moving. Those and the lack of oversight by any government agency or capability to do

[00:53:15] So. those set of you know

realities, I guess, makes me concerned that we’re moving too fast with the wrong people in charge and nobody actually providing the filter that you’re talking about, at least not at the human level, right?

Okay. So, first of all, mythos when it cracked

face did exactly what it was asked to do.

Okay. It was not a loss of control. It was the thing actually doing what it was asked to do. Right.

Except the guardrails were stupid.

Right.

and ineffective mostly because the people working on this had no experience with cyber security.

and therefore it didn’t put in place the right protection. probably inadvertently. It’s unlikely that it was deliberate. Although it’s kind of strangely good marketing,

you know, because you can claim your system is so powerful that it can like, you know, hack into other people’s system. Okay, that’s one point.

[00:54:22] Second point, when you have a system that can detect vulnerabilities and crack you know computer system, you can use it to protect and this is the way a lot of security holes are plugged. you know, someone some white hat hacker discovers a vulnerability in a piece of code

and then does not publish it, you know, tell the people who can actually fix it or if it’s open source, they can fix it themselves.

you know, before it gets exploited by black hats. so it’s more of the same in terms of cyber security. It’s like just an increased level of detection and protection,

but it’s basically, you know, my good AI against your bad AI. Usually the good guys have better AI than the bad guys. So like it doesn’t change the dynamics because that’s what’s interesting is that to detect the and try trace the the hacking face used

an open source LLM from

Yes. Exactly.

[00:55:25]

Chinese model, right?

Yep. That’s right.

and so

the Chinese defended us against several models. I mean it’s making an important point which is

we’ll we’ll not get to security of AI systems through of security.

the best way to get AI systems to be safe and secure is for them to be more open.

This is the complete opposite argument that you will hear from Dar and Sam and a few others. Yeah,

they what they claim is that AI is intrinsically dangerous and should not be put in every hand,

right?

that it’s right for them to

Right. Right. right

have it because you know they’re smart enough, right? And they have good intentions but not the

the unwashed masses.

I think that’s a new form of currentism.

I mean

humans do have a tendency to use things to do bad things though, right? I mean that is whatever we’re building there always going to be bad actors who are going to try to use it for nefarious purposes. At what point do we and how do you think do we regulate to try to prevent those things from happening?

[00:56:48]

Okay. So, I think I mean clearly you want to regulate applications and products.

Whether they use AI or not, if you’re going to have a you know driving assistance system in your car, you want some sort of government agency to say, “Okay, this is safe, right, or not.”

but those regulations already exist essentially for all the domains that you know that make sense.

And AI doesn’t have there’s no oversight agency that says, “Okay, this is safe now. You can

you need one. You need one for X-ray, you know, for health equipment.

You need one for cars. You need one for every application domain. Whether it uses AI or not is irrelevant. You need the agency of course to understand, you know, what AI can bring to the table there. so you need the FDA to understand what an AI system can and cannot do in terms of diagnosis, aid and things like that to authorize or not. such a system on the market.

Well, I mean, think about mythos. I mean, if you’re like, okay, if everybody could get Mythos simultaneously and immediately apply it, right? And if it was like there was no distribution error or noise, it was just perfect, right? In that world, it could make sense that it was like, okay, everybody’s got it immediately and now it can’t really it can be an offensive tool. But that’s not how technology gets open.

[00:58:11]

AI would absolutely never do this because they would be sued into oblivion. You mean release

release anthropic? I think anthropics orthropic whatever if you know if you release a system

that can obviously be used for just nefarious uses

without any protection that’s negligence. I mean you would be sued into oblivion.

So those companies are actually have a pretty good incentive to actually do it right.

now that doesn’t mean you don’t you know you don’t want explicit regulations for certain aspects of you know what those systems can do and cannot do.

But that’s like because what the argument you just made is similar to like okay you expect food companies to make sure that their food is safe and healthy and so on. Now that we know that’s not totally true, right? That’s true. You’re right. And because they will make tradeoffs for the sake of profitability that aren’t necessarily in the public agreement.

It’s the same in the you know in you know airplane design and uh

you have airplanes start falling out of the sky. It’s not good for business.

[00:59:20]

I mean there is the FAA but there is also

Right. Exactly.

There’s also private companies that actually

certify certain like maintenance processes and stuff like that and inspections. And you think about like pharma and the FDA. Like it was the wild west in the early 1900s. People were putting anything out on the market, calling it medicine. People were dying. They had to do something different.

That’s what I’m saying. Those regulations already exist. Whether AI is used for those products or not is relevant.

I mean, every technology that’s ever been put out has always had like a you know, a dark side and you know, bad side effects. And the the question for society is what type of appropriate regulation will preserve the benefits of that technology and minimize the bad side effects.

and there is a website that I would recommend people visit because it’s educational and hilarious at the same time. Yeah,

it’s called the pessimist archive.

[01:00:29]

Okay.

So, it’s a gentleman who as a hobby basically collects newspaper clips

from various periods of history where people have said, “Oh, this new technology, this new cultural phenomenon, this new communication technology is going to destroy society.” And just go back a few years, video games were going to destroy the youth.

And a few years before that,

rock music,

rock and roll,

right?

And comic books,

right?

And recorded music and jazz and go back, you know, long enough novels.

Novels were going to destroy young women because they would spend their time just reading instead of working in the fields,

right? So,

you know, why do you waste your time reading books? Yeah.

And and there’s kind of progression depending on your age or your willingness to adopt

new things. Was that Plato or Aristotle who said like writing was a terrible idea because it would cause people to

not have memory anymore.

[01:01:34]

Yeah.

Um

it would destroy oral tradition and the ability to think. Yeah. Exactly. Build arguments in your head.

So we got to be careful like not to overreact. I mean there’s clearly some areas where it makes sense to to regulate.

Yeah.

But but I think the more general question is what you know what people have been talking about over the last several years is if you think AI is intrinsically dangerous regardless of the application you should regulate research and development in AI.

And in my opinion that’s obscurantism. That’s ridiculous.

Yeah.

That makes no sense. I do not believe AI is increasingly dangerous

and regulation at the level of development doesn’t seem to actually make a lot of sense. I think it’s at the level of deployment that there should be some check. Right.

So is there the larger danger and I’ve heard you say this before maybe just the consolidation of power and compute and data and so on and so forth in a few companies.

[01:02:40]

Right. Right.

That’s really dangerous.

Yeah. And so speak to that a little bit.

Right. So I mean more and more we’re using AI assistance as our main source of information.

Yeah.

so we rarely go to search engines anymore, right? We rarely go to Wikipedia or to a library

or to a primary source of information. We just ask or you know

Yep. Put in a question and system comes back. Yeah.

And they come from three or four companies on the west coast of the US or China at the moment.

Yeah. that might be acceptable for some American governments, it’s completely unacceptable everywhere else in the world.

talk to you know democratic governments in Western Europe or in Asia.

Yeah.

they do not want their citizen to get all of their information diet through AI assistance that our proprietary systems produce in the US.

For sure they don’t. completely unacceptable and so they

there’s like a cultural sovereignty that’s lost sovereignty and even epistemology I think

political biases value systems

y

I mean you know it there’s a lot of implied value systems in AI system there’s you know they speak certain languages and not others

they have access to particular type of data not others

and so there’s a lot of kind of you know underlying values and centers of interest in the

[01:04:14] Current crop of AI systems. And so in my opinion, the only way to fix this

is for I mean basically foundation models, AI foundation models

are quickly becoming an infrastructure

and infrastructure always wants to be open.

and so the future of the AI market, if you want

which is completely inevitable, the question is how long it’s going to take, is one in which basic foundation models are open and free.

And then commercial applications are built on top of it.

Yeah. by integrating multiple models, fine-tuning them for vertical applications, making them palatable for the public, having them, you know, adop certain personalities, being fine-tuned for particular

cultures,

linguistic value systems, cultures, political biases, whatever. Well, you need a diversity of AI assistant for the same reason you need a diversity of the press.

Yeah.

[01:05:15]

You don’t want to get all of your news from a single political philosophy and value system. that’s unthinkable for most countries in the world.

so that’s the future I want to see. It’s one in which you have you know the expensive part of producing an NLM can be done collaboratively with you know multiple countries, regions, private companies etc in the world to build a to train a sort of global system that would be sort of a repository of all human knowledge.

Yeah. but then because it’s open and free on top of this you know private actors can you know build practical systems on top of them and have a good business from it. that’s the future I want to see. So I mean it’s like you know these laws that exist in various European countries I don’t know if there is the equivalent in the US where a printing press cannot discriminate what they print.

[01:06:23]

Right. Right. Yeah.

It’s like your internet service provider cannot filter what you know they have to be a neutral

pipe.

Right. And they certainly aren’t now. and we don’t know how they’re being trained or manipulated or how the data is being weighted. I mean, we just don’t know.

Well, we don’t know. And, you know, they’re necessarily biased. It’s not It’s not for nefarious reasons. It’s not because the companies that build them,

you know, want a particular type of bias. It’s just that necessarily you build a system, it’s going to have some bias, right? like you know every newspaper is going to have some

bias in the way they

you know and so there is no fix for this like you’re never going to have a completely neutral system and so the only fix is diversity

is diversity

you’re not going to get diversity unless you have a free foundation model on top of which people can train diverse models

so you need you know basically financial model neutrality in the sense that the same way the printing press you know in France for example there’s a law that says you know printing press cannot choose what they

accept to print first of all but then

so the news distributors cannot discriminate either they have to they have to distribute all the

right

all the you know newspapers magazines

right

they can’t choose

they can’t choose and be like no we’re

[01:07:54] Not going to send this one out. Right.

Right. You know, they have to be neutral in that respect.

Is it possible to take the foundation models that have been built so far and somehow open them up to development by other you know maybe countries stacks so that other applications can be built on top of them or will they always have an embedded bias in them?

They always have an embedded bias. But but if you make them open fine tunable by anyone

then you make it possible to have highly diverse

set of assistants built on top of them. You cannot do this with the models currently produced by open eye and Google because they’re closed.

But could they open them or is that not possible? It’s like

it’s not in their kind of business model.

Well, definitely not. They would they wouldn’t allow it because they’re I mean

and

they want to capture the world.

That’s right. And and given the level of investment that they’re currently

Yeah.

[01:08:56]

you know kind of money they’re raising they they think that’s the only way for them to kind of preserve that business model. Now

it’s a sort of winner take all view. So the the reason that you know anthropic and diamed are saying is again

you know we’re not going to open it because it’s too dangerous

right

like you can’t you know let the unwashed masses

have access to such powerful technology which I think again is obscurantism

right

a new form of medieval obscurism

now other companies the Chinese companies are opening models they’re open and in fact they’re increasingly used for that reason. Now they are biased in some other way initially but because they are open they can be fine tune to remove that bias or create another bias rather.

Yeah.

And they’re they’re very popular. They’re getting

they’re popular and they’re good and they’re increasingly popular but they’re Chinese and so there is a little bit of you know

trepidation. Trepidation you could call it this way.

[01:09:58]

Yeah.

and

there’s certainly yeah there’s certainly bias in their models as well because of the whole context in which they’re being developed. I don’t imagine the companies in China would not I think be able to behave the way the companies here are. The Chinese government would not put up with entertaining the idea of loss of control. they wouldn’t put up with the idea that, you know, these things are dangerous and could like I just don’t think they’d allow a company to behave like that because the one thing that I know about the Chinese party is that control is the one thing that they are going to maintain at all costs.

Yeah, that’s true.

Yeah.

and it’s interesting that you know the Chinese companies are mostly opening up their models because they see it as a way to have visibility in the market. you know the Chinese market in terms of services is really an isolated ecosystem. Yes,

that’s a way of having

an impact, you know, around the world. and the business model is still variable for them like they derive enough revenue to kind of justify their existence.

[01:11:08]

Um,

and they’re doing it at much lower cost as well because their whole industrial model is to stay slightly behind, right? So,

they’re leveraging the advances that these companies are making. So I think you know ultimately the solution is and it’s it’s it’s a project that I’ve I’ve advocated for for several years now. it’s called project tapestry and it’s um

the idea that a a global foundation model that could be the a repository of all human knowledge.

Yeah. could be built trained collaboratively

with partners that can be countries or regions or universities or private companies collaborating to

to training a global system

using a form of training procedure that does not require each of the participant to share that data. So there are techniques by which you can train a model

where each participant contributes data and compute.

[01:12:12]

But they keep control of their data. They don’t share their data.

So it’s like zero knowledge data. Like it’s it’s

it’s their own data.

They’re just training on their data, but it’s never

it’s never distributed. It always belong to them.

It’s never made publicly available. It’s never shared with the other participants,

right? what those participants share are parameter vectors. So basically the big list of numbers that define the model right

and when they train on their own data they periodically exchange the parameter vectors with the other participants

and the system eventually arrives at a consensus parameter vector

that is as good as if

you had train a single system on all the data from all the participants

but they don’t exchange that data. So they keep control and sovereignty on the data and in my opinion there is a lot of data in the world that is currently not publicly available not usable

or maybe not even digitized which if it were used in that way would produce models that are actually better than the commercial systems.

[01:13:18]

Yeah. and a big chunk of what makes the quality of an AI system is the post training which is you know asking system to the questions to the system and then giving it the answer or rating the answer it produces

right

they can be crowdsourced across the world

right

kind of like a little bit like a form of Wikipedia but instead of writing

you know articles about particular topics you just train an AI system to answer question or produce information about particular topics.

And so this tapestry idea would be a way to actually get a lot of this data off the sidelines.

So it would be a way to have an open

system train on all the world’s languages with all the world’s value systems and biases and whatever you want to call them

without violating data privacy

without violating I mean ownership of data from countries. So you know India has 22 official languages and you know literally hundreds of local languages

and dialects

and you know

even more dialects

uh

like who’s going to train do you think a model that speaks all those languages is going to be trained by anthropic open or

like they don’t even have the data for it

but India can build the data for that

yeah and they are actually I think they’re going to build their own stack.

[01:14:45] Yeah.

Well, they have a they have an effort called Baren which basically

is that but then why should they not collaborate with

you know Vietnam that has the same problem or Indonesia has the same problem or Korea or Japan or France or Germany or every other country in the world or Kazakhstan

um

and so you know all of those countries will have invested interest in participating in a program like this. Well, it seems like the interest right now among nation states, the big concern

when you talk to political leaders is sort of sovereignty. That’s a big one. which is really difficult to accomplish. Not just because of the closed model thing, but just because there really isn’t it’s it’s impossible to sort of create a sovereign stack. It really is. It’s not just at the frontier model level, frontier models being part of the problem. but this sort of winner take all approach that these companies seem to be pursuing certainly puts them in you know puts them in a position of being very concerned about whether or not there is going to be sovereignty

going forward because a super powerful AI that everybody is using

you know it doesn’t bode well for countries or nation states

right so I mean this there’s a future that I think we can all live with. there’s there’s an interesting parallel also in history.

[01:16:14]

If you go back to the early days of the internet,

yeah,

if you wanted to put together an internet service of any kind, you had to buy computers from Sun Micros, you packer, Dell or one of those which would run a proprietary operating system.

based on Unix or Windows

and on top of this you would have a proprietary software stack

right

that lasted only a few years

right

after a few years this was completely wiped out and replaced by community hardware

open source operating systems Linux mostly

Linux yeah

and then completely open source stack on top of it Apache

MySQL

PHP and most websites and internet services were built with this. Facebook was built with PHP

on top of Linux and

you know Apache. So uh

so you can like privatize at the level of the application layer just not the infrastructure layer.

[01:17:20]

The infrastructure really wants to be

open because it’s more secure, it’s cheaper, it’s portable, it’s you know and it it creates the landscape and the substrate for a vibrant industry. I saw that effect at Meta in 2023 where when Meta open sourced Lama 2 this completely jump started the AI industry

right

it made people realize oh now we have you know an open source model we can fine-tune for anything we want

right

LMA 3 was great four was not so great so disappointed a lot of people

and then caused Mark Zuckerberg to completely reboot AI at Meta and for one year

Meta kind stopped adopting this sort of open stance and now he’s coming back to it.

Which I think is a really good evolution. we see kind of a similar move towards openness from Nvidia

with you know they bought hanging face

right

they’re partnering with Mistral in the Neotron

consortium which also has towards an open model as well

I think Nvidia has like a vested interest in the ecosystem being open

they because they need more actors right they have the GPUs do that to all of them. So

yeah,

it’s much better for them to have a wide diversity of clients rather than like three.

[01:18:51]

Right. but but ultimately the you know it it has to be I mean this sort of tapestry has to be like owned by a nonprofit foundation

possibly with commercial outfits on top of it. the so there’s obviously incentive inside the company to say that the models are dangerous and therefore this captive you know closed model approach is the right one. It seems like that is the fact that these companies are now also very involved in national security and there’s been this sort of you know increasing collaboration I guess you could say between the national security apparatus and the military-industrial complex and these companies would seem to be like feeding into this idea that these have to be controlled closed systems and we would also have an incentive to ensure that they are the systems that are used most globally. And it’s a terrible combination of factors. But it seems like what I worry about is that currently the US government is inclined to actually keep them closed because of the national security implications, I guess. So, strangely enough, you hear people like Alex Karp, the CEO of Palanteer, actually advocating for a more open ecosystem.

[01:20:22]

Interesting.

you know, he doesn’t like the idea of having just a few proprietary systems controlled by a few companies.

Yeah.

He thinks that’s actually counterproductive.

Um,

there are arguments on both sides of the political spectrum in the US. Yeah.

In favor and against open model.

so a lot of people on the sort of hawk hawkish conservative side

saying you know this is sort of revealing our secrets to the Chinese, right?

And people also on the kind of more progressive liberal side also you know

can be quite hawkish also.

Okay. But there are people like Eric Schmidt who basically completely changed his stance on it.

He was kind of a little bit opposed to open source because he said like that’s giving our best secret to the Chinese.

but then deepse happened.

Yeah.

He said like okay the Chinese are you know at the same level they’re really good like you know

we’re shooting oursel in the foot by not having open source

open source. Yeah. So, so open is now a big supporter of open source

models is actually funding a few projects open source project LLM projects academic mostly but yeah

so and he’s very much you know

interested in all issues of defense and

he’s actually very involved he’s been for a long time

yeah so he totally changed his stance on it

and surprisingly enough There

[01:21:55] Is actually quite some support from the Trump administration for open models,

right? I don’t agree with basically anything from the Trump administration but they

but on this I have to say they are on the right side of history and and you know one reason the a lot of people in Silicon Valley got scared

before the election you know by the some rumors about you know what the Democrats Democratic party’s attitude towards AI

was because of talks of regulation, you know, from the idea that would have basically made open source models illegal and there were, you know, proposals of laws in California,

SB 1047 and various proposals and the,

Biden administration

for regulation that spooked a lot of people in Silicon Valley,

right?

[01:22:56] I actually gave a I was in front of the Senate subcommittee on intelligence talking about those issues exactly that issue like you know is it dangerous to open source or we should take oursel in the foot by

betting

open source and that was my argument that the vibrancy and vitality of the research and development ecosystem in the US is due to openness

right

and if we go closed, we’re actually going to fall behind.

Well, thank you. Thank you for sharing the idea and thank you for sharing this hour with me.

Pleasure.

All right. Thanks, Yan.

Curated by Kyle Harrison. Organized by Claude. Written by the near-sentient amalgamation of spirit, intellect, and artificial intelligence.