The Centralization of Power in AI

Summary

Kyle on the MLOps Community Podcast (#181) with Demetrios Brinkmann, built around the Contrary Research report The Openness of AI. The most technical of his appearances, and the one aimed at practitioners rather than investors — it is also, at ~61 minutes, one of the longest.

The framing: three buckets. Incumbents with established products and distribution (Microsoft, Canva, Adobe, Notion) who can simply roll AI out; companies trying to be the foundation model, which requires training data, access and capital and is “a very unique game”; and everything else — tooling, orchestration, fine-tuning, and the use-case layer that gets the pejorative wrapper label. Kyle’s caveat is that not every application company is a wrapper, “but there is a spectrum,” and that middle bucket is where the centralization risk lives.

On moats, the line that does the work: “if OpenAI can roll out your product in a couple of months, you never had a moat to begin with.” Most of what looks like competitive advantage is first-mover advantage — Copy.ai and Jasper were first to point GPT at copywriting and branded around it, which is not defensibility. His rule: spike in something that is not the thing everyone else is racing to commoditize. If your strength is precisely what companies with billions of dollars are trying to turn into a commodity so it becomes the install base, “you’re screwed.”

The two worked examples are portfolio companies. Replit is “not an AI company in exclusivity” — it’s a developer ecosystem that would still be a good product with no generative-code breakthroughs at all, and Ghostwriter feeds a flywheel that exists independently. Nomic spikes somewhere stranger: Atlas, a tool for visualizing every individual data point a model is made of. Kyle’s illustration is a canvas of Stable Diffusion printed for Contrary’s New York co-working space — billions of dots, and “you can see, oh, this green corner is Kermit the Frog corner.” That competency is what let them build GPT4All, roughly GPT-3.5-performant, locally runnable and far cheaper on compute. His point: GPT4All versus OpenAI head-on would be brutal, but it isn’t the business — the data-understanding engine is.

Why closed is winning, and the honest reason. Kyle traces the arc: AI was a literal black box because nobody understood why an output was the output; transformer architecture made models progressively more legible; and OpenAI is now “deciding to put it back in a black box, because it’s a lot easier to monetize something that people don’t understand.” The stated justification is safety; he argues the operative one is competition. The strongest version of the counter-argument he credits to Clem Delangue of Hugging Face — at Cerebral Valley and in Congressional testimony — that the number one danger in AI is concentration of power. And the rebuttal to the bad-guys framing: essentially every technical capability eventually reaches bad actors, so “the thing that keeps the bad guys in check is that everybody else has it too.”

Incentives explain the alignments better than values do. Why is Meta — “the company no one thought would be the poster child of openness” — releasing Llama? Because it has no cloud business. Microsoft’s swings are downstream of wanting compute to happen on Azure; Meta is advantaged by capabilities expanding as fast as possible so it can apply them to products that are already free. The historical precedent is TensorFlow versus PyTorch — Meta’s framework “ate TensorFlow’s lunch” — so Meta as an open-source bastion isn’t new. What is new is the moral sub-narrative: there was no ethical valence to a machine-learning framework, and there is one here. Databricks is his example of doing open source meaningfully alongside a sustainable business.

The privacy argument, and why it’s an audit problem. Recorded a day or two after ChatGPT Enterprise launched, so the enterprise checkboxes are fresh: SOC 2, encryption, the bare minimum most companies look for. Kyle’s objection isn’t that OpenAI is lying — it’s that the claim is structurally unverifiable. “Is there any way to confirm that none of my data has ever been used for you to train and improve GPT-5? That’s pretty hard to audit.” You can be SOC 2 compliant and still have data exhaust: even if your data never trained the next model, what the model can leverage from what you’ve done has implications nobody can currently attribute. He predicts the same curve social advertising ran — years of checkbox indifference, then a GDPR-style outcry once attribution becomes legible enough that people realize what “we didn’t train on your data” actually did and didn’t cover.

Two groups won’t default to OpenAI: highly regulated industries (healthcare, financial services, government) who want to run locally, and companies with enough in-house expertise to build their own — with Uber’s open-source output as the precedent. Both are gated on the talent shortage, which he characterizes with the going joke that “an AI researcher can make as much as an NFL quarterback.”

The playbook, named. Kyle argues OpenAI wants general-purpose model use cases commoditized as fast as possible, because once GPT is in everything, “when you do have that outcry in a couple of years, it’s so much harder to get them out, because they’re everywhere.” That is the Microsoft playbook: Teams didn’t win on quality — in his opinion it’s a much worse product than Slack — it won on ubiquity. His phrase for it, which Demetrios immediately seizes on: “corporate malware.”

On not stuffing LLMs into everything. Invoking only a Sith deals in absolutes — and noting the irony that this is itself an absolute — his rule is that any company deciding what to build from what’s hypest “is probably not a great product organization.” Ramp is the counterexample: ask what your core competency already is, then ask how the technology extends it. Bolting on AI to earn a .ai on a landing page is “a very aesthetic, surface-level way of doing things.”

How far along we actually are. His numbers: SaaS maybe 70–80% mature, cloud computing 50–60%, AI maybe 10%. Clem’s line that practitioners look at ChatGPT and see “autocomplete on steroids” rather than AGI. He borrows Matthew Lynley’s reframing of the iPhone moment — not perfection, but the point where a technology is general-purpose without needing massive sophisticated services around it — and notes cloud computing arguably still hasn’t had one, given that systems architecture is still outsourced. So the doomer and the imminent-AGI camps are “as stupid” as each other.

On agents, he’s enthusiastic about the demos and skeptical about production, and the diagnosis is specific: models handle one interaction layer well — an API call out and back — but booking a trip means traversing six or seven, and “I’ve never seen anything that’s good at that.” Demetrios supplies the compounding-error version from Honeycomb’s Phillip: a 1% error rate per hop stops being 1% very quickly. Kyle’s aside on prompt engineering is the nicest thing in the segment — chat is a hard interface because the model may be better at the task than you are at articulating it, though he thinks being forced to structure a sentence precisely could have a real side benefit. And image models function as “kind of like a personality test” — which model suits you says something about how you think.

The closing warning is about capital, not technology. Progress follows a curve; the thing that can genuinely accelerate it is excitement, and openness is what lets more people contribute to it. But “capital is a dangerous drug.” The early-2000s green bubble vaporized the overwhelming majority of what went in and set climate tech back years — “we’ve only just recently started to come out of that capital-induced winter.” Same read on the 2020–21 crypto fever: he thinks blockchain has real use cases and that the mania “did more harm than good… it invited more hucksters that are going to take years to wash out.” Put in 10x the capital a category needs, and when it’s destroyed the people who lost it become the loudest voices against the thing.

Transcript

MLOps Community Podcast #181, ~61 minutes, hosted by Demetrios Brinkmann. ASR errors and names cleaned, text paragraphed; nothing reordered or summarized. The transcription mangled Nomic into “Noam Chomsky” throughout — corrected here, along with GPT4All, Andriy, DALL·E, SOC 2, PyTorch/TensorFlow, Clem Delangue and Phillip. The show’s running joke testimonials (a “listener” from a bank, and the sign-off from a community member) are marked as inserts rather than transcribed as dialogue.

Cold open and intro

Kyle Harrison: I’m Kyle Harrison, general partner at Contrary, and I don’t drink coffee — but I have a crippling addiction to Diet Coke.

Demetrios Brinkmann: Welcome, welcome everyone. We are back with another MLOps Community podcast. I am your host, Demetrios, and today I am talking with none other than Kyle, who you just heard say he doesn’t like coffee.

This conversation was fascinating on so many different levels. I really appreciate the report that he put together, which if you do not know is called The Openness of AI — I’ll leave a link in the description. He got into why he wrote the report, what it’s all about, and the dangers he calls out in it. One of those dangers being the centralization of power when it comes to AI.

We’re talking about OpenAI a lot in this conversation, and the pros and cons that come with OpenAI. I thought it was fascinating how deep he went into the privacy part — for me it feels like, for lots of people, privacy is one of those things like peace on earth and the data mesh: everyone wants it, but at the end of the day you’re willing to sacrifice a lot if there’s an easier, less-friction way of doing something.

And to timestamp this: we are recording about a day or two after ChatGPT Enterprise came out. So we talked a lot about that, and how trustworthy OpenAI is when it comes to dealing with your data. Are they going to train GPT-5 on your data? Well — maybe, maybe not.

His ability to be super bullish on lots of companies opting away from using OpenAI was fascinating. But he also talked about how OpenAI’s strategy right now is to proliferate the market, get into the hands of everyone, and be the easiest way to do anything with AI — because that’s the playbook. As soon as they’re in everyone’s hands, then three years down the road, once they’ve trained whatever on everyone’s data and we don’t like it and we want to rise up, it’s going to be a lot harder to rip off that band-aid.

Kyle is a general partner at Contrary — he leads Series A and growth-stage investing. His portfolio includes Ramp, Replit, Cohere, Snowflake and Databricks. He’s also got a Substack where he shares his analysis on the venture capital landscape — it’s called Investing 101.

Three buckets

Demetrios Brinkmann: We came on to talk about all things AI, and I want to dive into the hype. You’ve been going deep into the openness and closedness of AI, and thinking about whether there are real businesses to be built on top of this. Where are you sitting right now?

Kyle Harrison: By every stretch of the imagination this is a huge paradigm shift in technology, and everybody is appreciating it.

The thing I find most interesting when I talk to companies that have been operating in this space — people who have been building in this world for years and years — is that so many of them feel like they’ve been, maybe not standing still, but slow and steady wins the race. And then all of a sudden ChatGPT was a wake-up call for a lot of people. But there wasn’t necessarily a fundamental change in the underlying technology. Transformer architecture has been around a long time and had slowly been improving in what it was capable of. ChatGPT was just the thing that put it front and center in everybody’s mind. So really what you’re going through is an awareness of the capability that exists.

What you’re now seeing is a bifurcation of where companies are positioned, and you effectively have three buckets.

First, you have incumbents with established products and established distribution — massive companies, whether that’s Microsoft or Canva or Adobe —

Demetrios Brinkmann: Notion, exactly.

Kyle Harrison: These incumbents can roll this stuff out. That’s one bucket.

Second, you have folks trying to be the foundation model — the provider that as many people as possible, for as many use cases as possible, are building on. That requires pretty extensive training, data, access, capital. It’s a very unique game.

And then you have everything else. Every other company trying to build AI tooling, orchestration, fine-tuning — the whole MLOps world. And then the use-case tools, where the pejorative term is the wrapper around ChatGPT. Not every company is that. You can leverage GPT and not be just a wrapper, not just middleware. But there is a spectrum, where some of them literally slapped some UI on and ChatGPT is the entire back end. That whole swath is the majority of AI companies.

The biggest thing I think is problematic is that a lot of the power and influence happens in the middle bucket, where companies are building the foundation models. That’s where there’s the most potential for centralization of power.

When we wrote The Openness of AI for Contrary Research, that’s a big section of what we dove into: what does it mean when you have companies that become a really important foundation for everything else, and they’re pretty centralized and controlled by a small subset?

Moats, and what to spike at

Demetrios Brinkmann: There’s serious revenue being generated at that application layer — zero to a million really quickly. What I find fascinating is the quality of that revenue and the ability to create a moat. What happens when OpenAI replicates what you’re trying to do, and they have all the attention in the room?

Kyle Harrison: Number one: if OpenAI can roll out your product in a couple of months, you never had a moat to begin with.

There are elements of how much a competitive advantage really exists versus just a first-mover advantage — and the vast majority of that stuff is first-mover advantage. Companies like Copy.ai and Jasper were just the first to get there leveraging GPT to produce copy, and they branded themselves as a copywriter’s tool. That’s first-mover advantage. It’s not a competitive moat. It’s not defensible.

The way I look at companies today and think about defensibility is: you have to spike in something that is not the thing everyone else is racing to commoditize. If the thing you’re good at is what everybody else with billions of dollars is rapidly trying to commoditize — because they want to get into the install base and be the thing everybody builds on top of — you’re screwed. You’re going to get swept under the rug in that race to the bottom.

Take Replit. Replit is not an AI company in exclusivity. They don’t think of themselves as we build AI products, full stop. They’re a developer ecosystem — a tool that can do everything you need to build products, write code, host it, build out applications. That’s what they spike at: a very good developer environment. They’re able to leverage AI to do things like Ghostwriter, their code assistant, and that’s great — but it feeds into a flywheel that exists independent of AI. Even if we’d had no generative code breakthroughs, if Copilot hadn’t happened — Replit would still be a pretty good product as a developer environment. They’re able to just take advantage of this stuff.

Same thing with Nomic. Brandon and Andriy, the two co-founders, were some of my co-authors on the Openness of AI report. Nomic is really unique because they spike in a very different place: they’ve built a data visualization tool that lets you visualize all of the individual data points an LLM is made up of.

We have a co-working space in New York, and Nomic printed us out a canvas of Stable Diffusion. You can literally see this image map — you can’t see the images because they’re tiny dots, there are billions of them — but you can see, oh, this green corner is Kermit the Frog corner. You can visualize all the different subsets of this image model.

That tool is called Atlas, and that’s where they spike. They’re an incredible tool for visualizing and fine-tuning and understanding the underlying data behind a model. They used that to build GPT4All, an open-source foundation model that is basically as performant as GPT-3.5, can run locally, requires way less compute, and is pretty well trained.

If it was just GPT4All versus OpenAI, that’d be really tough, because OpenAI is rapidly trying to monetize that. But what Nomic is doing is demonstrating their core competency — this entire engine around data visualization and understanding models — and they were able to use it for a particular use case.

Demetrios Brinkmann: So the lawyers who are suing Stability AI are going to be calling up Nomic pretty soon and asking for that visual.

Kyle Harrison: They can definitely drill into it. And that’s a whole other super interesting point — when you have models trained on the broad swath of the internet, where does the attribution come from?

Why the black box came back

Demetrios Brinkmann: Let’s dive into the report. How did you get the idea of creating it? Was it after Llama 1 came out?

Kyle Harrison: A big part of it is chatting with the guys at Nomic — we’ve invested in the company and worked with them closely. We published a report after Llama had come out, but there were still a lot of limitations, no commercial use cases and so on.

Really, the thing that spurred us down this rabbit hole was the increasing limitations on the visibility of GPT. Some of the folks at Nomic posted a bunch about this: the more OpenAI says we’re going to limit what visibility we share on the underlying data, the methodology, the weights, all these components of how we built GPT-4, the less capable anybody using it is of understanding it.

We’ve gone through an arc. Originally people referred to AI as a black box effect — and it literally was a black box, because nobody really understood why the output was the output. We understood the algorithm, and there was an output, and who knows. Then increasingly it became more understandable. Transformer architecture absolutely made it more possible to understand how models get built and to try to better understand attribution.

And OpenAI is increasingly deciding to put it back in a black box — because it’s a lot easier to monetize something that people don’t understand.

Demetrios Brinkmann: It goes back to the lawsuits. They’re probably seeing: Copilot’s getting sued, Stability’s getting sued, we’re one step away.

Kyle Harrison: And OpenAI is in it, because Copilot is built on GPT. The Microsoft–GitHub–OpenAI relationship is right at the center of that lawsuit.

So the increasing desire to close off some of those components — they do it in the name of safety. There’s a whole argument of AI safety that allows people to close things off. When in reality, the main reason for closing that stuff off is competition.

Clem Delangue, the CEO of Hugging Face, makes this argument the best. In the report we cite both a conversation he had at the Cerebral Valley conference a couple of months ago and his testimony before the US Congress, where he says the number one danger in AI is concentration of power. People are trying to limit visibility into their models because it allows them to maintain more control over what’s happening.

What that causes is a long tail of companies, of nonprofits, of governments that are more limited in their ability to respond and more dependent on one central failure point.

And the argument is: well, we don’t want the bad guys to get the capability. The reality is that the vast majority of technical capability that has ever been created, eventually the bad guys get hold of it. The thing that keeps the bad guys in check is that everybody else has it too. If we allow a couple of big companies to be the centralized source of this capability, you put everybody at a lot more risk — in the name of AI safety.

Incentives, not values

Demetrios Brinkmann: I have a friend at Google who said people there aren’t super excited about being transparent anymore, because they feel like they got hoodwinked. They put out the transformer architecture, and next thing you know there’s another company using it and not sharing.

Kyle Harrison: It comes back to incentives — who is incentivized to do what, and why.

Google for a long time was at the cutting edge of AI and thought of as the leader, at least in terms of technical talent. I had the chance a couple of weeks ago to interview Aidan Gomez, the CEO of Cohere — one of the co-authors of the Attention Is All You Need paper that led to transformer architecture.

Why would companies respond the way they respond? For a lot of the larger companies — the Googles and Microsofts — it comes back to cloud computing. Microsoft’s massive swings in this space are largely driven by their intention of getting as much of this compute as possible happening on Azure.

And the reason you have a company like Meta who is comfortable releasing Llama and being much more open is that they don’t have a cloud computing business. They’re advantaged if capabilities expand as quickly as possible, because they can then leverage those in their products, which are their core business. It’s much easier to see the through-line where that gets better for everybody, because you have products that are largely free, those products get better, and everybody else can benefit from the advances too.

If you have one company trying to maintain the advances themselves and centralize both power and profit into one platform, it becomes a lot harder for everyone else to benefit. It just leads to more and more monopoly power for a small subset of companies.

Demetrios Brinkmann: What a plot twist though — Meta, the company no one thought would be the poster child of openness.

Kyle Harrison: One of the case studies in the deep dive is the TensorFlow versus PyTorch evolution. TensorFlow came first, from Google, and was the machine learning framework everybody was using. And then Meta released PyTorch in 2016, and it’s just hands down — the quote from one of the engineers we talked to was that PyTorch has eaten TensorFlow’s lunch.

So there is actually precedent for Meta to be the company that becomes the bastion of open source. I think what surprises people more is that with PyTorch and TensorFlow there was no moral sub-narrative. It was just a machine learning framework — they did better, it was open, they had better contributors, it worked out.

Here, there is a moral sub-narrative, and that’s what surprises people. Open source feels like the morally better place to be in AI, and nobody expected Meta to be the moral leader — because that’s where they’ve fallen down, where people feel like it’s the enemy of democracy and all these issues. Which I don’t know that I buy into per se; there’s a whole other conversation about human nature, and technology usually just amplifies what’s already there as opposed to creating it.

Most people say, hey, I’ve got to run a business, I’ve got to turn a profit, it needs to be closed. But there are a lot of companies — Databricks being one — that have done a really good job of leveraging open source meaningfully while still having a sustainable business model.

The privacy problem is an audit problem

Demetrios Brinkmann: At the end of the day, if you’re the end user hacking together an app, as much as you love open source — are you going to go out of your way and not use OpenAI?

Kyle Harrison: There are two groups of people who will have very different responses. A lot of people will choose the path of least resistance.

On the one hand you have what represent pretty massive industries — healthcare, financial services, government — highly regulated industries that are very concerned about the implications of their data relationship with OpenAI’s models.

Just recently OpenAI announced ChatGPT Enterprise, and they tout a lot of enterprise-grade security and privacy: SOC 2, data encryption. Those are the checkboxes most companies look for, to say: have you done the bare minimum?

What will be unique about leveraging somebody else’s foundation model is that there’s a difficulty in understanding the attribution that leads to a particular model. OpenAI can say yes, we are SOC 2 compliant, we do have data encryption, we do have these protections. Right — but is there any way to confirm that none of my data has ever been used for you to train and improve GPT-5? That’s pretty hard to audit.

Maybe they’ll argue that of course we have barriers between your data and our models. But it’s super hard to audit, and you can still be SOC 2 compliant and still have data exhaust. If you leverage a model that’s been trained on data, and you didn’t technically use that data to train GPT-5, but GPT-5 can leverage other things you’ve done — there are implications there that are just not easy to say definitively we’ve never used that stuff.

So highly regulated industries are going to be much more careful.

[The show’s running joke testimonial is inserted here — a fictional listener from a bank declining to recommend the podcast.]

The second thing is that one of the limitations in the space right now is a dramatic shortage of technical expertise — human capital. There aren’t enough people, which is why these big companies compete so dramatically. The joke is that an AI researcher can make as much as an NFL quarterback.

But as that talent grows and there are more people who specialize in and understand this stuff, the more companies will pride themselves on building expertise internally and wanting particular capabilities and flexibility. The more evenly distributed that talent gets, the more those companies will be uncomfortable with the enterprise version of OpenAI’s products. They’ll look to build their own capabilities internally.

Which is the same as what happens in open source today. Uber is a really good example — dozens of open source products came from Uber that they used to build a ton of their stack. There will always be companies that say we can build this capability internally, so we want the open source foundation and then we can build on top of it.

Those two groups make up a big chunk of the industry that OpenAI can sell to. The long tail, for sure — it’s easy to just use the thing. But it’s not an open-and-shut case that OpenAI has already won the day.

Demetrios Brinkmann: It’s funny, because that long tail is who we hear the most from on Twitter — those are the Twitter demos. And then when you go into big companies, I’d argue they’re potentially not even using large language models. A lot of them are still doing traditional ML.

One piece worth noting: I have to give credit to John Savage in our community. I was posting that meme showing the guy getting all excited — ooh, all that private data to train GPT-5 on — and John said, I wonder if it’s not that they’re going to train GPT-5 on it, but that they’ll use it as test sets. There’s probably going to be a ton of models they create but never release, that never get outside their walled garden and nobody ever knows about — but your data is used.

Kyle Harrison: I think it’ll follow a very similar curve to what happened in machine learning more broadly with algorithmic input — with Facebook and social and ad delivery. The entire intelligence apparatus around advertising over gen one and gen two of social media.

It’s funny or confusing to people who work in data tracking — the public outcry of wait a minute, you’re using my data to sell me ads? That’s so immoral and disgusting. And it’s like, that’s just the best way to do it; there’s no better way to serve you personalized ads, we need to understand your user data, and you opted into it.

I think you’ll see a similar curve. The long tail is just like, yeah, OpenAI, SOC 2, sick, we’re good, check the box, use the thing, move on. Couple of years from now, the more we come to understand attribution of data and the implications for different models, the more you’ll have a bit of an outcry. It won’t be I didn’t realize my data was used to train that model, because they promised they wouldn’t. It’ll be I didn’t understand that my data exhaust can generate training sets that can then be used in different ways.

We better understand attribution, and then you can say: I can audit this model and say it came from this, which came from this, which came from your data. Surprise. And then you have a cookies/GDPR type of response.

The Microsoft playbook

Kyle Harrison: It’s also why — and I’ve had pushback on this argument — I think OpenAI in particular would love to see this get commoditized. They have a very unique relationship with Microsoft and unique corporate distribution in a way that Anthropic and Cohere don’t; they have some corporate investors, but not the same relationship.

So I’d say this is maybe even uniquely OpenAI: they want general-purpose model use cases to become as commoditized as possible, as quickly as possible. Because if they can get GPT into everything — and it’s already kind of happened. Most of the incumbents who have rolled out AI products are building on GPT or DALL·E.

Replit is a unique exception — they went and figured out their own stack, and they have a great blog post about how they thought about using this from Salesforce and this from different things. But very few companies do that, because they don’t have the technical expertise or the attention or foresight. So most people are just going to use GPT.

And if OpenAI can get it as broadly distributed as possible as quickly as possible, then when you do have that outcry in a couple of years, it’s so much harder to get them out — because they’re everywhere.

It’s the Microsoft playbook. You couldn’t get out if you wanted to. Microsoft Teams did so well not because it’s a better product — in my opinion it’s a much worse product than Slack — but it’s freaking everywhere. It’s like corporate malware.

Demetrios Brinkmann: Oh man, that’s so good. Corporate malware — that is a good one.

Don’t stuff LLMs where they don’t belong

Demetrios Brinkmann: Let’s talk about traditional ML. Your portfolio companies included — Ramp is doing incredible traditional ML. How should these ML 1.0 companies be looking at the ecosystem? Should they be stuffing LLMs into stuff where they don’t need to? It feels like the product manager’s life got a whole lot harder over the last six months, because everybody wants AI in their products.

Kyle Harrison: Every company is unique in the approach it should take.

There’s a quote from Star Wars — only a Sith deals in absolutes — and the irony is that that is itself an absolute. So generalities are very dangerous; the only general rule is that general rules are dangerous.

Any company that tries to say we need to do this instead of this, because that’s the hype, is probably not doing things well. They need to respond — but if that’s their decision-making for how they build a product, just based on this is the hypest thing and we need to put it in so we can put the dot-AI on the end of one of our landing pages, I think that’s dangerous, and probably not a great product organization.

The companies that have done this well — Ramp is one of them — are saying: what are we already doing? What are already our core competencies? And in what ways could this technology be leveraged to continue to improve the core competency we have?

Ramp has done that really well, in understanding their core competency of managing the financial stack, and then leveraging intelligence on top of that — both in terms of being able to ask and answer questions about your finances, and just better understanding the underlying data you’ve offered up to them. That is the right approach.

Anybody who says oh, we’re not doing machine learning anymore, we’re using LLMs — that’s a very aesthetic, surface-level way of doing things. Everybody needs to figure out for themselves what approach will best lend itself to their core competency. That’s how the best products and the best companies will be made.

How far along are we, actually

Demetrios Brinkmann: Do you feel it’s possible we’ve gotten 90% there with AI, but that extra 5 to 10% could be another ten years — and we all sit here talking about the beauty and hype, and then we have another AI winter?

Kyle Harrison: This is one of the reasons people who hear the AGI doomerism think it’s so ridiculous.

If I think about SaaS as a distribution mechanism, or cloud computing — even those are still maybe: SaaS is like 70–80%, cloud computing 50–60%, AI is like maybe 10%. The gap from where we are today — all the things that could be done to make it functionally capable in production — to AGI supercomputers that take over the world, there’s still so much to go.

Clem at Hugging Face makes this joke: a lot of practitioners look at ChatGPT as a product and it’s like AGI — and it’s autocomplete on steroids. There’s still so much we need to do and understand — capabilities, context, tooling, orchestration — before this is ever a seamless enterprise user experience.

One of my favorite writers, Matthew Lynley, who writes a Substack called Supervised, had a great piece a couple of weeks ago rethinking how we define the iPhone moment in AI. It doesn’t mean everything is perfect. It’s when a piece of technology becomes so capable that it can be general-purpose without really massive sophisticated services around it.

Cloud computing is still largely not that. You still have systems architecture built by outsourced providers because it’s so complicated. Even cloud computing arguably hasn’t had that iPhone aha moment where everybody can just spin it up easily. We’re getting close — Replit has built something like this, where they can enable you to spin up hosting for the application you’ve built, an end-to-end experience. So it’s close to I can just click a button and boom, have infrastructure. But it’s still not quite there for sophisticated production use cases.

And so we’re still not there for things we’ve had for 10, 15, 20 years. For AI, we’re so far from that. People say oh, just kidding, AI sucks because ChatGPT’s traffic went down. No — what you’re experiencing is a limitation of the general population’s ability to understand and conceptualize the use case of this thing, and to use it every day in everything they do. Some of it is performance, some of it is psychology, some of it is UI. Is chat the right UI for this stuff? We don’t know. We just started playing around with that.

Demetrios Brinkmann: Some of it feels like that age-old question of whether every problem needs to be solved with tech.

Kyle Harrison: I don’t use my iPhone for every single interaction in my life. When I want to talk to my wife, I talk to my wife. When I want to play with my kids, I play with my kids, or take them to Disneyland. There are still physical experiences — maybe VR someday tries to insert itself into those.

But are there dozens and dozens of things in people’s lives, both professionally and personally, that would be improved by having general-purpose, well-built AI in that experience? Absolutely. If I can say plan me a trip to New York for me and my wife for our tenth anniversary, and it does everything, books it, puts everything where I need it seamlessly — would that experience be better than I’ve got to figure out a hotel, what’s the best things, look on Instagram? For sure.

So there’s still so much more to tackle before we even reach the existential point — the Jurassic Park grade of we’re so busy thinking about could we that we never stopped to think should we. I don’t even know that we’re there yet. There’s a lot of low-hanging fruit we could take off the table with AI that we’re not there yet. And then there’s the broader implication of where that line should be drawn — what does it mean to be human, versus what does it mean to solve technical problems?

Agents

Demetrios Brinkmann: You’ve undoubtedly seen companies pitch you on agents. What are your thoughts?

Kyle Harrison: I think it’s super cool. I’ve seen a ton of really exciting demos from companies building these packaged services.

The analogy breaks down quickly, but there are components of microservices you can look at and say: in what ways are certain problems so well known, defined and replicable that we can package them into an agent and solve that problem with a very specific purpose-built agent? There are a ton of use cases that could fit really well.

Speaking of the phrase of the month — prompt engineering is really interesting, because chat as an interface is actually really difficult. There are components of certain models that are more capable at doing the thing than you are at articulating in written language what you’re trying to accomplish.

I actually think prompt engineering and chat as an interface could spark people being forced to go back and better understand logic and ethos, and how to structure a sentence in a way that articulates an idea — which could be a really interesting side benefit. But in some cases you don’t need to be so good at articulating the thing in language, if the thing is good at doing the job and you know that’s the job you want done. Let’s just package that and get it done, because that’s a much better user experience than forcing me to figure out how to communicate in words the thing I’m trying to accomplish.

Demetrios Brinkmann: Can agents get past demos? We all saw the rise of AutoGPT, but everyone I know who played with it — I couldn’t get it to do anything I wanted, it kept going around in circles.

Kyle Harrison: There are two problems that need to be solved.

One is esoteric but interesting. There’s an idea we leveraged in the deep dive — I’ll get the quote wrong, but it’s basically that the limits of your language are the limits of your reality. Your ability to understand, interact with and articulate what something is — that’s the limitation of how much of your reality you can control.

Even though large language models are very general purpose, they’re still built on a subset of human interactions and human content, built in a very specific way. Take image models, because those feel a bit more intimate and personal. Whether you like Midjourney, DALL·E, Stable Diffusion — it’s actually kind of like a personality test. Like a BuzzFeed quiz. Different models react better to different people who think in different ways. So sometimes it’s not this thing doesn’t work for me — it’s that maybe you don’t have the product whose underlying logic fits your brain and your way of articulating images.

Finding the right thing is really valuable, and that’s tough, because you’re not going to get a broad swath of people to experiment with every single thing — especially when it’s still as clunky as getting onto a Discord server and figuring out how to use it.

The second thing is just performance — being able to use an agent in production for live use cases. Booking trips for me in real time, interacting with my employees in real time. For those to be trusted in production, you have to have a lot more performance.

The one thing these models have not been good at: they’re pretty good at one surface layer of exchange. There’s an API, and that’s the one interaction — I am Notion, and there’s GPT, and Notion’s interface goes through the GPT API to get language, bring it back, produce an output. They’re pretty decent at that.

But if I have to go from me, through a chat, to a model, and then that model has to go through me into my Expedia preferences to book a hotel, and then book a flight across multiple airlines based on my preferences and where I want to go and where the best prices are — that’s six or seven interaction layers that agent has to go through. And I’ve never seen anything that’s good at that.

I’ve seen things that are good within one application — hey, we have this financial model, we want to do things and create things and put them here. Excel is probably the main surface area where I’ve seen people build a lot of this. It’s really good at that, but it’s always staying in one interaction layer. Once it has to start going through multiple applications, I have yet to see anybody who has maneuvered that very well, because the internet is still so complex that going from thing to thing to thing in one seamless path is very difficult.

Demetrios Brinkmann: It reminds me of a conversation we had with Phillip from Honeycomb — if your error rate on the first hop is 1%, then with the amount of hops you’re doing, the likelihood of hitting an error is much higher each time.

Capital is a dangerous drug

Demetrios Brinkmann: Are we going to sit here for the next ten years and try and try, and still be like flying cars — where’s that hoverboard, man?

Kyle Harrison: People who believe there’s an AI spike — tomorrow everything’s going to change, it’s a matter of days or weeks — those hype evangelists are as stupid as the people who think this is nothing, that we’re going into an AI winter and nobody will accomplish anything they promised. I think both are stupid.

The vast majority of technology — the vast majority of everything — follows a very specific curve. The example I always use is e-commerce penetration. It steadily rose as a percentage of retail happening online, up to call it 18–20%. Then COVID happened and it shot up. The people saying this is the new normal, you’re going to get 80% penetration because nobody will ever go back to a store — those people were stupid. But the people who thought this will exhaust everyone and we’ll never buy anything online again, it’s the end of e-commerce — those people are also stupid.

If you look at the curve, since the ’90s it followed a very steady rise, shot up during COVID, and then didn’t drop back down to where it was in 2019 — it dropped back to about where it would be if it had kept following that curve. That is how the vast majority of technology plays out.

AI will continue to progress. The one thing that can accelerate that curve a little bit is excitement — people are excited about building, younger people will specialize, companies will push boundaries. And all of that gets better if we can keep as much of it open as possible. If it’s all happening in the closed garden of one company, it’s so much harder for everybody to make contributions and get excited. The more open we can be, the more we can push those contributions forward — maybe we can’t step-function change how AI progresses, but we can slightly accelerate the pace.

Demetrios Brinkmann: And the amount of money being poured into AI right now by the VC world is going to help drive innovation forward. You’re part of that, right?

Kyle Harrison: Maybe. I’m with you on the excitement generally. Once you get into capital, you get into dangerous territory.

There was a very similar green bubble in the early 2000s, where billions and billions of dollars went into new battery and wind and solar technology. And something like 90-plus percent of it just vaporized — absolute zeros. Huge destruction of capital. That set back a lot of people trying to work on climate tech more broadly, and we’ve only just recently started to come out of that capital-induced winter.

So I actually think excessive amounts of capital can cause a lot more problems. Excitement, yes. Progress, yes. You still want capital. But if you put in 10x the capital that the category really needs, it’s very dangerous — because when that capital gets destroyed, you have a bunch of people who have lost capital, and they are going to actively crap on the excitement. And that makes it harder to raise capital, harder to progress.

Capital is a dangerous drug. You get hooked on it and it’s dangerous.

Demetrios Brinkmann: Look at what happened with crypto.

Kyle Harrison: I was never a huge crypto bull by any stretch, but I think blockchain can be really powerful — there are some use cases that make a ton of sense. But that 2020–2021 crypto fever is going to set the industry back more than it’s going to help it. I think it can still be important technology over the course of time, but that did more harm than good in my opinion. It invited more hucksters that are going to take years to wash out.

Demetrios Brinkmann: Kyle, this has been awesome. I really appreciate you coming on and sharing all about the openness of AI, and how crucial it is not to let this power crowd into one space. So many of us in the MLOps community love open source — whether they’re contributors or just users, that’s a huge thing, and we can get on board with that message.

Kyle Harrison: Thanks for having me. It’s super fun to get to chat, and I’m super excited about how things will play out. It’s always fun to riff on what could happen.

[The episode closes with the show’s community sign-off.]

Connections

The show and the report

MLOps Community · Demetrios Brinkmann — a practitioner audience, which is why this is the most technical of Kyle’s appearances.
The Openness of AI — the Contrary Research report the episode is built around. Co-authored with Brandon and Andriy of Nomic.

The three-bucket frame

Incumbents (Microsoft, Canva, Adobe, Notion) → foundation-model builders → everything else, including the tooling layer and the “wrapper” layer.
Moats: “if OpenAI can roll out your product in a couple of months, you never had a moat to begin with.” First-mover advantage ≠ defensibility.
The rule worth keeping: spike in something that is not what everyone else is racing to commoditize.
Replit — a developer ecosystem that would still be good with no generative-code breakthroughs; Ghostwriter feeds a pre-existing flywheel.
Nomic — Atlas as the spike (visualizing every data point in a model; the Stable Diffusion canvas with its “Kermit the Frog corner”), and GPT4All as its output rather than its business.

Openness and power

Clem Delangue (Hugging Face) — “the number one danger in AI is concentration of power,” from Cerebral Valley and Congressional testimony. Also the source of the autocomplete on steroids line.
The black-box arc: opaque → legible via transformers → deliberately re-closed, because “it’s a lot easier to monetize something that people don’t understand.” Safety as the stated reason, competition as the operative one.
The bad-guys rebuttal: capability always leaks eventually, so “the thing that keeps the bad guys in check is that everybody else has it too.”
Open Source · PyTorch vs TensorFlow — the precedent for Meta as an open-source bastion, and the observation that this time there’s a moral sub-narrative where there wasn’t one before.
Databricks — open source leveraged meaningfully alongside a sustainable model.
Aidan Gomez (Cohere) — Attention Is All You Need co-author; the Contrary Research interview Kyle references.
Incentives over values: Microsoft’s positioning is downstream of Azure; Meta can afford openness precisely because it has no cloud business.

Privacy

ChatGPT Enterprise (launched days before recording) — SOC 2 and encryption as checkboxes, against an unverifiable claim: “is there any way to confirm that none of my data has ever been used to train GPT-5?”
Data exhaust — you can be compliant and still have your data shape a model in ways nobody can currently attribute.
GDPR — the predicted end state: a cookies-style outcry once attribution becomes legible.
Who opts out: regulated industries running locally, and companies with in-house expertise (Uber’s open-source output as precedent). Both gated on the talent shortage.

The playbook

“Corporate malware” — Teams beat Slack on ubiquity, not quality, and Kyle reads OpenAI as running the same play: commoditize fast, get everywhere, and be too embedded to remove when the outcry arrives.

Building well

Ramp — the counterexample to LLM-stuffing: start from the core competency, then ask what the technology extends. Only a Sith deals in absolutes.
Prompt Engineering — chat is hard because the model may be better at the task than you are at describing it; the possible upside is people relearning how to structure an idea.
Agents — good at one interaction layer, untested across six or seven. Compounding error per hop (via Honeycomb’s Phillip on Honeycomb).
Maturity estimates: SaaS 70–80%, cloud 50–60%, AI ~10%; the iPhone moment reframed as general-purpose-without-services.

Capital

“Capital is a dangerous drug.” The early-2000s green bubble vaporized 90%+ and produced a capital-induced winter in climate tech; the 2020–21 crypto fever “did more harm than good… invited more hucksters that are going to take years to wash out.”
The e-commerce penetration curve as the model for how technology actually adopts — COVID spike, then reversion to the trend line, not to the old level.
This thread runs directly into Capital Inferno Heaven (April 2024) and, much later, TBPN — May 1, 2026’s carpetbagger VCs warning about defense tech.