Nat Friedman Leads $15M in Superintelligence Insurance Company | AIUC

[00:00] No one really today knows how these AI agents fail. You might have seen the OpenAI launched their chat GPT agent, and Sam Oldman on stage puts out a big caveat, like, "Don't use it for anything important." We don't yet know how this works. Certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built the standard AIU-C1, [00:19] that outlines what are all the things that can go wrong with an AI agent. And for each of those risks, what are the things an AI company should do to mitigate that risk?

We are coming out of stealth, having raised a $14 million seed round from Nat Friedman, Emergence and Terrain. Your investors are super ingrained into this space, especially Nat Friedman. I mean, he's now a part of the super intelligence team at Meta. What kind of advisement are you getting from them? Because you're an underwriting company that does insinuate you're a financial institution, how do you put dollars against this? [00:49] break down what is a kill switch in AI? - It might be what's called a loss of control scenario, where basically the AI agent becomes so agentic that it starts to take action that we no longer want.

So a kill switch is like, [01:02] you. [01:11] Runa, welcome to Sorcery. [01:13] Thank you. Thanks for having me. I think we have some big news today. We do. [01:17] Can you share this big news? We are coming out of stealth, having raised a $14 million seed round from Nat Friedman, Emergence, and Terrain. [01:25] Wow, this is big. How big was the round? What was the process like? It was $14 million. [01:32] It was a very quick process. [01:33] We got an inbound term sheet preempted. [01:38] And that prompted us to look up.

I knew I wanted to work with Nat, got introduced to him. [01:43] A week later, we had a temp sheet and we got it done. [01:47] That simple. [01:48] Yep, pretty much. [01:49] OK, given that, I think we should understand more about how we got here, how it became that simple. Yep. So what exactly is your background? I know you're very impressive. Most relevant to this conversation, I started working in AI when I joined Anthropic. [02:03] back in early 2022 as the first person to work on product and go to market.

This is before ChatTBT. [02:10] Is there that moment in time where it just become obvious that something radical was happening here, but the world just did not know yet? [02:18] And I joined Anthropic to help think through like, [02:21] Where is the value going to accrue? How does Anthropix sit into that? How are businesses and individuals going to interact with this AI? [02:28] So now you're working on the artificial intelligence underwriting company. That's right. [02:34] Bit of a mouthful. [02:35] Fit. [02:36] Of a mouthful. Very intense. Super fucking intense, yes.

Yeah. We insure and certify AI agents. So what does that mean? Basically... [02:47] No one really today knows how these agents fail. You might have seen actually yesterday, OpenAI launched their chat GPT agent. And Sam Oldman on the stage puts out a big caveat, like, don't use it for anything important. We don't yet know how this works. [03:01] If you're ordering a burrito and you're not that hungry... [03:03] go ahead and use this is an awesome tool um [03:06] But when you start putting this in meaningful context, if you run a hospital or a bank or any kind of big business with real customers, no one really knows...

[03:15] whether they can trust any particular agent. And certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built a standard, AIU-C1, that outlines what are all the things that can go wrong with an AI agent. [03:29] And for each of those risks, what are the things an AI company should do? [03:32] to mitigate that risk. [03:34] And spilling that out in enough detail across hallucinations, jailbreaks, all these terms that people have heard about, that you can have an independent third party go in and audit a company and test that agent to see how likely it is to fail.

And if you do well, you pass the certificate. The insurance comes on top of that. If you meet the certificate... [03:54] you may want to go a step further in taking risk off the table from your customers and ensure them in case it fails. So if it hallucinates a refund policy, if it spews Nazi propaganda, if it leaks your customers' data, if it discriminates against the people you're hiring, [04:10] the enterprise is going to be on the hook. [04:13] And so everyone's asking, who's going to pay [04:15] And that's where insurance can step in and help pay the bill.

How did you come to this model? [04:20] Yeah, so... [04:21] It started with a, like a, [04:24] deep trip into the rabbit hole and like, [04:26] throughout history how have markets solved these collective action problems and the [04:32] The most interesting example starts back in 1752, [04:36] Ben Franklin. [04:37] All-American hero. [04:40] He was living in Philadelphia at the time. Philadelphia's population was growing 10x over that century. And the house was just getting closer and closer. And they were just burning down frequently. So the whole city would burn down. And Ben Franklin wanted to solve that problem.

So he started an insurance company, the first fire insurance company. And all of a sudden... [04:59] He was on the hook financially. [05:01] when houses burnt down. So he had a direct financial incentive. [05:04] to go and reduce fire risk. So they asked themselves, how are we going to make sure that all the houses are built in a way that makes them less likely to burn down? They built the first building codes saying, oh, how far should houses be apart? Can we make sure to not build houses under trees that are likely to catch lightning?

[05:20] And then they wanted to make sure that people obviously met these standards. So they built the... [05:25] playbook, that probably wasn't the term he used, but the playbook for fire inspections, really the audit mechanism. So this triangle between insurance standards and audits was something that he stumbled into, called the incentive flywheel. [05:38] And that then shows up again and again throughout history. So when the light bulb came out starting around the year 1900, light bulbs also led to the houses burning down. The insurers again were like, we don't want to pay for this.

So if you look at any of these light bulbs, they will have a little UL certificate, the underwriters laboratory funded by the insurers that like, [05:58] specified how do you need to test light bulbs and toasters to make sure they don't burn down [06:04] Later on, Cass... [06:07] killed a lot of people as they were getting really like everyone needed their own car post-world war ii [06:12] And it was the insurers that were taking the charge on the [06:17] looking at the data for what's causing accidents, figuring out what are the interventions that we need?

What is the kind of [06:24] car crash testing we need to incentivize the car manufacturers to build safer cars. And a lot of their research is what then eventually became embedded into laws like seat belt laws, airbag adoption. Before that was law, that was often incentivized by the insurers. You get cheaper insurance if you have a car with an airbag. So that model of insurance, building the standards and running the audits to the [06:47] both build confidence in technology. [06:50] and incentivize security. [06:53] is really battle-tested. And that's what we think is missing when it comes to AI.

I think it would be good to take a step back, [07:00] AI has evolved so fast. Like, we really weren't talking about agents and, like, a tangible aspect until maybe a couple months ago. Like, now there's real applications and usability with it. So... [07:14] I'd love to get your perspective of the evolution of AI and AI safety. Some companies baked it in from the start. Some just progressed and were all gas, no brakes. So what was that evolution like? Yeah, I think. [07:29] When you started all the way back in like GPT-2, it's kind of obvious what was coming.

Like already at GPT-2, you could start to see what GPT-3, 4, the agents, what are they going to look like? And there are kind of two schools. One was like, there's no risk today. GPT-2 was like literally could not spell its own name. So like there's no risk. Therefore, all gas no brakes. And then there's another school that's like, we can obviously see this is going to be a really big deal. [07:59] able to help people [08:02] They're going to be able to help create bio weapons. They're going to be able to do all these kind of things.

And we need to think about that in advance. And that debate kind of played out for a long time. [08:11] without really any evidence either way, because it's just too early, as the models were just getting better and better. And now we're getting to this point where, [08:19] everyone agrees pretty much that this is getting pretty serious so you have [08:25] Companies like Anthropic have always been in the camp, like, hey, this is really risky. Companies like OpenAI have gone a little bit back and forth on what they think. And with their release yesterday, they're saying like, wow, now the risk levels are really high.

They are proactively sharing that with the world, that they're now like enhancing the security levels. So I think we thankfully haven't had the... [08:42] big incidents yet, but everyone's agreeing that now they're sufficiently good that they can do real damage. [08:49] When the models are not good, they're not going to do any damage. But as they get really good, we're going to delegate real work to them, real task, real critical systems. [08:56] and [08:57] No one has sufficiently good answers to say, oh, there's no risk here. [09:01] Right. So there's companies like, I call it, [09:05] Super safe, super intelligence.

I don't know why. Extra double safe. I just add more supers to it. Yes. There's companies like that. Ilya takes it super seriously. Yep. [09:16] Again, super. And then... [09:19] With Anthropic, maybe we could start with Anthropic and then go to SSI. They have different levels of safety. So what do those levels mean? And could you break those down? Yeah, so they call them security levels, security level one, two, three. Really, it's a process of saying when models get sufficiently good and therefore sufficiently dangerous, we need to match that with the levels of security we're implementing.

What are the level of tests we're going to run? What is the level of cybersecurity protection we need to have in place? [09:49] that match each of these security levels. And so we're slowly climbing through the temperatures now. [09:56] You were talking about just before we got on here about whether security level three is going to be a thing. And I think the short answer is, yeah, it's just a question of time. And is it going to be this side of Christmas or after? I don't know yet. But it's really like...

[10:07] companies ways of [10:09] matching proportionally. [10:11] the risk with the mitigations. [10:13] Mm-hmm. [10:14] And then what about super safe super intelligence? They've spoken much less publicly about it and also... [10:20] They don't have a, their whole strategy is one shot to save superintelligence. So they haven't put out a product yet. And so therefore a bunch of the risks that come when you put the product out in the world, they don't have to deal with. There's still some risk when you're building it. [10:32] as you're building it in-house. But these things are much more relevant to Anthropic and OpenAI, where you have thousands, millions of people using this every day.

[10:39] We're assuming some companies have that. [10:42] We also assume a lot of companies don't have that. One of the largest companies is telling people, caution, watch out, don't put your sensitive data in it. That's where you come in. So could you explain a little bit further on AIUC? [10:58] Right? I'll use the acronym. There you go. What we do? Yeah, yeah. Where do you come into play and how do you start integrating with partners? Yeah, so the starting point for all this is... [11:08] If you're a... [11:09] Business... [11:11] looking to adopt AI, you face a hard choice.

On the one hand, if you're a bit nervous, you face that your competitors are going to basically make you irrelevant by moving ahead. Or you can lean into this new technology wave, as everyone's saying they're doing, and then you risk making headlines. And so we talked about some of the examples before of like hallucinated refund policies, brand disasters happening, data leakages, et cetera. The question they're all trying to [11:39] Where we come in is creating the confidence infrastructure between the AI companies and the people that are adopting this. So we work with the AI companies to earn the confidence of their customers by saying, hey, these are all the things you guys need to pay attention to.

[11:54] helping them meet that standard, these best practices for what businesses really need to have confidence in it. And then as an extra layer of protection saying, if something goes wrong, [12:03] we will pick up the bill so if you work backwards from that conversation between an air company and enterprise [12:09] It's really like how can you equip their companies to say the most... [12:14] confidence building things in a sales in a sales meeting [12:18] And typically the problem they have is that [12:21] They all want to say how responsible they are.

[12:23] But enterprises look at them and be like, we obviously can't trust you. When it comes to financial products, you obviously have third-party auditors. Even in cybersecurity today, if you get a pen test, [12:33] You can't do that yourself. No one's going to trust you to grade your own homework. So you need a third party that can basically say whether you're up to scratch. That's the role we play. We're the independent third party that can kind of broker the confidence between them. [12:44] You mentioned your SOC 2 for AI agents.

For people who aren't familiar with SOC 2, can you break that down a bit? Yeah. So SOC 2 is a security standard that says... [12:55] If you're selling a SaaS product, what are the risks that you need to look out for? Things like privacy and security. And what are the things that every company must meet to get SOC 2 certified? And SOC 2 certified is basically like a gold star you can put on your shirt when you go and you meet your customers and say, hey, we meet all these best practices that we know that you'll need to be able to adopt this.

[13:15] Because of course, enterprises have real security requirements. And SOC 2 is just a way of summarizing that into one kind of trusted badge. There's a question about like how trustworthy it is, but the purpose is really to broker trust between software companies and their buyers. [13:28] But Sock 2... [13:29] doesn't speak to all these new concerns that are coming up with AI agents. It doesn't say anything about what are you doing with my training data? What are you doing with my data as it comes to your training models? [13:40] How are you thinking about measuring hallucinations?

Oh, we read this blog post about jailbreaks. [13:46] Are you guys protected against that? What mitigations do you have in place? Do you have a plan when your AI agent inevitably fucks up? All of these kind of new questions suck to a silent on. And so... [13:57] we've seen a lot of demand, both from the AI companies, but also from the enterprises being like, who can create this like, [14:04] language for what are the risks that really matter and actually like [14:07] down to the level of a code base, what do you need to do [14:10] to mitigate that risk.

[14:12] how do you go down to the code base for this? Because, you know, [14:16] Things are moving so fast. How do you model this out? Core of the standard is we actually just have to test things. [14:24] how the agents perform in real life settings so if you're worried about [14:29] If you want to know whether an AI agent can get jailbroken, the best way to figure out is to try and jailbreak it. [14:35] and [14:37] The field is moving really fast. So what's really important is that you build this connection between the latest research papers, [14:43] And what is standard?

[14:45] does and the kind of testing that's required. So where some of the old standards get updated every 5, 10 years, we're going to update our standard, AIU-C1, every three months to make sure that as new kind of threat vectors come out, as new capabilities come out, that the testing that's being done [15:00] reflects what's possible. Because you're an underwriting company, that does insinuate you're a financial institution. Right. [15:06] How do you put dollars against this when you're underwriting? Like, what are you underwriting and how do you actually cost that out?

People are probably looking at AI, of course, as a trillion dollar industry. And so if these companies are growing so fast and are, you know, like $10 billion, now there are hundreds of billions of dollars. They're approaching a trillion dollars in value. How do you underwrite this? Yes. The first thing is, what are the risks that you want to cover? [15:36] confidence infrastructure between AI companies and businesses. You start from what are the things, what are the risks that are keeping the business up at night and are holding up adoption?

And those are the things we talked about before, hallucinations, jailbreaks. Are you going to mess up the refunds? Are you going to infringe IP, et cetera, et cetera? [15:53] For each of those risks, you have to figure out what is the, and insurers think about this in terms of what is the frequency of the risk. [15:59] How often does it fuck up? And what's the severity? How bad is it when that happens? The core of underwriting these systems is testing. [16:07] really getting a robust sense [16:10] of any particular agent, how likely is it to fail?

And you can only do that by like, [16:14] just getting it to fail thousands and thousands of times over. And then... [16:20] you want to connect that with [16:22] Some of these things are actually not new. So what is the cost of a brand disaster? [16:27] Well, brand disasters are not new. Maybe AI-generated brand disasters are new, but we have tons of data on what does it actually cost. And so connecting these two up is the core to underwriting it. And then, frankly, there's going to be some trial and error [16:38] you, this field is moving.

[16:42] there are gonna be mess ups that no one will have predicted so it's also a way of like how do you structure the risk so that you don't lose the skin on your nose when that happens [16:50] Back to the amounts of, you say, like trillions of dollars. We obviously don't have a, it's not a $14 trillion seed round we've raised. Is there, this is also insinuating there's an AI reinsurance company. [17:02] Yes. Coming up next, next podcast. In the first instance, we work with existing insurers. Okay. So when we help companies get insured, ultimately...

[17:15] The guarantee of the money comes from established insurers, some of the oldest ones in the industry that have the balance sheets to provide this. What... [17:23] these insurance companies are missing is [17:26] Frankly, they don't know how AI works. [17:28] They don't really understand the risk and they don't know how to run these tests. So they bring the balance sheets and some of the rigor in how to think about the risk. And we bring the technical capabilities to saying like, [17:39] what do we think the risk is here? What is the kind of data?

What is the kind of monitoring that's required here? And when you put that together, that's what unlocks this industry. You've mentioned a couple scenarios, a couple use cases. I mean, there was the whole Gemini situation early on with their image generation product. There was the Nazi thing with Grok. There's like all these instances for that, but I'm not sure many people have talked about the enterprise level of this and what kind of data is being affected at the enterprise. So could [18:09] into that a little bit further. Yeah. They're actually...

[18:12] kind of related. I think what those examples are showing is that it's actually really kind of hard to build a powerful model [18:21] that is never going to say something outrageous. [18:24] And so if you're an enterprise and you're deploying a software, [18:30] a customer support chatbot, for example, [18:33] builds on top of those models, someone is going to be able to get [18:37] your customer service chatbot to say something outrageous. [18:41] And so that's something that keeps him up at night. [18:44] You'll have seen... [18:46] Lots of people have questions about like, oh, is OpenAI training on my data?

[18:50] Great. Enterprises have... [18:53] teams dedicated to privacy that all they're doing is checking what is happening with our customers data and so they have the same questions like what is actually how are you training on our data um [19:04] Data leakage is a big problem. You've seen some of these chatbots leaking other customers' data. So all of these kind of [19:12] You might think of them as kind of mundane concerns, but they just really matter for business. Like a bank and a hospital makes really strong promises to their customers, and it has to flow through the entire chain.

So before they can work with an AI company, they need to know that those promises are going to be met. [19:26] That's the beginning of that answer. [19:28] I'm curious what the state of adoption is in enterprise, if you have any sort of like inside look onto that. I've heard rumors about, you know, the AI legal space and there might be huge high flying companies that rhyme with Schmarvi, but, you know, not many companies. [19:45] actual lawyers are using it in-house. So like implementation inside is low, but adoption is really spread across.

How deep is AI like actually implemented within the enterprise and how long do you think that's going to take? Yeah. So we're transitioning from this phase of [20:02] Everyone for a couple of years have known that they must say to the world that they're using AI for everything. [20:06] Every board would have said, like, we have an AI mandate. We're going to... [20:10] adopt or die. And so that means everyone talking a big game about how they're using AI. In practice, there have been a lot of [20:17] proof of concepts, [20:18] sandbox, enterprise deployments, internal deployments, kind of in low-stakes scenarios.

And that's enough that you can say you're using AI. But there's still a gap for this too. When you look at what is the number of lines of code that are being contributed to JP Morgan's code base, that's generated by AI. [20:33] As of very recently, that's extremely low. It's now starting to happen. We're starting to make our way into real use cases, both hospitals adopting this, banks adopting this, and business adopting this. But it feels like it's really in the last six months that you're moving from toy projects [20:50] to kind of real adoption.

[20:52] But still, that's just the frontier companies that are... [20:55] And [20:57] Like when you look at the long tail of businesses, they're still like... [21:00] at [21:01] still lots of the world runs on paper, right? Like there's still a long way to go. [21:06] Sorcery is brought to you by Brex, the financial stack trusted by more than 30,000 companies, including one in three venture-backed startups in the S. Nearly 40% of startups fail because they run out of cash. Brex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle, chipping away at it with fees, Brex is designed to help you spend smarter and move faster.

[21:35] powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one. But same day and even same hour liquidity. Access your funds anytime. Companies like Scale AI, DoorDash, Service Titan, HIMS, Anthropic, Flexport, Robinhood, and Plaid trust and use Brex. [22:05] As we think about superintelligence, superintelligence has become like the next word. I feel like we've already skipped over AGI. AGI is out. We're over it. We were tired of hearing about it. Now we've skipped to superintelligence.

Another word we also don't know exactly what it means, but it sounds really important. Sounds really important. So what do you think about superintelligence? Let's start from what it is. [22:33] I think... [22:35] AGI was kind of like... [22:37] Can we build... [22:39] a system that's as smart as a human across all the domains. [22:42] And everyone was like, "That's the milestone of the future. When that happens, everything's gonna change overnight." [22:48] And now we're kind of in this situation where we have systems that, [22:51] They still fail in a number of dumb ways.

[22:54] But it's starting to, like, they're really, gosh, they're really smart. They're really good at coding. They're really good at science. They're really, like... [22:59] getting pretty good at common sense. They're not very good at robotics. They're not very good at a number of things. But in a lot of ways, when you talk to something like O3, [23:07] from OpenAI, you're like, [23:10] this one is just a lot smarter than me. And so in some ways, we have AGI now, [23:16] And yet the world hasn't completely changed. So then what's next?

And super intelligence is like this idea of like, [23:24] Well, if you plot... [23:26] very roughly, AI in a curve over time, how is the IQ going up? [23:31] We have now like hit... [23:33] human level for a lot of use cases, but it's not going to stop here. Now you can use AI to build AI. Great. This is just going to keep going up. And so superintelligence is just the idea that you will have something that is like radically smarter than us, where we look like [23:48] what monkeys look like to us.

And so far, it's a little bit hard to pin down reasons why that's not going to happen and why that's not going to happen in like... [23:56] within a decade. And that seems like the most important fact. You were an early employee at Anthropic. I'm sure you're still friends with people there and you're very connected to community. Are there like actual concerns on the inside about the rapid adoption or I guess the rapid advancement? [24:13] of these models. I mean, clearly you're building the company in here, but I'm curious about the other externalities around it, whether it's like jobs or...

[24:21] Yeah, I guess let's start with jobs. [24:24] Yes. Short answer. I think... [24:27] Everyone who's building these models is in a kind of a dual state where everyone's like, wow, this is just so exciting. We're just making new breakthroughs. This is going to advance science. This is going to allow us to cure cancer, short order. Who does not want that? And the field is just moving so fast. And so it's just like an electric space. And at the same time, everyone's also like, wow, this is just deeply uncharted territory, both at the model level, but also at the societal level.

And... [24:54] Just no one has good answers for what's going to happen to employment. Everyone is just doing their best. [25:00] guessing really and so those are those are real concerns and i think a lot of people are kind of responding to that is like [25:06] Everyone had those concerns when, like, back when the Industrial Revolution was going to happen. It's like, what are going to happen to all the jobs? The computers came out. Isn't that going to eradicate all the jobs? And it turns out it didn't. The question is, is this time different?

I think there's really good reasons to think that this time is really different. And in particular, because... [25:24] When you look at the marketing claims of all the aging companies, [25:28] the value proposition is we're going to do the work for you. So everyone's like, well, so what's going to be left? And so I think that's just a very real risk. Anthropic... [25:39] have made some pretty bold statements as to what they think is going to happen to white collar jobs i think they're like i can't remember the details but like by the end of next year you're gonna you're gonna have models that can automate something like 50 of all white collar jobs and that's like [25:50] a terrifying prospect.

You're now starting to see Congress waking up and using terms like AGI. They haven't made it to superintelligence yet, but that seems to be about next month. Yeah. Thinking about, this is going to just rip the social contract. And that is not a problem that the model companies are going to solve. [26:09] Yeah. [26:10] I was actually just at the RAISE AI conference in Paris. It was amazing. Like everyone was there. I don't know if you were there, but maybe you were. Nope. There was no sign of super intelligence talk there either.

And so I was like kind of joking. It's funny about your joke about Congress, but I was joking. I tweeted like degrowth is real. We're not even talking about super intelligence in Europe. They're way behind. Absolutely. I mean, I grew up in Denmark. This is... [26:38] Yeah, they live in a... [26:41] Most of them live in a different world. You'll see a lot of... [26:45] commentary based on like people who tried [26:47] their first version of dbt4 and they're just confident that this thing could not do math yeah and [26:53] it takes about 15 seconds on a mobile phone to be like, no, it can really do math.

And at the same time, I also have a lot of respect for, like, if you... [27:03] build products with these models every day. It's also hard to believe that AGI is going to happen by the end of 2026. The super intelligence, like it's actually still kind of hard to get some of these things to work and practice as reliable as you need it to for this to like... [27:17] I don't know. [27:19] scheduled doctor's appointments with the level of precision like life and death scenario and so i think it's like [27:24] actually really useful to [27:26] embody both of those perspectives both like wow the real world is really freaking complicated and the last mile really matters and like the models can't do it on their own yet so there's a lot of [27:38] hold in mind like oh two years ago [27:41] thinking about whether we could like get [27:43] the precision that's required to base life and death decisions on it wasn't even a question.

[27:48] It was terrible. I couldn't write marketing copy. Yeah. I was really fucking awesome with marketing copy. And it's a real question about whether it can run tasks that life and death depends upon. And that is just, [27:58] insanely fast um [28:00] And so you need to hold both of those perspectives in at once. [28:03] In today's high-speed business world, staying ahead means using the smartest tools possible, including the powerful capabilities of artificial intelligence. Meet Turing Intelligence. Turing builds customizable AI systems designed to solve your mission-critical challenges, no matter your industry.

From expert guidance to tailored projects, Turing helps top companies realize AI that's more capable, more adaptable, and more effective. With Turing, discover how AI [28:33] What are the craziest things that has happened recently that gives it more tangibility against like, oh no, this is really moving super fast is Google's launch of VO3. Because now we're seeing what deepfakes like actually look like. Like these videos are, they're crazy. [28:56] hilarious. Some of them are just so scary. Yes. Then I think what was that show? I think Mountain Head came out. And and so like they were kind of like modeling the premise on the story, like this big tech CEO put out this like deep fake generator for the social media platform.

And then all of a sudden, like war breaks out, like terrorism, like all this kind of stuff, like [29:19] rips the earth and shatters it. And so... [29:23] I think like [29:24] once we kind of saw and we are, you know, we're creatures where until something bad happens, we'll really react to it. We're not. [29:34] as proactive as some people would like. I mean, there are definitely some more cautious people out there, but I think like that instance made it much more tangible. I think like the Grok 4 release made it more tangible because that was like a huge leap in performance.

From your perspective, do you think... [29:53] Talking back to the government layer of this, do you think AI regulation will become federal law this year? CalShare has a market on this, and it's like, I think the last time I checked, it was like 25%. Like, there's a 25% chance right now on the markets. Yes. Change your question directly. I think there's not going to be federal law this year. [30:13] But it's worth noting that [30:15] There's about a thousand... [30:16] state bills in the US, so not federal but state level, that [30:21] regulates ai and or is proposed to regulate ai in some level so [30:28] The public consciousness...

[30:30] has really caught on to AI because actually deepfakes are just quite visceral. Like it doesn't feel that futuristic. Like you can just imagine that happening to your daughter or your cousin. And that is happening, right? You're now starting to see people lose their minds to AI. And... [30:48] So the regulatory machinery is actually working. I'm just not sure it's going to happen at the federal level. That's working really fast. And... [30:56] There's right now a debate between [30:59] the [31:00] All gas, no brakes. [31:03] which is basically just relying on the company, their company's voluntary commitments, saying, like, we're going to be really responsible.

And then the top-down legislation, [31:13] Which of these is right? [31:15] We think both of them have a space, but actually... [31:18] the challenge is because AI is moving so fast, [31:22] the chance that the regulators are going to get it right [31:24] It's just really quite small. Take EUAI Act. [31:28] starting with like a regulatory first approach they started drafting that in 2021 [31:33] It's a long time ago in AI world. And just as it's about to be implemented, [31:37] there's now concerns about like, oh, is it actually fit for purpose?

Do we need to like redraft it? And so, [31:42] you [31:43] the pace is really slow. [31:45] And we're proposing a third way, somewhere in between, for markets to come in and regulate. And that's throughout history. [31:52] um [31:53] When you look back at like early fire insurance, when you look at when the light bulb came out and lots of the houses were burning down, when you look at early cars were really quite dangerous. [32:03] Often the market stepped in before regulation to try and figure out like, [32:10] how do we actually prevent this?

That was often led by the insurers, because the important thing about insurers is they pay the bill when bad stuff happens. And so they are directly incentivized to figure out what are the standards that are required for people to build safe cars or build houses that are not going to burn down. And how do we enforce that? [32:29] before we give people insurance policies. We're proposing that the market has a big role to play [32:35] uh, [32:36] because it's going to be able to adapt [32:38] a lot faster and [32:41] it's going to enable faster adoption rather than slower adoption.

So the EA Act is definitely not created to like, [32:47] speed up European AI adoption. It's focused really on downside risk and we think in a world where the US is competing on a global scene with China, [32:57] you really needed to both be fast and secure. [33:00] We think the market has a lot to offer there. [33:02] Yeah, I think it's clear the S. is not going to... [33:06] Hold back on this one, especially because it's we're we're on a global stage here. To your point, it really comes down to like this is the new.

[33:14] race. Yep. The AI race is the next one. Yep. Um, [33:18] I'm actually attending. This might come out on the same day, but winning the AI race, it's being put on by the All In Summit. Yep. Well, the All In podcast and the Hill and Valley group. So that is a main question. And I think it's like coming together and making sure that we can win this and that we have the right tools to do it. [33:48] but like we need the insurance. [33:50] to make sure if something slips up, [33:52] someone's accountable.

Yeah, because... [33:54] If all you want is to move really fast, [33:57] you still have to look out for the kind of disasters that can set the industry back. So take a look at nuclear. Nuclear is set back by at least a generation by, of course, both Chernobyl, but also the Three Mile Island in 1979 in the, where you give... [34:13] the public and politicians have reason to shut down because if you're not doing it responsibly, [34:18] You shouldn't be trusted to do it. [34:20] So we think investments in security is like, [34:23] required for progress to be fast.

[34:26] And... [34:27] If you want progress to be fast, you need the businesses to adopt it fast. That's where all the revenue comes from. And... [34:33] you need to earn their confidence. So that confidence infrastructure is required. And so if you're all brakes, no gas, you should want businesses to be able to move forward with confidence. If you're a business, [34:42] a doomer, you should also want well-incentivized market players to try and put a number on what is this risk and drive adoption of these best practices that reduces the risk.

[34:52] The alternative is also a kill switch. Yes. Right? Yes. Calci has a market on this. So I think Calci's market, last time I checked, it was like 5% to 8%. I think it's likely that we'll have a kill switch. By end of year? I'm not sure what the range is. It might have been, I'm not sure. We can check. But could you break down what is a kill switch in AI? Yeah. [35:14] Yeah. So a kill switch is like... [35:17] Does anyone anywhere have a button [35:21] that they can press, if something really bad happens, that shuts off AI.

And this is really interesting when... [35:29] One of the... [35:30] concerns that's coming up now that AI is getting really good and getting really agentic is that [35:36] There might be what's called a loss of control scenario where basically the AI agent becomes so agentic, [35:43] that it starts to take action that we no longer want. It might copy itself onto servers that we can no longer control. And all of a sudden... [35:51] humanity stands with a big disaster on our hands. And so a kill switch is like, can we [35:55] in all deployment scenarios, retain a button, that big red button we can press to make this go away.

[36:00] And maybe an analogy is like, [36:03] In any given building... [36:04] You're going to run a fire, what's it called, a fire drill, where everyone exits the building. You cannot run a company where people can't exit the building pretty fast if something really bad happens. And that's the same thing here. [36:17] Whether that's going to happen by any of you, I don't think so, but... [36:20] It's... [36:22] It's growing an awareness, even in this administration, you see... [36:26] David Sachs articulating, like these kind of catastrophic scenarios, like, [36:31] are not outside of the realm of possibility.

[36:34] And you need to plan for it. You actually also hear Chinese government taking, like, using the term kill switch pretty seriously and thinking, like, you cannot develop AGI without knowing what the kill switch is. Technically, it's pretty hard. [36:47] to get right, such that you in all deployment scenarios have this kill switch. But that's the kind of thing that would let us have the confidence to move a little bit faster. [36:55] if we knew that there was a big red button, if things were going wrong. [36:58] For national security, it would also...

[37:01] make sense to have a kill switch because if we have adversaries that deploy their AI, whether it's through robotics or through cyber attacks, you would want to know how to shut off [37:11] their systems. Yeah, unfortunately, I think when people are thinking about kill switches, no one is planning to build a kill switch system. [37:18] that the US would not allow China to push our kill switch. And so I think it's only going to be for your own AI. It's really focused on like, [37:25] Is our AI going rogue?

[37:27] Thank you. [37:28] But when you're thinking about a military scenario, [37:31] Unfortunately, the kill switch... [37:34] may not quite help us. [37:37] Yeah. [37:38] I'm coming from a defense tech manufacturing conference. It's top of mind. I'm like looking at all of these one-to-many drone deterrence systems like Epirus. I'm like, how do we have that against other countries? It's deployed. Before we were talking about real life scenarios, and maybe this is an extreme, but you mentioned the Terminator. I was talking about [38:08] robot. So could you explain that a bit?

Yeah, the... [38:12] and [38:13] The kind of scenario that is keeping a bunch of people up at night is... [38:18] Kind of taking the current AI development to its... [38:21] perhaps a logical conclusion and being like, well, they sure seem to becoming more agentic. They seem to be able to take tasks, be able to... [38:30] do task effectively that are longer and longer and might take a human a day to do or a month to do. And while all these science fiction... [38:39] movies depict as a scenario where at some point, [38:44] the interest of the AI.

[38:46] starts to diverge from humanity and then we're in real trouble um [38:51] Today, that remains, thankfully, mostly science fiction and... [38:57] It's a little bit up to your general attitude on risk, whether you think we're seeing all the signs that this is literally going to happen within the next decade or whether... [39:07] AI is mostly just [39:09] mimicking... [39:11] the fact that it's Red Eye Robot. So if you ask it, what's your plan to take over humanity? It will [39:17] It has a plan because it's read all the science fiction, but actually that doesn't correspond to having a plan or having any intention to do it.

It's just the parrot of the training data. [39:26] Jury's still out. [39:28] Jury's still out. Who knows what's going to happen. [39:30] As we kind of wrap up the conversation on safety in particular, I think, [39:37] Do you think that all of these forces will converge into one dominant standard for AI safety? Or do you think it'll be fragmented? I think it's going to... [39:47] merge into one dominant standard so what you often see is [39:51] When... [39:52] Standards serve the purpose of creating confidence, whether that's in financial markets with something like Moody's or ICO, or even in security where you have things like SOC 2.

Often one or two emerge that everyone agrees that that is the thing we're looking to. I think that's going to happen here. I think it's going to happen both at the application layer. It's going to happen at the model layer. It's going to happen at the data center layer, specifying what are these security levels that are meaningful. [40:21] And then I think... [40:23] Frankly, this is also... [40:26] a geopolitical question of [40:28] Is global AI going to be built on American standards or on Chinese standards or something else? You see both...

You see China... [40:36] investing a lot in [40:38] adding people to the international standards organization, ISO, to start to influence standards. You see private companies like Huawei, [40:49] really taking the lead on things like 5G. And of course, a lot of the world is running on 5G. And so [40:55] I know a lot of people in the White House are also thinking about, is AI going to be built [40:59] based on American standards, or is it going to be Chinese standards? Is it going to be European standards with the UAI Act?

And so I suspect that that's also going to be a bit of a winner-take-all dynamic. And I suspect there's going to be a lot of interest in making sure that AI is built on American standards. [41:14] I think we're winning, so... [41:16] Well, I think America is winning on AI as it stands, but the lead is really shrinking. [41:24] And when it comes to standards, for better or for worse, the debate that's happening right now between companies is like, [41:31] What do we think about the EU AI Act? [41:34] And the US doesn't...

[41:36] have a standard that can quote unquote compete or complement it and so [41:41] From that perspective, it's not obvious that the S. is winning now. I think the S. has a lot of interest, and I think the S. has all the things to make that happen. But at the moment, it's not obvious that that's happening. [41:52] And at the moment, it's less obvious today than it was a year ago that... [41:57] at the model layer that the US is going to be [42:00] way ahead. Love premium merch just as much as we do?

That's why Sorcery uses Fourth Wall. Everyone from creators like Marcus Brownlee to podcasts like Acquired to Orgs like the Smithsonian Institute are using Fourth Wall. They let you create and sell premium products without having to stress the details. They handle everything from production, shipping, customer support, taxes, even giveaways. When it's time to level up and make gear that people are actually proud to wear, [42:30] Wall for all of our brandware at Sorcery since day one. Use my link in the description to get free credits for your first order. And for any VCs, DM me on X and I can get all your portfolio companies set up with a free samples credit deal.

Your investors are super ingrained into this space, especially Nat Freeman. I mean, he's now a part of the super intelligence team at Meta. What kind of advisement are you getting from them? How close are you? And like, what is the relationship? [42:59] between you and your investors. Yeah. [43:02] We're really close. They're just... [43:04] Like Nat is just the, I would say, the preeminent AI investor. [43:09] both made a lot of great investments, but also, as you just mentioned, is very close to the metal as to how this is actually being built.

Both the small questions of how to build AI, but also some of the big questions. [43:21] Nat himself spent a lot of time on [43:24] some of the risk questions that we were dealing with when he was leading github they built go pilot which is like the first real ai product that worked and [43:33] he led the efforts that led to Microsoft. [43:38] basically ensuring all of their customers against IP risk to say, oh, they're really worried about IP. We're just going to step in. And if they get sued, we're going to cover their ass.

So you spend a lot of time in this particular problem set. [43:51] And so it's just immensely valuable. We've only been working with him for a couple of months now, but he's really an end of one. [43:57] And then in terms of your team... [44:00] this is a new problem. How did you assemble a team for this and who's a part of your team? Yeah, so the key thing to get right is you need to both have [44:09] a foot in the insurance world [44:11] because you need to play with their balance sheets.

You need them on board. [44:15] and in the AI world. So the founding team has both a former McKinsey partner in insurance who then went on to [44:25] be the COO of an AI audit organization that does pre-deployment testing, for example, of open AI models. And then our CTO is a former Thiel fellow who built an AI company and then an underwriting company. And so I think there's like that marriage between the technical and the financial understanding that's really critical to pull this off. [44:43] How big is the team so far?

We're just the four of us now. [44:46] Just for. Yeah. [44:47] Are you going to be hiring with this round? What's the plan for scaling? Yeah, we're going to be hiring, though I think there's a lot of wisdom in the startup community now where it used to be that big teams were really prestigious, now small teams are really prestigious. I think there's a lot of wisdom in that. So we're going to be hiring now, [45:06] uh we're going to be hiring both on engineering and insurance we're also going to be doing it pretty [45:12] Slowly enough that it's going to be really painful.

Okay. [45:16] Building a company like this that has many different legs, you just have to be really close to all parts of the business to really succeed. [45:23] Are you going to have any AI agents? Yes. [45:27] on the team. [45:28] We already do. [45:30] What are their names? [45:33] Try to not name them because then you get too attached to them. It's really hard to turn Bob off at night. [45:40] As we... [45:42] lookout over the next couple of years, maybe three to five years, although things are moving really fast.

So that's a couple of years after super intelligence. That's a couple of years. What comes after super intelligence? Super, super intelligence. Super duper intelligence. OK. Yeah. What does this whole world look like? Is every AI agent going to have AI UC? Like, how is this going to work in enterprises? Yeah, I think... [46:06] Every agent... [46:08] is going to need to be certified. [46:10] And every AI agent is going to need to carry some kind of insurance. And like a simple analogy is like, if you're driving on the road today, you obviously must have insurance.

And the reason for that is if you hit me, I need to make sure that I have recourse. And as agents start to do real meaningful workloads, the question is going to be like, who's responsible and who's going to pay when they mess up? And we think insurance is going to be the answer. So we think every agent is going to have some kind of insurance attached to it. [46:40] Yeah. [46:41] What's been the biggest challenge for you starting this company? Building a new standard is really kind of a chicken and an egg problem.

It's like the whole purpose... [46:50] is to build confidence. But at the very beginning, everyone's like, [46:54] where's the confidence going to come from? You guys are brand new. And so... [46:59] Finding a way to build partnerships. We build a big coalition around the standard with both some of the top professional services firm that... [47:07] enterprise work with today, academics at Stanford and [47:11] and MIT and lots of folks in the AI world, bringing them together for us to have a consortium where trusted people are saying, this is the path forward.

[47:21] Getting that ball rolling, getting the first 10 of those was... [47:25] really hard, the 11th becomes a lot easier. [47:28] And I think the other point is for this to really work, you need to have the insurers on board and... [47:36] the... [47:38] Average insurer is spending less time on Twitter than you are. Spending less time engaged in what's coming and come from a perspective of like what could go wrong and what do we know here. And so finding, striking that right balance between like inviting them into the future while also just taking it seriously.

The reason why people trust insurers is that they're really, really rigorous. And so having to meet that has been also just like an interesting challenge. Well, what was the process like getting insurers involved? [48:08] interested in this. Yeah, so one of the founding team members, Rajiv, used to serve insurers [48:15] at McKinsey, where he was in their insurance practice. So that immediately meant we started from a perspective of having connections, knowing their language. But it's interesting to move into the insurance world. The way they build new partnerships like this, where they're getting into new waters, is so different than in Silicon Valley.

Silicon Valley is all about what's your product? Whereas if you go and hang out at Lloyd's of London, the world's oldest insurer in London, they take you to the pub and they look you deep into the eyes [48:45] and they ask questions like, [48:47] How are you planning to raise your kids? [48:49] They're really just trying to understand, can I trust you? Of course, I need to know what your product is, but... [48:56] Is much more of a personal... [48:58] trust relationship feels much more like one big handshake and much less like which features are you building and when are they coming out um so that's also been just interesting and fun [49:09] We can't gloss over the fact that you were the first...

[49:13] Product and go-to-market hire at Anthropic. So how are you thinking about go-to-market with this company? Yeah, what... [49:23] we think is really important is [49:26] The people who have the most burning... [49:28] version of this problem are some of these [49:31] hot AI companies that have raised hundreds of millions of dollars. They want to grow extremely fast and they want to win the big logos. They all want to move into the Fortune 500. [49:42] and [49:43] If you're a two or three or four, five-year-old company trying to sell to those companies, you have a trust problem.

They're willing to do almost anything to earn the confidence of their customers, anything they can do to take risk off the table for them. So that's where we're starting to work with AI agent unicorns like Ada, like Cognition. [50:01] solving their problems. [50:03] I think it's going to be the right way to establish this market. And then eventually it's going to be the enterprise themselves. Once Walmart has like 50 AI agents deployed, [50:11] They're not going to want each individual provider to ensure them. They're going to want to have one big coverage policy.

So that's broadly how we're going to market, like helping some of the [50:19] most interesting agent companies earn the confidence of their customers. [50:24] Do you think that trend of just spending a ton of money on go-to-market is going to continue? Like, how does that evolve as things advance? Yeah, I think we're in a super cycle. I think we are. [50:35] it is still it is wild to think this but it's still early days and um [50:42] I think people are still losing their minds over the salaries that Meta is willing to pay, but...

[50:46] It is a direct consequence of taking... [50:49] this worldview really seriously that in 10 years, everything looks unrecognizable. And that whoever's on top now, [50:57] may not be on top in a year's time. [51:00] That is what's predicating this enormous amounts of investment, both into the technology, but also into go-to-market. The question is really like, [51:06] You can buy more compute. [51:09] but it's hard to [51:10] by your customers confidence you really need [51:13] someone else to say that you're trustworthy like no matter how many billboards you put up saying like we're really the good guys we're really responsible in practice it just turns out that in most industries having third parties [51:23] go and look at that it's going to be more effective um so hopefully we can find ways for them to like [51:28] actually earn the confidence not just get a stamp on it but actually do the things that

[51:32] means that they deserve the confidence of their customers. [51:35] Yeah. [51:37] It's going to be fun. I'm going for a wild ride. Chaotic. Chaotic. Yeah, I think chaos is like... [51:43] I think there's a lot of good stuff coming out of AI, but chaos is really like the key term. Um, [51:49] Yeah. [51:50] As we think about chaos, [51:53] You're the founder of the company. [51:55] One of our sponsors is Brex, and they're all about spending smarter, moving faster. [51:59] all based on performance. We are a Brex customer. [52:02] - You are?

- Yes. - Do you have a metal card? [52:04] I don't actually. We're going to get you a metal card right now. Yeah. And honestly, I have a feature request. How can you become a best customer and not... [52:11] get shoved a metal card in your face. [52:16] I'll file out the tickets. We need them ASAP. Okay. Yes. What do you like about Brex? How'd you get Brex? Well, Brex is just really freaking easy to set up. And what I like is that they're a technology company. I feel like, [52:29] they see me and so far it's been smooth.

[52:33] And Anthropic is a customer. [52:34] Yep. [52:35] Well integrated. Okay, so on the question of performance, at least for you and the company. So how do you think about measuring metrics for milestones? What are you looking to achieve? What like, what is the next kind of evolution for you? What's coming? [52:55] Businesses that are buying AI, [52:57] who today are like, feel like an analysis paralysis between like moving fast and making headlines or getting left behind. But they know that they can come to us and use the AIUC1 standard and push that upon all the AI companies as a way to get in confidence.

So the kind of metric I'm interested in tracking is how many times...