A podcast with Sarah Hastings-Woodhouse, who is on Substack, Twitter (@littIeramblings) and potentially elsewhere
Topics discussed
Current thing: Fable gets export-controlled
Sarah defends government-enforced staged deployment
Non-misalignment misuse as an existential risk, maybe
Internal deployment (other current thing smart people talk about)
Sarah
goes wokethinks that rationalists should not say true but useless and unpalatable things; invisible graveyard of those who’d otherwise have gotten involved as AI safetyWhy Sarah left UK AISI after 10 months
Agglomeration effects (London) vs being able to buy things (Sheffield)
Current thing: it’s very hot in the UK and Sarah is coping
Trump on AI: do we want high variance or reasons-responsiveness when they conflict?
Transcript
[00:01] Aaron: Hello.
[00:02] Sarah: Hello. Yeah, well, my laptop no longer has a functioning built-in webcam because I marinated my laptop in Coke Zero, so.
[00:09] Aaron: Nice.
[00:09] Sarah: Yeah.
[00:10] Aaron: I feel like maybe... All right. Whatever. I’ll try to convince you to get a $10,000 computer at some point.
[00:15] Sarah: I don’t think I need that.
[00:17] Aaron: Sorry. I think I’m projecting.
[00:21] Sarah: If you want a $10,000 laptop, you should get one.
[00:24] Aaron: Thank you. I probably won’t, but you know. I read 1.5, listened to 1.5 of your recent blog posts.
[00:34] Sarah: Thank you.
[00:35] Aaron: And not the one that you requested me to listen to. Although, to be fair, I did the listening before I heard from... Yeah, before you responded.
[00:47] Sarah: That’s fine. I had no idea of anything specific I wanted to talk about. I just felt like yapping.
[00:51] Aaron: Okay.
[00:52] Sarah: Did you generate any takes from listening to 1.5 of my blogs?
[01:01] Aaron: So, the full one I listened to was the Fable. And honestly, I could’ve done a lot more preparation, so we should talk about the other stuff. But the only thing was like, oh, maybe you’re very slightly sympathetic to the, oh, maybe it’s a coincidence — or sorry, not a coincidence — maybe it’s not the Trump administration just being hopelessly corrupt, and a little bit more sympathetic to that point of view than I am. But I didn’t have any burning hot takes.
[01:33] Sarah: Don’t we kind of have evidence that it wasn’t a malicious, petty, anti-Anthropic move because they’re now doing a similar thing with OpenAI? This was the update heard the day after I published that, and I was like, “Oh, no, my point has already been somewhat undermined.” Well, not undermined, because I left it open as to what I thought would happen, but...
[01:52] Aaron: So it remains TBD whether they will in fact be export controlled. And so one thing you can imagine, and I think kind of expect this to happen, is just a rapid approval process or a staged rollout over a month for OpenAI’s. It remains to be seen. I could be proven wrong, but staged rollout over a month. First of all, also giving warning beforehand instead of arbitrarily. Actually, let’s set that aside because, fair enough, maybe they just woke up or whatever. But yeah, basically similar from a high-level point of view, type of treatment, but in fact OpenAI’s just gets approved way faster, with fewer restrictions. And they probably didn’t go through all the effort of turning Mythos into Fable, which was — that amount of effort or time, money, et cetera, is probably bigger than what OpenAI will have ever spent, I’m expecting. Or I guess we’ll never know. But I’m guessing that it’ll be bigger than what OpenAI will have sort of spent or implicitly spent, including via the staged rollout. Also, I feel like the wording is such that the staged rollout is like — isn’t that the term that they’re using, or am I wrong? Am I misremembering?
[03:14] Sarah: Yeah, something like that.
[03:16] Aaron: Because that’s very much a “we’re going to let you do the thing,” just not like, “oh, no, we’re going to ban them,” and then retroactively, or then maybe we’ll have further negotiations. But these are in fact very different things — an indefinite export control versus an informal staged rollout. I don’t know.
[03:35] Sarah: Yeah, I can see how they’re different. I’m just saying it appears like there is now a general principle that the government wants some lag time between the model being developed and it being publicly deployed, and maybe this is partially to harden our infrastructure to cybersecurity threats or whatever. And it now appears that that is a genuine intention that they have, as opposed to they are just mad at Anthropic. They might separately be a bit mad at Anthropic, but like—
[04:02] Aaron: No, I think it’s both. And I just have a very high prior also on the Trump administration being ridiculously cynical and — is cronyism the right word? Just like if Jensen Huang asks for no export controls, it’s like, okay, no export controls. And Dario’s kind of autistic and woke.
[04:26] Aaron: And more importantly, not just totally bending the knee to Trump, and trying to actually be part of civil society in a non-sycophantic way. Now I’m saying sycophantic like it’s a word that you use.
[04:41] Sarah: Yeah. But, okay.
[04:42] Aaron: Yeah.
[04:42] Sarah: I don’t want to defend the Trump administration. That’s not my thing.
[04:49] Aaron: No, do it.
[04:50] Sarah: But I guess the most charitable possible interpretation of events here is that Anthropic did in fact do a much more — their comms strategy around Mythos and Fable was a lot scarier, right? And presumably policymakers don’t have, or at least not all of them have, the time to actually analyze the benchmarks and observe that Mythos Preview and GPT-5.5 were similarly capable. And so maybe they just got kind of jump scared. They scrambled. They did a clumsy, poorly executed thing. In OpenAI’s case, they have not been similarly alarmed because OpenAI doesn’t do that kind of comms, which in my opinion they probably should. And there’s a version of this story where they’re being kind of incompetent, but they are not in fact just maliciously going after Anthropic specifically—
[05:39] Aaron: I mean—
[05:39] Sarah: ...for personal reasons, right?
[05:42] Aaron: I agree this is in principle possible. I think what actually happened is Trump and one or two top people that he talks to just told the Commerce Department. It’s not like there’s an abstract policymaking blob. There is, but separately, I think this specifically ran through Trump and maybe one other person.
[06:01] Sarah: But I feel like that makes it more plausible, right? Because it’s a knee-jerky sort of impulse reaction.
[06:07] Aaron: But we already know — we literally know that Trump personally dislikes Anthropic.
[06:15] Sarah: Yeah. I don’t know. Honestly, I’m not—
[06:19] Aaron: I think—
[06:20] Sarah: ...really sure whether it really matters. The thing I’m more confused about is that there are people coming out now, like Zvi Mowshowitz, for example, being like this kind of principle of doing staged rollouts is the maximally bad policy because it widens the gap between the capabilities that the labs—
[06:36] Aaron: Oh, right. Yeah.
[06:36] Sarah: ...make available to the public. And I’m kind of confused by this take. I think it might be somewhat bad in the sense that now the capabilities are not as publicly auditable, and maybe if you really care about external orgs being able to do safety research with the current frontier, then this is a problem, and it’s bad that we don’t know exactly what’s happening. But it’s clearly not the maximally bad thing. The maximally bad thing is the government is just letting them YOLO and has no insight or visibility into this process at all. At least now there’s some plausible information flow. Just as a consequence of the government having to do the interaction with the labs to agree on the staged rollout, they are now getting more information.
[07:12] Aaron: Yes.
[07:12] Sarah: And the government getting more information seems like the only plausible path I can think of to anything sensible happening. So I’m kind of confused why people seem to be so upset about it, given that the—
[07:21] Aaron: Yeah. But I—
[07:22] Sarah: You know what I mean?
[07:23] Aaron: Sorry. Yeah. No, go ahead. Yeah.
[07:25] Sarah: That was my point.
[07:26] Aaron: No, I tend to agree with you. So, internal threat model being: Anthropic, ASI, or some sort of AI system is developed in a lab, and basically takes over lab infrastructure and takes over from there.
[07:44] Sarah: Yeah.
[07:44] Aaron: Takes over other systems from there. I think that is more — it seems to me like that is more likely, sorry, conditional on that happening. Proper x-risk, existential catastrophe, is more likely conditional on that than conditional on a different situation happening, which is a model is publicly deployed and therefore bad people have access to it and misuse it.
[08:17] Sarah: Okay.
[08:17] Aaron: Does that make any sense?
[08:18] Sarah: Yeah. Is the argument there that it’s better to publicly deploy the models because that means we’re more likely to get a sub-existential catastrophe that is like a warning shot or something?
[08:28] Aaron: Well, that’s even more galaxy-brained. I was just thinking about, straightforwardly, if you compare the two, there might be some trade-off where more public deployment means more of just, quote-unquote, “normal” bad things happening along the lines of crippling infrastructure around the world.
[08:46] Sarah: More public deployment doesn’t mean that you don’t have the parallel risk of internal deployment causing a catastrophe, right? That doesn’t mean—
[08:59] Aaron: So I think I’m imagining a situation where companies can choose to spend resources on developing systems internally or serving models to the world, or some combination of those things.
[09:13] Sarah: So the internal development actually gets sped up because now they are using less of their resources to do product and external deployment and whatever, and the government—
[09:23] Aaron: Yeah. I mean, this is in my head. This is something that I just sort of... This is not a well-thought-out argument from before this episode started. This is just me on my feet, so I’m not sure this checks out.
[09:36] Sarah: I can see that. I just think, I don’t know. The hill I’ll die on is the only way this situation is not completely terrible is if the government is woken up and does something, and they’re way more likely to be woken up and do something if they just have these information channels with the companies.
[09:56] Aaron: Yeah. I think that’s right.
[09:58] Sarah: That at least makes it the case that this isn’t the worst possible policy that they could have. I don’t know. I’m not saying it’s good, I’m just surprised by the strength of feeling that people seem to have about it.
[10:11] Aaron: Is it just Zvi?
[10:13] Sarah: Zvi and... I don’t know. I can’t remember now. I was doing a little discourse audit the other day, and I saw several negative reactions.
[10:23] Aaron: Okay.
[10:24] Sarah: But yeah. I don’t know. I also don’t have a really hot take here. I need to write about it.
[10:31] Aaron: Yeah. Nice. Based. Okay. Maybe this seems less crazy to me because the government is standing in the way, and therefore you basically force a dialogue mechanism. And there’s also the “holy shit, it’s in the news that Walmart just lost all their data” or something type of feedback, and it does seem like you get the latter more from just having the frontier be in fact deployed, right?
[11:08] Sarah: Mm-hmm. Yeah. This sort of seems like a version of the point I made earlier about warning shots, right? Or just—
[11:15] Aaron: Okay. Yeah.
[11:16] Sarah: ...put people out in the world and let them fuck stuff up a little bit.
[11:18] Aaron: Yeah. Oh.
[11:19] Sarah: Yeah.
[11:19] Aaron: No, that seems like a bad policy. Okay.
[11:22] Sarah: I mean, I don’t know. I don’t want to sound like I’m endorsing people causing sub-existential catastrophes with AI, but this is a common theory of change that people think about, right?
[11:35] Aaron: Yeah. Wait, let me think for a second. I guess what you don’t want is just a systematic... I feel like it’s not crazy to think that you don’t want a systematic bias toward protecting only against sub-existential stuff.
[11:59] Sarah: Yes.
[12:01] Aaron: You could imagine trying really hard to make sure — I mean, at the extreme, try really hard to make sure that nobody ever... there’s no embarrassing things for OpenAI where ChatGPT tells someone to kill themselves or helps them kill themselves or something. And then you don’t care at all about larger-scale or existential-scale stuff. And then it seems like then there’s a systematic discrepancy, and people will be fundamentally misinformed about alignment stuff because you just don’t observe the stuff that would happen in the world where alignment is hard, or at least from a policymaker’s perspective.
[12:41] Sarah: Well, this is what I feel like is currently happening, or this is my current concern—
[12:45] Aaron: Yeah.
[12:45] Sarah: ...with the current regime. It appears from the Fable thing, and then subsequently from the 5.6 thing, it feels like the government is playing jailbreak whack-a-mole, right? They’re like, “Well, what we need to do is patch all the jailbreaks.” And they haven’t internalized this more general principle that you cannot, in fact, patch every jailbreak. Or not even — with the Anthropic one, it wasn’t even a jailbreak. It was just like, yeah, this is a thing that the model can do. You don’t even really—
[13:11] Aaron: Oh, right.
[13:12] Sarah: ...even need to jailbreak for them to do this. It’s a red herring, right? The broader thing is we actually don’t really know how to control these things. Absent technical breakthroughs, we will not be able to patch every jailbreak for every system that comes out in the future. And it seems like they’re just trying to put plasters on a bunch of gaping wounds, and that’s the thing they’re currently optimizing for, and that seems bad to me. And my theory of change is that, in having this dialogue with the companies, they internalize the broader thing, or it just becomes more salient or obvious to them that actually this whole thing is kind of insane and not just a case of doing a little bit of window dressing to prevent people jailbreaking things.
[13:54] Aaron: Yeah. Sorry.
[13:56] Sarah: That’s it.
[13:57] Aaron: No, no, no. Also, I need to make... Yes, you should speak more than me because you know your shit.
[14:06] Sarah: Well, I—
[14:06] Aaron: No. Okay, whatever. You don’t get to reject that claim.
[14:12] Sarah: I stand by it.
[14:13] Aaron: Nice. So I was... Okay, maybe this is a spicy take. I’m not sure. So listening to Shah on 80K recently. Are you familiar with this episode? Or have you listened to it?
[14:29] Sarah: No.
[14:30] Aaron: Rohin Shah.
[14:31] Sarah: Oh, yes, the DeepMind guy on 80K.
[14:35] Aaron: Yeah.
[14:36] Sarah: Yes, I listened to that. Yeah.
[14:37] Aaron: Nice. Wow. Okay. You’re really on top of your shit. So I think maybe this is trivial, and I should’ve come to it myself, but I think he was making the point that jailbreaks were an issue largely through — no, honestly, still through now, but that is because companies haven’t actually worried about the consequences of jailbreaks. So I’m not convinced that if you make that... Maybe Fable is jailbreakable, but it seems to be like maybe — what if it’s just not? I don’t think we’re going to get a technical proof that it’s not, but it doesn’t seem crazy to me that in fact it’s not.
[15:14] Sarah: Hmm. Yeah, and then would this in fact be good or bad, I guess?
[15:22] Aaron: Oh.
[15:22] Sarah: Maybe not bad. It seems like a—
[15:29] Aaron: Why is it bad?
[15:30] Sarah: Well, maybe the general thing is: sub-ASI systems being well-aligned and hard to jailbreak is plausibly not evidence that we’re any closer to solving the ultimate problem of aligning ASI. So then the better job we do in the current regime where models are not superintelligent, the worse off we are because the greater our false sense of security we have, or something like that. So if you couldn’t jailbreak Mythos, I feel like maybe this would be kind of bad.
[15:58] Aaron: I think I disagree with that because I think this is a problem that has to be solved at the limit. At some point, we have to figure out how to... preventing jailbreak is not just a minor thing, whereas true misalignment is the major thing. Maybe directionally, if I had to categorize them or put them in order, that’s true, but it seems like jailbreaking — fixing that — is actually really important for preventing x-risk. Because the misuse isn’t just... You don’t have the classic Yudkowskian story, but I don’t know, it seems very plausible that misuse of a publicly deployed jailbroken model could be on the path to existential catastrophe.
[16:54] Sarah: Yeah, I agree. I was just saying maybe we could in fact just solve the misuse problem because we are really good at preventing jailbreaks, and then no one jailbreaks a model to build a bioweapon, and we’ve eradicated that particular threat model. But it doesn’t help with the “will the model ultimately do what we say or care about our values at all” thing. Those things don’t really seem—
[17:12] Aaron: Yeah.
[17:12] Sarah: ...Yeah. It matters that the model — I guess, jailbreak, I don’t really know if this is true, but jailbreak prevention seems like a kind of after-the-fact layer that gets added on top to prevent the model’s true capabilities from being accessible, but it still has them, right? So it’s like the fact that a person couldn’t get the model to do a thing that the company has said they’re not allowed to do is not evidence that the model couldn’t just do a thing itself that felt like it.
[17:43] Aaron: There’s some conceptual stuff there. It’s like what would it mean for — if it is in fact just the case that there is no combination of words that will elicit a response that is undesirable in some specific way, what would it mean for the model to want to be able to... could do it if it wanted to? It sounds like, no, it probably couldn’t. That’s kind of a conceptual thing.
[18:07] Sarah: Mm-hmm.
[18:09] Aaron: But I feel like that’s me being philosophy-brained or something. So there’s still the problem of Anthropic has an unjailbroken — I don’t know exactly how they’re dealing with this internally, but presumably, and I think just in fact, Anthropic has access to the unjailbroken version, and so then you get into internal security stuff.
[18:40] Sarah: Yeah.
[18:41] Aaron: And it doesn’t solve the problem at the limit, but I don’t know. I feel like it does help you bumble along a little bit longer or something. If only seven people who are rigorously, very strongly vetted could access the capabilities, and even then it would raise alarms internally. Whereas the entire world, once you have a single jailbreak or a strategy, or people for hire who can do it, or a model that can do it, or anything like that — it’s 10 billion people, 8 billion people is just a lot more than seven. And so I feel like the tail risk — you always want to abstract that away and say, “well, but both probabilities are non-zero.” It really makes a difference what the risk is from misuse of a less solidified version of a model.
[19:41] Sarah: Mm. Yeah.
[19:42] Aaron: I feel like I didn’t say that well.
[19:45] Sarah: Yeah. There’s less risk of catastrophic misuse if seven people have access to a jailbroken model versus 8 billion, I agree.
[19:55] Aaron: Yeah.
[19:56] Sarah: I don’t know. I still feel spooked by the prospect of, I don’t know, there’s some crazy capable model that’s just deployed inside of an AI company, and there’s a ton of information siloing or whatever. So even only a small fraction of that company even have access to that model or know what’s going on with it. That just seems like a — both from a concentration of power perspective and from a “we now have fewer people with eyes on the problem of making this thing not go crazy” perspective. I don’t know. I don’t feel great about that. I guess that’s one — yeah, this is an argument for the staged deployment thing in fact being bad, which is now all of us have less of an idea precisely what is happening, and maybe terrible things are currently happening or will be happening in the near future, and we just won’t know.
[20:40] Aaron: Yeah. You could imagine a mandate. This is just popping into my head. I have no idea if this is a good idea, but you can imagine a mandate that — maybe you don’t need full Fable-level safeguards internally, but you need... there’s some threshold. I think you probably do need this at some point, which is just restrictions on internal deployment, which basically don’t exist now, right, as far as I know? There’s no legal stuff.
[21:13] Sarah: Not to my knowledge.
[21:15] Aaron: Okay. Yeah.
[21:16] Sarah: Yeah.
[21:17] Aaron: And then this is a weaker version. You could also just tie it to what’s publicly available. You could say you have to do X, Y, and Z to launch something publicly, and then that is also the model that you’re allowed to use internally. I don’t know if this is viable at all. It just popped into my head. Maybe it’s probably not smart, but it’s something that could be done in principle.
[21:39] Sarah: People definitely have done... I now no longer know what I’m talking about. I think there’s an Apollo paper which is about this. So it’s about the risks of internal deployment and some mitigations that you could do. Because it does seem like, yeah. This is another thing with my red herring point, that I think no one in the government is thinking — or almost certainly hardly anyone in the government is thinking about the risks of internal deployment, right? They are conceiving of AIs as things that are dangerous when the public or foreign actors get hold of them, and not as things that could be dangerous when they’re just inside a company.
[22:11] Aaron: Yeah. No, I agree. Is there probably someone in some — like ASI-pilled in some department somewhere? It’s like, yes, but no. I agree, and it’s just like, “Oh, that sucks.”
[22:25] Sarah: Yeah. It’s pretty bad. Anyway, maybe we should—
[22:31] Aaron: No, go ahead.
[22:31] Sarah: ...move off this thread.
[22:33] Aaron: Yeah, no. I had no commitment to only talking about the most serious stuff. Okay, cool. I feel like you have lots of takes. How do I elicit latent Sarah takes?
[22:45] Sarah: I don’t even know how my takes should be elicited. When I try to do an audit of my own takes in my brain, I can’t think of them, and they have to be triggered by someone else.
[22:54] Aaron: Yeah. That’s why I should’ve actually listened to your four hours of blog posts.
[22:59] Sarah: No, I wouldn’t subject you to that. Don’t worry.
[23:03] Aaron: Okay. Rat discourse. So you had a blog post that was extremely well-written and also not incredibly convincing, according to me.
[23:11] Sarah: Okay. Want to push back?
[23:14] Aaron: Well, my pushback is you’re critiquing a blog post and essentially it was like, “Look, this is such a good microcosm of the entire rat ecosystem.” And I’m like, “I don’t know. I’m not convinced of it.”
[23:26] Sarah: I didn’t say the entire rat ecosystem. I said—
[23:29] Aaron: Okay. Sorry.
[23:29] Sarah: ...one documented instance of a phenomenon that I have observed on multiple occasions. I’m not making any claim about the percentage of rats that have this particular brain worm. Maybe it’s actually quite low. I just—
[23:41] Aaron: Yeah.
[23:42] Sarah: ...it happens too often for a community that is ostensibly so concerned with, I don’t know, value maximizing and being smart. You know what I mean?
[23:50] Aaron: Wait. Can you actually spell out what the “it” is here?
[23:55] Sarah: Yeah. So I was trying to think about... I frequently have this experience, right, where I’m on Twitter, and I’m experiencing this whiplash of going between — usually it’s AI safety discourse which concerns itself with “will AI literally destroy the world in two to five years?” — and then some other stupid culture war topic, which just feels like a massive sinkhole of value and time. And the specific one that I was providing commentary on in that blog post is a blog about how women should get sexier, and the advice mostly amounts to don’t eat and have big boobs or whatever. And this just happens a lot, right? There’ll be some cultural thing — I don’t know, should we be allowed to flirt at EAG, or like—
[24:40] Aaron: I would never discuss this, ever.
[24:44] Sarah: Would never participate in that discourse. Or some eugenics-y adjacent... I think it’s because the rats have a tendency to — they like commenting on unpalatable things, and they’re somewhat optimizing for the goal of saying true but unpalatable things, even if there is zero value to saying those things, and if there is in fact negative value to saying those things. This disagreeable — let’s say the “true thing that no one wants to say” heuristic, I just think is manifesting in this way that is just so distasteful to me, and that actively kind of makes me feel repelled from the discourse. And if I was not already so bought into AI safety, and if I didn’t know that most people in this community are actually lovely, I think it might just turn me off altogether. And that makes me think there are many other people who are in fact turned off by this.
[25:33] Aaron: Yeah.
[25:34] Sarah: There are hit pieces in The Guardian that get written about us, right? Because someone at Manifest decides that they’re just going to do some race science. And it’s obviously hugely discrediting to the entire movement, which I think is unfortunate because I think the movement is generally correct about many things, including extremely consequential things. I don’t know. It just bugs me a little bit.
[25:55] Aaron: Yeah. We should fight about this. Okay. So I agree that there is a suboptimal dynamic. I’m not sure that it’s a realistic better option than... I mean, just hold on, hear me out for a second. Because I feel like the ask is not just a change of norms, it’s like “if everyone would just do better according to this one perspective” — and it’s just individuals are just going to... So I feel like the stronger case here is people who are really bought into AI x-risk shouldn’t throw away points on stuff like this. But it’s a harder case to make that there’s even a lever of change over individuals who are broadly rat-adjacent but don’t necessarily prioritize — just aren’t super altruistic, for one thing. Don’t prioritize AI, or maybe are vaguely bought in but just in fact don’t care that much or something. And it’s like, I don’t know, policing individuals, it’s just a hard thing to do.
[27:10] Sarah: Oh, I mean, to be clear, I’m not suggesting that we should police people’s speech, and I’m also not making the ask that people stop doing this, per se. I’m not saying... I think I even said in that post, I was like, “I’m not trying to litigate what people can and can’t say.” My point was only that I think it reveals about people that they haven’t internalized the values that they say they have, right?
[27:31] Aaron: Ah.
[27:31] Sarah: The value being — I think the broad thing that connects all of these communities, because it’s actually a bunch of adjacent communities, right? It’s the rationalists and the EAs and the pro-progress people and other factions.
[27:42] Aaron: Yeah.
[27:43] Sarah: They’re united by this thing, which is: we are willing to entertain ideas about the future or about reality that are outside of the Overton window of the average person, right? We have the imagination to consider dystopias or utopias, or the possibility that, I don’t know, bugs are having a really bad time, or maybe Claude is having a really bad time or whatever. And they just appreciate this vastness of reality and how things could be way worse or way better. And I think if you really had internalized that and you really thought that reality was that huge and radical and weird and big, you just simply would not concern yourself with whether or not women should look like pin-up dolls to get a husband. Obviously, I think this is obviously going to be the case, right? Because it’s extremely hard to internalize those kinds of values—
[28:28] Aaron: Yeah.
[28:28] Sarah: ...and people concern themselves with all sorts of pedestrian things all the time, so it’s not surprising that this happens. But that it happens in such an egregious way and in a way that is like, I don’t know, it makes the community feel hostile for me. I don’t want to be sitting there on Twitter reading about how I’m too fat to be attractive, or... We have high-profile members of this community, like Roko, for example, who has said on Twitter that—
[28:52] Aaron: Oh, he’s been a persona non grata for ages.
[28:56] Sarah: No, but I agree with you, right? That is true, and that is a credit to us. But what is it about this culture that is producing people like this in the first place? I don’t know what it is, but it’s something.
[29:10] Aaron: I feel like Roko — I want to push back on Roko specifically. That’s just a perfect example of doing the kind of, I mean, policing — but quote-unquote policing, not necessarily in a bad way. I think some community policing is genuinely warranted. And this is a great example of the rat sphere doing the good thing of excommunicating this guy who seems pretty bad and has terrible takes.
[29:35] Sarah: Mm. Yeah.
[29:36] Aaron: So it’s like—
[29:37] Sarah: And I’m not even making the case for exiling more people. I just think it’s interesting. He’s not the only one. There are a bunch of these anime accounts that are anonymous, have very similar takes to him, and they at least have some level of visibility and are engaged with by people whose takes I would otherwise respect, even if those people aren’t agreeing with these people, but they’re giving them the time of day.
[30:02] Aaron: Ah.
[30:02] Sarah: They’re in the water. I will see this stuff even though I don’t care about fertility rates or how to get sexier or IQ and race or whatever. But this stuff is in my algorithm. It’s in my feed because I care about AI safety. And the fact that there’s some link between those two things seems like a problem to me.
[30:25] Aaron: Okay. Nice. I like it. I see what you’re saying. I think maybe you should consider just using the Following feed that’s on Twitter.
[30:33] Sarah: Well, no, because I do genuinely — often the algorithm will bring things to my attention that I do want to see, right?
[30:40] Aaron: Okay. Yeah, fair.
[30:43] Sarah: I don’t know. I just—
[30:45] Aaron: Yeah.
[30:45] Sarah: ...I worry about the graveyard of people who might otherwise have become part of this community or contributed in a positive way that have just been totally turned off by this stuff. Like—
[30:57] Aaron: Yeah. No, I see that. It’s just hard for me to imagine a different situation, so maybe it’s just not tractable, and it’s just like, “Oh, this sucks,” which is something we can agree on. I guess insofar as it’s laden with some sort of normative ask or something, like, “Guys, come on,” then it’s very hard for me to imagine a world where you don’t have random anime accounts who are vaguely vibe-adjacent to rationalism but also don’t talk about unpalatable issues.
[31:31] Sarah: Also, I don’t think that I’m going to put out a call with my blog post, and I’m like, “Everybody, please stop doing this,” and they’re like, “Sorry, I was being unreasonable. I’ll stop.” I don’t know. It’s more of an observation. I don’t know. I feel like if someone—
[31:45] Aaron: That’s a reasonable thing to potentially ask. It’s not like everyone’s going to take it as the word of God, fair, but this is the kind of thing that you could totally reasonably do if you wanted to.
[31:55] Sarah: Yeah. I don’t know. I just want people to introspect a little more, be like, “Okay, what are my values? What do I believe?” And do those really cash out in a specific concern about hip-to-waist ratios for single women? It would be surprising. Whose value set is that? Anyway, maybe we can move on from this.
[32:17] Aaron: Okay. I’ll see what I want to cut and see how hard I want to get canceled today. So tell me everything you know about the British government.
[32:27] Sarah: Oh.
[32:29] Aaron: Just kidding. Wait. Okay. Tell me what you were doing at AISI. Wait, not American.
[32:39] Sarah: I mean, basically, I worked on our comms team for 10 months, and the team was very small. When I joined, it was literally just one person. Then I was the second person. And so there’s maybe 10 different research teams at AISI. Everyone’s trying to push out research all the time. If you’re on the comms team, you get a cool bird’s-eye view of all of this. So it was like, do we want to have a splashy comms moment around this paper? Usually, I was either writing or editing the blog posts that went out. I was also somehow the only person, or one of the only people, that had access to the back end of the website, and one time I broke the entire thing for a weekend by trying to write my own custom code.
[33:21] Aaron: That’s extremely funny.
[33:23] Sarah: Yeah. Well, not write it myself. I tried to insert it in a way that was wrong, and then the entire—
[33:28] Aaron: I remember this. I remember this.
[33:33] Sarah: Yeah. Anyway. I don’t know. I loved it there. It was a great time.
[33:40] Aaron: Oh, cool.
[33:40] Sarah: I was sad to leave. The only reason I left is because I felt like I just wanted to yap freely, and so that’s what I’m doing now.
[33:48] Aaron: Yeah, yeah.
[33:49] Sarah: It’s pretty hard to yap freely in the civil service, famously.
[33:53] Aaron: Yeah. Wait, so do you know what would actually happen if you just yapped freely? Okay, not freely about anything you’ve explicitly signed an NDA about, but everything else.
[34:06] Sarah: Are you saying what would’ve happened if I just said whatever I wanted while I was there?
[34:11] Aaron: So I guess I don’t know what — about things you didn’t promise anyone or the government that you wouldn’t talk about.
[34:20] Sarah: I mean, the rule was more that you don’t — this is a government-wide rule for the whole of the UK Civil Service, which is that you don’t publicly comment on the thing that you work on.
[34:31] Aaron: Okay. No, that’s actually—
[34:33] Sarah: That strict. Our technical researchers would be like, “This is a cool paper,” or whatever, but it’s more expressing opinions, especially opinions that involve policy positions or whatever.
[34:45] Aaron: Yeah.
[34:45] Sarah: That’s not an AISI-specific thing. It’s just like you can’t do that.
[34:48] Aaron: Yeah.
[34:48] Sarah: And especially if you’re a person, it’s more so assumed that your opinions affect—
[34:53] Aaron: Yeah.
[34:53] Sarah: ...your employer, even if you say that they don’t, which is probably fair enough.
[34:57] Aaron: I think you made the right choice to yap freely.
[35:02] Sarah: Yeah. But I don’t regret it, and I’m pro-AISI. AISI is cool, and people should go work there, even if temporarily. I think it was a good decision.
[35:10] Aaron: Nice. Yeah, but you did publish all of the blog posts, right? By AISI?
[35:17] Sarah: Yeah. I would say — no. I mean, maybe 50% of the ones that were published while I was there, I wrote the first draft of, and the other 50%, the research team drafted them, and then we synchronously edited them.
[35:32] Aaron: Okay. So it sounds like you wrote both, or you were heavily involved in—
[35:36] Sarah: Yeah.
[35:36] Aaron: Okay, nice. That’s actually just impressive. Wait, doesn’t it not say your name?
[35:41] Sarah: No, yeah. There are no bylines. It’s—
[35:43] Aaron: Oh, yeah. That’s nuts. Wait, are you allowed to claim retroactively that you wrote stuff?
[35:49] Sarah: I mean, I can say which ones I wrote. Yeah, I think it wouldn’t be the norm for this kind of output that you would put author names on.
[36:02] Aaron: Okay. I feel like you should probably do that unless there’s a really good reason not to. Because I’m just curious which ones you wrote versus edited. I guess basically all of these now are. Yeah. There’s a lot here. You can go to aisi.gov.uk/blog and see all of Sarah’s beautiful writing.
[36:22] Sarah: I mean, they’re not all mine, like I said. It’s a combination — I won’t just go through and say all the ones that I wrote on here because that feels weird.
[36:31] Aaron: Okay. Maybe I’ll convince you, then we’ll add it to the episode description.
[36:35] Sarah: Sure.
[36:36] Aaron: Yeah. Wait. Do we have anything that’s not just a normal... Hmm. What would we talk about if we weren’t on a podcast, but that also isn’t very extremely personal or something? I feel like that’s always the best podcast content.
[36:50] Sarah: I can talk somewhat about personal things — not personal in that they’re extremely revealing.
[37:03] Aaron: No, they’re not.
[37:04] Sarah: But they’re not super important.
[37:05] Aaron: Yeah. Okay, cool. How’s your personal life?
[37:10] Sarah: It’s pretty good. I’m moving back to London in two weeks. So if anyone lives in London, is an AI safety person, and wants to be my friend, they should just DM me on Twitter.
[37:23] Aaron: Am I allowed to do that even if I don’t live in London?
[37:28] Sarah: You—
[37:28] Aaron: If you come to — go ahead.
[37:29] Sarah: Yeah. You can DM me on Twitter whenever you want.
[37:32] Aaron: Okay. Thank you. Cool. Okay. Yeah, honestly, I’m sorry, but I’ve kind of lost track of your various living and working situations a little bit.
[37:43] Sarah: No, that’s chill. I honestly don’t even totally know what you’re doing right now either, so we’re even.
[37:48] Aaron: Yeah. Okay, cool. Nice. Okay. Wait. Remind me. I think you just said this, but why are you moving to London again?
[37:58] Sarah: I—
[37:58] Aaron: Or is it not for a specific thing? It’s just a vibe.
[38:02] Sarah: No, it’s basically AI safety reasons. I’m moving—
[38:05] Aaron: Nice.
[38:05] Sarah: ...very close to the LISA offices. And I just realized that living in Sheffield... I do love Sheffield. It is one of my favorite places, but it does not... Actually, it does have an EA presence that I only recently discovered. And a bunch of them, the northern EAs, are dying on the hill of “it’s so stupid that all of the EAs move to London and spend all this money on rent that they could donate.” And so there’s a big campaign to get people to move up north, but I defected and was like, “Sorry, I’m part of the coordination problem. I’m going down south.” But I don’t know.
[38:39] Aaron: Nice.
[38:39] Sarah: I unfortunately just don’t want to make the sacrifice of trying to be the flag bearer for moving to Sheffield and doing EA stuff.
[38:47] Aaron: No, I have totally switched my opinion. So I was on about Tulsa at one point, and how everybody should move to Tulsa. And I have actually just come around to — okay, straightforwardly, maybe not straightforwardly, I don’t know if it’s the right word, but the agglomeration effects are just so strong, but they’re strong for a reason. And yeah, from an EA perspective — well, I don’t know if this is true in all worlds, but in the current world, you can imagine something where there’s basically zero funding or something, and there’s a very small community. But setting that aside, in the actual world, I think people should just not do the Sheffield thing, and it’s just straightforwardly worth it to donate less and live elsewhere.
[39:31] Sarah: It is a lot less, to be fair, though.
[39:35] Aaron: No. Yeah.
[39:38] Sarah: People just don’t realize how cheap Sheffield is.
[39:44] Aaron: Yeah. It just seems like really strong evidence, the fact that so many people just do, in fact... So setting aside EA world, just the fact that there is so much demand commercially, societally and socially and economically — I guess I meant that instead of commercially — for people to concentrate in one city just seems like really, really strong evidence that there’s really high returns to almost any sort of... No matter what you care about, it seems likely that probably you’re going to accomplish that best in a large city.
[40:18] Sarah: Yeah. But it’s very chicken and egg, though, I guess. Everyone believes this, and so they—
[40:25] Aaron: Yeah. But — philosophically, yeah, but I think in fact, I don’t really imagine... I think it’s sufficiently stable at this point. Maybe in 1591 or whatever — so I did not mean 1491, I meant a random year, so 1595 or whatever, when people were starting to go to New York — it’s like, “Oh, should we go to the other side of the river?” Maybe at that point, but at this point, I don’t think it’s really a coordination problem that’s possible to solve because there are really strong institutions in specific places.
[40:58] Sarah: Yeah. But everyone’s — I don’t know. I maybe buy that it would be good if more people moved to cheaper places and donated money, and that those people could probably do... Given that so much work is remote now anyway, I would buy a lot of them can just do as good of a job from those other places. I mostly want the social aspect, and I feel like if I go into a co-working space, I absorb a bunch of ideas via osmosis. But that’s kind of—
[41:24] Aaron: No, that’s important.
[41:25] Sarah: I don’t think I literally couldn’t do what I’m currently doing if I—
[41:30] Aaron: No, but the osmosis thing is really, really important.
[41:33] Sarah: Well, I do think it is. I do think it is somewhat... Yeah.
[41:37] Aaron: This is one influence about why I’m... I’m going to be working remote, at least to start, and I’m still going to Berkeley. Maybe this is evil of me and I should move to Tulsa. I’m not sure.
[41:48] Sarah: No, I definitely don’t think that. Are you going to try and get into Constellation? I don’t know if that’s hard.
[41:54] Aaron: I haven’t actually thought about this at all. It would be cool if somebody invited me or I could go, but it’s not like, oh, my professional plan is to work from there specifically.
[42:05] Sarah: Yeah.
[42:08] Aaron: Although that would be, I guess, depending... Although maybe I’d have to tell somebody. Yeah, maybe it depends what I’m doing, probably. I just have to figure that out, and how to communicate it. But I need to liberate my mind. Do you have any burning — or not burning — any interesting things at all?
[42:31] Sarah: It is like getting blood from a stone, isn’t it? I know.
[42:36] Aaron: That’s so British.
[42:40] Sarah: Is that not a thing that you guys say?
[42:41] Aaron: Blood from a stone? No, I don’t think... Honestly, maybe this is a Dostoevsky moment, and that is a thing people say, but it’s not a thing I say.
[42:51] Sarah: It’s hard to get something out of something.
[42:54] Aaron: Oh, yeah. Sure.
[42:54] Sarah: Like interesting takes out of me.
[42:56] Aaron: No, not interesting takes. It’s hard to get goss, but that’s a different story.
[43:00] Sarah: No, that’s true. Maybe I would give you gossip if you just prompt me more specifically.
[43:05] Aaron: I should jail—
[43:05] Sarah: Don’t prompt me.
[43:05] Aaron: ...Okay. Wait. How do I jailbreak Sarah? When are you moving to London?
[43:12] Sarah: Two weeks in theory, but there’s going to be another heat wave right when I intended to do this, so I may in fact—
[43:19] Aaron: Oh, yeah.
[43:21] Sarah: ...not do it when I said I was going to do it.
[43:25] Aaron: People should yell at Sarah to get air conditioning. It’s high impact.
[43:28] Sarah: They do. To be fair, I’m coming around to... The other night when it was 32 degrees during the day or something here, I just had too many fans, just one on either side of my bed, just blowing air at me.
[43:44] Aaron: Was that pleasant or was that still too hot?
[43:48] Sarah: Well, it was pleasant. But only the top half of my body was cooled. It was not—
[43:52] Aaron: Okay. Sounds not pleasant, actually.
[43:56] Sarah: The whole thing was unpleasant. Honestly, it’s so funny how ill-equipped we are to deal with the heat, to the point where it’s all anyone talks about. I’m still talking about it, and the heat wave has ended. Watching the news during that week, it was alternating between the fact that our prime minister quit and then just people on the news being like, “It’s really hot right now. Here’s some various shots of places in London where it’s hot. This is Tower Bridge. It’s really hot there right now. There aren’t many people about because it’s so hot. Here’s a woman splashing herself with water because it’s so hot. God, when is it going to get less hot?”
[44:26] Aaron: Yeah.
[44:26] Sarah: Just that on the news for days.
[44:28] Aaron: If only we had the technology.
[44:31] Sarah: If only there was something that we could do about this.
[44:34] Aaron: Yeah.
[44:34] Sarah: You know?
[44:35] Aaron: Are you going to get AC at your new living situation, whatever that is?
[44:38] Sarah: Maybe I’ll get a little one of those units that are a few hundred pounds that you just put in your room or whatever.
[44:44] Aaron: Yeah. Should I — wait, I’ll show.
[44:47] Sarah: Okay.
[44:47] Aaron: Well, I guess my background is blurred, but this little guy—
[44:51] Sarah: Oh, okay.
[44:51] Aaron: ...this window unit. I think it was 100 — well, I think it’s gone up, but it’s under 200 bucks, and this is saved.
[44:57] Sarah: Oh, okay.
[44:58] Aaron: This is the reason why. So it’s going to be — wait, let me check again. It’s going to be... Oh, it went down. Yesterday, it was projected to be 103 on Friday. Now it’s only projected to be 102. So that’s actually pretty bad for DC.
[45:10] Sarah: Yeah.
[45:10] Aaron: That’s pretty uncommon.
[45:12] Sarah: I was in DC in the summer of July 2024, and I had the flu, I think.
[45:18] Aaron: Oof.
[45:18] Sarah: It was one of the worst experiences of my whole entire life.
[45:22] Aaron: That’s why you never come.
[45:23] Sarah: Wait, I come to DC so much.
[45:26] Aaron: Okay. Yeah. Okay, fair. Wait, 103 to Celsius. I could do this, but I’m not going to be able to do it in my head while I’m looking at someone.
[45:35] Aaron: Okay, that’s 39. 39—
[45:37] Sarah: Whoa. That’s a lot. Okay.
[45:40] Aaron: 39.444. It’s easy to say.
[45:43] Sarah: That might be hotter than any temperature I’ve ever experienced. I think when I was in DC, it was 37.
[45:49] Aaron: Okay. No, this is very uncommon.
[45:52] Sarah: Sorry. This is not an interesting topic, probably. There was one day in 2022 where it got to 40 degrees in the UK, and I remember this because me and my housemates tried to calm down — cool down, even. We didn’t have air con, so we just—
[46:06] Aaron: Oh, God.
[46:06] Sarah: ...slept on ice cubes, and we watched the movie Frozen because we thought that—
[46:11] Sarah: ...we thought if we just thought about ice, it would just feel immersive, and we would cool down. It kind of worked. But that’s a low intervention—
[46:20] Aaron: I’m going to re—
[46:20] Sarah: ...is just watch ice-themed content.
[46:22] Aaron: I’m sure that will work if I turn off all the AC in my—
[46:29] Sarah: Yeah.
[46:30] Aaron: ...house or whatever on Friday. How do we convince the world — not the world, the UK — to adopt air conditioning?
[46:38] Sarah: I think we just will eventually because if the summers are like this every year, it will be totally impractical not to, right?
[46:49] Aaron: Yeah. I guess you could — yes, but I guess much like AI stuff, there’s a question of how do you accelerate or decelerate. Something might be inevitable, but there’s a question of how to control the timelines within some bounds of plausibility, so I don’t know.
[47:07] Sarah: Yeah. I think there is just a contingent of people here that are just on principle opposed to—
[47:14] Aaron: Is it American? Is that a vibe — people think it’s American-coded?
[47:19] Sarah: But yeah, people just want to suffer.
[47:22] Aaron: Not me.
[47:23] Sarah: I don’t, either. I don’t want to suffer.
[47:26] Aaron: Yeah. Okay, cool. Should we wrap up with one last topic of your choice—
[47:34] Sarah: Yes.
[47:34] Aaron: ...so I don’t have to choose?
[47:36] Sarah: Oh, man. Maybe I should ask you a question. How has your level of optimism/pessimism about AI safety changed in the last, let’s say, year?
[47:52] Aaron: Oh, that’s actually a good question. Wow, nice. I feel like we should probably have done more of this where I make you ask the question — not so I can answer it, but because that’s a good question, and then we can both discuss it. So a year. Okay, so what was happening in June/July of 2025? I feel like it hasn’t changed that much. It’s definitely changed since Trump, too. I’ve gotten more pessimistic broadly in Trump’s second term. But I think that was all pretty built in, and honestly, I don’t have as much of an explicit model with a change log, like people like Zvi do. So wait, hold on. Wait, what important developments have happened in the last few years?
[48:42] Sarah: Well, I keep a blog on this, so let’s see.
[48:44] Aaron: Oh, wait. Yeah. Right. Okay, totally. I wish I knew of a resource, and it’s nice.
[48:49] Sarah: So in the last year, around about a year ago — GPT-5 came out.
[48:57] Aaron: That’s crazy. That was four centuries ago.
[48:59] Sarah: That was August. What happened in June?
[49:04] Aaron: Wait, what is it? Wait, what’s this—
[49:10] Sarah: In June, there was—
[49:10] Aaron: Sorry.
[49:10] Sarah: ...CAISI, the US AISI, but something CAISI. I don’t know why I spent so many words talking about this in the blog. It’s actually not that interesting of a development. And then we had the federal preemption thing ended up dying in July last year.
[49:24] Aaron: Oh, right. Wait. So I just pulled up your blog post. Oh, wow. It’s like a whole list. I just—
[49:29] Sarah: Yeah. There’s a list.
[49:34] Aaron: Yeah. So I guess looking at this, broadly, slightly pessimistic because of — I guess the main thing is the administration seeming to be actively picking sides in a way that I think it wasn’t obvious, at least to me. Picking sides between labs, sorry, to be clear, in a way that wasn’t obvious to me a year ago.
[49:58] Sarah: I mean — oh, between labs as in they’re picking OpenAI over Anthropic.
[50:03] Aaron: Yes, and to be clear, this is literally importantly informed by my priors, but also I think it’s been a substantial update. I think Trump respects wealth enough to make it — oh, which is not a good reason to think that he would be on a level playing field, but evidently he’s not afraid to antagonize even people who are billionaires, which usually he’s pretty friendly to billionaires, but not in all cases. I don’t know.
[50:41] Sarah: Yeah.
[50:42] Aaron: That’s not a good theory of why a president should be neutral, or at least neutral as a starting place, with respect to various companies. But it was maybe a vibe that I was sort of hoping for.
[50:56] Sarah: That’s not an axis I’ve been tracking, of “is the government displaying favoritism towards one particular company?” I haven’t really thought about that.
[51:06] Aaron: Oh, okay.
[51:06] Sarah: I was thinking of it in this maybe overly simple paradigm of “is the government situationally aware or not?” I have this—
[51:17] Aaron: Can you say that—
[51:17] Sarah: ...I have this assumption that if we achieve true government situational awareness, by which I mean the government is AGI-pilled and thinks about AI in vaguely the same way as me and a bunch of my friends, they will in fact just converge on vaguely sensible things, and nothing that came before will actually matter that much because it will be such a substantial update to them. But if they were previously embroiled in a bunch of petty drama where they are mean to Anthropic and they like OpenAI or whatever, they’ll just have an epiphany or a moment of clarity where it will all become clear. And they’ll be like, “We know what we have to do.” And that’s not to say that necessarily they would do the correct thing, but I’m more saying I think that would be a watershed moment where everything that happened before doesn’t necessarily really matter. And so when I was writing this blog post, I was really primarily tracking this situational awareness thing and the extent to which it seemed more or less plausible. Not that it’s a total discrete moment or whatever.
[52:11] Aaron: Totally. I think this is a better model for the national security state specifically. It’s like when you look at the revealed preference of the American Republic, it’s like what actually people care about, or what the powerful people care about. It’s the Fed and the military.
[52:29] Sarah: Mm-hmm.
[52:29] Aaron: And the national security state broadly. So that said, there’s just no getting around the individual who is Donald Trump as president for the next couple of years. He is just — what the president personally wants to do just does matter tremendously for a lot of things. And like—
[52:51] Sarah: Okay. I don’t like Donald Trump for all the normal reasons. I’m a broadly liberal person, et cetera. I would not have voted for him if I was an American, whatever, but for AI safety specifically, he’s just so high variance. I just can’t comment on whether he is in fact better or worse than, say, Harris would’ve been for AI safety because I literally feel like he could just roll out of bed one morning and be like, “I don’t like AI anymore.”
[53:18] Aaron: Z—
[53:19] Sarah: That’s a possibility that feels like it’s on the table, which it wouldn’t have been for any other president who’s just a little more sane and more predictable. He’s unpredictable, but in a way that could just be massively in our favor. I don’t know.
[53:32] Aaron: No, I see where you’re coming from. I guess I would just say that if that happens, it’s not going to be because he agrees with you about reasons to be concerned.
[53:43] Sarah: It could. He could get personally—
[53:47] Aaron: Yeah. It could. Yes, it is not literally physically — there’s one Everettian branch out of 10 to the 10 to the 10 to the 10 to the trillion that is this happening.
[53:58] Sarah: I don’t think it’s that unlikely. I remember there was a video clip that went around ages ago. I feel like the AI safety community tends to grasp at any little thing that looks like it’s directionally good or whatever, but there was this one clip where he’s on a plane and he’s being interviewed by some random journalist, and they ask him about superintelligence, and he refers to it as the rabbit that we can’t let get away, or... Do you remember this? I’m pretty sure that was the phrasing he used. And this was one thing out of a million random things he said that day or whatever. But it’s not impossible to me that he could just be like, “Whoa, there is this thing these companies are making that seems really powerful and scary, and is a threat.” At least in my head, it’s a threat to his power, but the actions you might take to prevent loss of control of AI as the president could still be broadly good, even if they are motivated by a desire to not—
[54:43] Aaron: Yeah.
[54:43] Sarah: ...play the crazy AI. I don’t know. There’s something there. It doesn’t seem like one in 10 million. It seems like—
[54:50] Aaron: No, yeah. Sorry.
[54:51] Sarah: Oh, really?
[54:52] Aaron: No, the really unlikely thing that I was claiming, just to be clear, is where he — or even if we didn’t even talk about the administration broadly — but I don’t know, actually, let’s just talk about Trump specifically. He comes around to a worldview that is based on reasons. That is the thing. And that then leads to sort of rational action. I agree that he could, for whatever reason — maybe Sam Altman or Jensen Huang, they come around for other reasons or something, and then they have... sorry, that’s actually not relevant or load-bearing, so forget about that. I agree that the chaos is a thing and that could potentially be good. I just disagree that it’s going to be downstream of him having reasonable takes.
[55:37] Sarah: I think if it’s downstream of him just being scared, then that seems like it could be fine. He doesn’t have to read LessWrong and understand the arguments for AI risk. He could just be like, “Wow, my panel of scientific advisors are telling me real scary things, and I don’t like that Dario guy, or that Sam guy or whatever. I’ve got to do something about this.” That seems like it could lead to some not—
[56:01] Aaron: Yeah.
[56:02] Sarah: ...maximally bad outcomes.
[56:04] Aaron: No, I guess I would rather live in the world where the president — and I think this has been the case for — I think he is more or less, probably... Listen, I’m not a historian of presidents. I don’t know who was the president in 1840 or whatever, but at least in recent history, I think he has been the least reasons-responsive in an important sense, such that other administrations you could model as sort of rational actors, at least broadly, in a way that I don’t think you can with Trump, and I think that’s actually negative, not positive.
[56:41] Sarah: Yeah. I think it’s probably going to be negative, but with the variance thing, there’s some small chance it’s extremely positive. Maybe this doesn’t cash out to be any better than a different president. I don’t know.
[56:54] Aaron: That’s also true with — you can imagine a whatever. In my mind, I’m thinking of Obama, but even Biden, even anybody but Trump in recent history. You could also imagine them doing some sort of very high pro-safety action, but that would be not because of random variance. So it’s not clear how it checks out. If your worldview is like, “Oh, something radical needs to happen,” then it’s not clear whether you’d prefer reasons-responsiveness or high variance. I think that’s the actually coherent way of saying it.
[57:35] Sarah: Yeah. I guess with, say, the Biden admin, it seemed like they were vaguely, or they were at least concerned about AI ethics issues or like—
[57:44] Aaron: No. Yeah.
[57:45] Sarah: But I wasn’t expecting... It was cool that they had that executive order with the reporting requirements or whatever, but it kind of always seemed to me like they were just going to keep doing these fairly pedestrian things. Whereas with Trump, it seems like they are, at least in posture or aesthetically, extremely accelerationist. But because of this high variance thing, and because actually if you look at the content of some of the stuff they put out, it’s not as accelerationist as you’d think — I think that that pro-innovation thing is kind of not a very good signal of what could actually happen in the future. Some people are like, “Oh, the Trump admin is so bad because they’re just so hawkish and they want to beat China, and they just talk about innovation all the time.” And it’s like, that’s true, but I think you have to kind of look beyond that to, A, the high variance thing, and B, the fact that the actual substance of, say, the AI action plan or other things that they’ve put out is... And it’s also borne out in this recent staged deployment thing. They are not in fact totally foot on the gas pedal. They just act like that because it looks cool.
[58:49] Aaron: One thing I will say, I think it’s very important — the actual chronology of the Biden versus Trump thing is very important. What we’re not observing is Biden in the current 2020-something world.
[59:03] Sarah: Yeah. That’s true. Yeah.
[59:06] Aaron: And likewise, fair, we did not observe — I don’t think Trump would be near... I think he would probably look directionally more or less ASI-pillable in the era of Biden just because of chronology stuff.
[59:25] Sarah: I don’t think—
[59:25] Aaron: Not ASI-pillable. Sorry. The whole world is sort of moving in the direction of—
[59:30] Sarah: Yeah. I think that the—
[59:32] Aaron: ...consequentialism.
[59:33] Sarah: ...I would guess the political right wing are way more ASI-pillable than the left.
[59:39] Aaron: No, I agree, and I actually think deferring to woke or whatever is a real problem. I also — yeah, you still have the reasons-responsiveness thing. I do think the president is pretty... I guess in terms of actively doing things that undermine national security, I feel like Trump is uniquely capable of doing things that actively undermine national security. I also think you’re giving a little bit too much credit to actual ideology here. I think it’s pretty clear that Trump doesn’t actually care in any important sense about the sort of natural extension, or the very close, basically almost identical extension of where his rhetoric points. Because if he did, he would not be deferring to Jensen Huang on export controls. It’s very difficult, and I’m trying not to say this as a partisan. I’m trying to say this insofar as I can, just my overall actual impression, but it’s hard to give him too little credit, I think.
[1:00:44] Sarah: I wouldn’t say that I’m trying to give him credit. I’m just trying to think about how to model him or predict him.
[1:00:49] Aaron: No. I get that. I guess I just think even the ideology model isn’t that good.
[1:00:57] Sarah: Yeah. I don’t know. I just think there’s something to this argument that the left is uniquely un-AGI-pillable, even though I would broadly agree with them on most things. They just have this baked-in skepticism of everything and —
[1:01:11] Aaron: Yeah.
[1:01:12] Sarah: ...and there’s all this—
[1:01:13] Aaron: There’s the online — sorry, I cut you off. Sorry.
[1:01:17] Sarah: Yeah. I don’t know. For some reason, this polarization has emerged between the ethics-y people and the safety people, and maybe that’s just kind of a weird contingent fact of history, and it didn’t have to happen that way, but it has happened that way. And so just — there’s at least something to that, that maybe this means that if a singularity happens under a Republican admin, this is slightly more likely to... Yeah.
[1:01:42] Aaron: Yeah. No, I think this is true, but Trump’s flaws just point in the other direction, however.
[1:01:49] Sarah: Yeah. It probably is outweighed by him being crazy, but I feel uncertain.
[1:01:57] Aaron: Yeah.
[1:01:58] Sarah: Yeah.
[1:01:58] Aaron: Okay. We should probably wrap up soon.
[1:02:00] Sarah: Cool. Seems like a good place to end.
[1:02:02] Aaron: Do you have any final — do you have any last words?
[1:02:05] Sarah: Do I have any last words? People should not be too overconfident about their AI takes. It might just all be totally chill.
[1:02:16] Aaron: I don’t know if I... Well, we can’t get into that because it’s 3:26.
[1:02:19] Sarah: Yeah.
[1:02:19] Aaron: Okay. Sarah, it’s been lovely talking. We will redact parts as appropriate.
[1:02:24] Sarah: Yeah.
[1:02:24] Aaron: Maybe we’ll keep in this part that says we’re redacting parts as appropriate.
[1:02:28] Sarah: Yeah. That’s appropriate.
[1:02:28] Aaron: And go enjoy whatever you have to go to.
[1:02:32] Sarah: Cool. Take care. Thank you for having me. Bye-bye.
