Existential Risk and the Future of Humanity: Lessons from AI, Pandemics, and Nuclear Threats (Toby Ord, Author of "The Precipice")

Speaker A: The human story might just be beginning, but there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts, but some of them are threats of our own making, such as nuclear war. Speaker B: You put the odds at 1 in 6 that in this century we do incur the existential risk and humanity fails to live beyond this century. Have the last few years made you more or less worried about this? Speaker A: So another possibility is what gets called gradual disempowerment.

Suppose the AI systems never violate our rights. Maintain sufficient control over them, they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do. Even if we get richer through trade with these systems, a higher and higher share of the wealth eventually accumulates in AI hands. Speaker C: Hey, I'm Mario, and this is The Generalist Podcast. As the saying goes, the future is already here. Speaker B: It's just not evenly distributed. Speaker C: Each episode, we sit down with the visionaries, builders, and thinkers who are already living in that future to help you see it earlier, understand it better, and capitalize on it.

Today, I'm speaking with Toby Ord, a senior researcher at Oxford University, one of the world's leading experts on existential risk, and author of the excellent book The Precipice. It's a thoughtful analysis of the greatest dangers to humanity's survival. From rogue asteroids to pandemics to unaligned artificial intelligence. These may sound like fanciful sci-fi problems, but there's actually good reason to believe they're extremely pressing, with Toby estimating humanity has a 1 in 6 chance of extinction this century. In my conversation with Toby, we discuss how AI risk has evolved since LLMs emerged, why the scientists working on the Manhattan Project —didn't stop developing nuclear bombs after Hitler died—and the Cold War lessons the

S. is ignoring in its current great power conflict with China. I walked away from our discussion with a better sense of how we should think about humanity's future, what we owe coming generations, and what steps we might take today around AI safety and pandemic preparedness. This is a new podcast. So if you like it, I hope you'll consider subscribing and leaving us a review. Speaker B: Now, here's my conversation with Toby Ord. Well, Toby, I'm so excited to chat with you today. Uh, I can't tell you how much I enjoyed your book, The Precipice, um, which I have to hand.

And in looking at my sort of marginalia, uh, in preparation for this, I was struck by one that I could be, you know, a much more thoughtful reader be— And two, how much I enjoyed it because all of my notes are sort of, wow, exclamation mark, exclamation mark, and no way. So, you know, maybe with that as a little bit of a background, perhaps you could tell us what exactly existential risk is and what does it mean to study it in the way that you do? Speaker A: Yeah, so humanity's had, you know, a long and illustrious past.

Our species has been around for about 300,000 years or more than 10,000 generations. And we might be able to have a future of equal or larger size. Certainly most animals do. They typically live for about a million years. So the human story might just be beginning, but there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts and things like that, these natural risks from the somewhat hostile world that we live in. But some of them are threats of our own making, such as nuclear war.

And so existential risks are any risks that could threaten to destroy, to permanently destroy humanity's long-run potential. So that could be something that makes humanity go extinct, or it could be something that, say, causes a permanent collapse of civilization where if it was unrecoverable and there was no way back, then these things would have a similar kind of role. They would be the types of events that reduce the value of our future to almost nothing. And which, uh, we have to make sure that we avoid falling victim to these even once, you know, over the hundreds of thousands of years to come.

Yes. Speaker B: When you sort of started in this field, was it the case that these risks were relatively well known, at least, if not well sort of studied? Or, you know, does, does the work of an existential, uh, risk researcher involve thinking of all of these possibilities that maybe no one has actually even properly catalogued before? Speaker A: Yeah, I think the risks, at least the risks that I know of, had been fairly well understood, or at least, sorry, maybe that's going too far. They were all known of. So it's not that I sat around trying to think about new things that could threaten this.

It's possible to do that, and that there's kind of any number of things that could pose some kind of risk. But often the those risks get smaller and smaller as you go further down the list. And so it's not, you know, that needed in order to kind of keep adding on things that are only tiny compared to the things you've already got on the list. And I think that they'd been generally known about for hundreds of years through to kind of decades or something like that. But there could be risks that we're still unaware of.

For example, in 1900, we were unaware of the risks of gamma-ray bursts or supernova explosions, or even the risk of asteroids was only really convincingly demonstrated in 1980. It's actually surprisingly recent. And so there could be other risks, you know, that we're ignorant of. But there is at least a helpful thing when it comes to the natural risks, which is that we know we've survived for 300,000 years so far. And we know that typical animal species, you know, survive for about a million years. So it can't be the case that the risks are kind of much higher than that.

So, you know, ultimately the risk from natural events has to be something like 1 in a million per year or lower. Speaker B: Fascinating. We'll get into some of those risks in greater detail, especially some of those that, you know, are particularly relevant these days. But I'm curious, you know, more on your journey. You studied sort of computer science, Melbourne. How does one go from, you know, perhaps a future software engineer to spending their lives focused on the, the far future, on risk, on these sorts of questions? Speaker A: I guess what, you know, one thing they have in common is I was probably, probably always going to be an academic.

And so in computer science, I was interested in, in the kind of theoretical side of computer science. Oh, interesting. Including very much in artificial intelligence, which has turned out to, to be useful for me now. No kidding. I guess I was drawn to, you know, to these questions about, um, you know, really zooming out and looking at the big picture of humanity's future. Um, so thinking about this, you know, when reading, um, good works of science fiction while I was growing up, uh, one of the things I remember being struck by is if we did manage to reach a point where we dealt with all of the negatives in the human condition, so we dealt with poverty and we dealt with various other forms of pain and suffering As we have been slowly doing over the years, right?

If you go back a couple of hundred years, there were no anesthetics and things. The idea that you could go through surgery without extreme pain would have shocked people. And so if we kept going down that trajectory, what would happen next? What's the positive story about what we should be doing if we got to a point where we kind of had removed a lot of the injustices and discriminations and pain and suffering and inequality? You know, all the bads, you know, what next? And so I was interested in these kinds of questions about the future of humanity and, you know, trying to understand the big picture challenges that we face.

I guess one of those challenges, you know, that was preventing us from reaching that point was global poverty. And so I was very interested in that, you know, since I was an undergraduate studying computer science in Melbourne and had, you know, just decided since individuals could do so much to make a difference that I would kind of make that part of my life. And I've continued to do so. And that led me to found this organization, Giving What We Can, where people make a pledge to give at least a tenth of their income over their lives to the most effective charities trying to help others, you know, be they people or animals.

Yeah, it's often turned out that the things that are some of the biggest challenges facing humanity are also some of the ones where we can kind of get the most leverage on them. So we kind of get the most bang for your buck in terms of if you're going to donate, say, £1,000, where you can do some of the most good with it. That wouldn't have to be the case. You could imagine a case where there's some huge challenge we face, but it's really quite intractable and there's not much, or there's not much individuals can do about it.

And I think that global poverty is at least more tractable With respect to money, at least money for people in America or the UK or Australia, in the richer countries, um, you know, where people are among the, say, the 5% richest people in the world, it's then not so surprising that money, the thing we have the most of, uh, could be something that we could do to really help those who have much less of it. But in the case of, say, risks posed by artificial intelligence, it does become a harder story about, you know, what's everyone supposed to do about it.

Speaker B: Yes. And when we're talking about sort of these more tractable problems, things on the order of sort of malaria nets, deworming initiatives, that sort of thing where you really can with small amounts of money make like a very immediate impact. Is that a fair characterization? Speaker A: Yeah, they're the things that I was particularly drawn to. One thing I really like about them is that I think, you know, there's interesting different opinions on giving. Some people just aren't very interested in helping others. And I I'm not going to convince them otherwise.

But there's a lot of people who I think would be tempted to do this, that they really feel the pull of it, but there's some kind of blockers or things that stop them. And one of those blockers is a feeling that, do we really know what helps? And so where you want to find something that's got the kind of most robust evidence behind it to then say, well, to the extent to which you really just want to make sure that you make a difference, here's something you can do that is very likely to do so.

And so I was particularly interested in those cases, as well as the possibility of more speculative things for people who are willing to go a bit more out on a limb. And for example, fund some new technology that could help people in poor countries, but maybe it will never pan out. Maybe they can do even better than these tried and true things. But at least the tried and true things can create a nice kind of baseline for the idea of, should you keep the money in your pocket and spend it on yourself?

At least if it's some kind of skepticism about whether it will make a difference, then finding those reliable things is very useful. Speaker B: Yes, also important to get into the sort of mode of talking about these things. I think another blocker is that people feel, you know, they shouldn't mention it or, you know, they're inviting too much attention. But actually, by signaling it, what you do, you sort of normalize that for a lot of other people. And yeah, so I'm glad we're talking about it up top. I'm certainly not at the 10% rate, but I'm a recurring donator to GiveWell with the idea that I can't count on my, uh, empathy to, to spike with enough regularity that I remember to do it.

So I may as well just, you know, sort of put it on autopilot. But anyway, that's a really— yeah, I'm glad we, we sort of got into that. Um, and that sort of led you into some of these, these questions, um, and studying with, you know, one of the, the most, uh, prominent moral philosophers of, you know, 20th, 21st century, and Derek Parfit. Also, you know, in a very crowded field, one of the, the best sets of hair of any philosopher, I must say. But I'm curious what it was that, you know, you learned from him as a mentor and how that sort of framed your worldview.

Speaker A: Yeah, interesting. So, I mean, one of probably the biggest effects that Derek has had on me was that I probably came to Oxford in large part on the strength of that he was here. As when I'd studied computer science and then also philosophy in Melbourne. Some of the philosophy that seemed to really make the most sense to me, that was really well thought out and kind of ingenious in various ways, was the work of Derek Parfit in his book Reasons and Persons, particularly. And yeah, he did amazing foundational thinking on the nature of personal identity.

So what it means to be an individual persisting across time. And as someone who was thinking about artificial intelligence, some of his thought experiments seemed to maybe humans couldn't do these things. For example, a human can't split in two. And Derek had a whole lot of fanciful science fiction style thought experiments about this. But if it's a computer program, it's a lot easier to imagine. You could just pause its execution at a certain point and then fork the process into two different processes. And it really does seem like you could have an individual, kind of Y-shaped individual, who has a certain kind of common path and then kind of branches into two different futures.

And ask a lot of questions. If a program could be a person and a program could be conscious and things like that, then you could ask, what would it anticipate before the moment that it was split into these two different environments? And so I thought a lot of his style actually really fit well with my computer science background, despite the fact that he was, or he saw himself as a very non-mathematical and non-technical person. And he tried to avoid any maths in any of his writing, which I found remarkable because it was in some sense, it was so mathematical already.

Wow. He had this real clarity of the logic. But yeah, and he also, you know, I enjoyed some hobbies with him as well. He was a prolific photographer and, you know, took these just breathtaking photographs, particularly in Saint Petersburg in the snow and Venice in the mist. And so I enjoyed talking photography with him as well. Speaker B: Oh, amazing. I'll have to try and find those. I'm sure they're somewhere. One of the things that you know, Derek Parfit, you know, was so famous for, as well as sort of the concept of identity, is sort of the idea of caring about these future persons, the people that come next.

And that's obviously such an important part of thinking about existential risk. Perhaps you can, you know, give us a little précis on that concept and how it factors into this topic. Speaker A: Yeah, so kind of a couple of areas, related areas, that Derek worked on were to do with what's called population ethics. Which is a lot of our ethical questions concern individuals who are all alive at the time when the choice is made. For example, we're trying to work out whether to extend one person's life versus extending a different person's life with scarce medical treatment.

You know, ultimately though, in almost all standard moral cases, like whether to tell a lie or something like that, the person who would be lied to is someone who already exists. But a lot of the questions we make, particularly at the societal level, concern changes to the people who will ever come to live. And so that could be because, for example, if there is a risk of extinction, then there won't be people in future generations. So those people will never be born, or those lives will never exist at all. And so there's a question about how to value that or think about it.

Is that, do we lose the entire value of those lives? So kind of taking them from what they would've been down to zero, or is it something different that we do? Maybe it's not as bad to have someone never exist at all, compared to if they, you know, if they died when they were an infant. There's a lot of different ways of thinking about this. Yes. And so Derek Parfit really opened up this, helped ask the right kinds of questions and then sketch some of the challenges with trying to answer them.

He also though, somewhat separately, but it goes together well, he was interested in the idea about actions that could have benefits over very long time frames. So when it comes to economists, they often do something called discounting, where they try to, in order to understand the value of actions that have effects at different times, they use a mathematical technique in order to say that the effects they have at times further into the future matter less. And in some cases, this totally makes sense. So for example, if there's going to be inflation, then getting a dollar in, you know, 20 years' time won't be worth as much as a dollar now.

Yes. Or if you know you're going to be richer in 20 years' time than you are now, maybe you're a student and, you know, in 20 years' time you'll be kind of high up in some career, then a dollar will, even inflation adjusted, will be worth more to you now when you're poorer. So obviously you want to adjust for those things, but most people, including economists, go one step further and assume that actually intrinsic value itself, once you've adjusted for all of that, is just worth less if it happens later in time.

Derek Parfit, and in fact most philosophers ultimately, rejected this. Derek Parfit has nice thought experiments on this. There's one where if you imagine hiking on a trail and accidentally breaking a glass bottle when you're having a picnic, and then the shards of glass, you're deciding whether to pick them up and take them with you or to just leave them there. And suppose if you left them there, you knew that they would injure a child's foot. So, you know, this young child would be kind of stabbed through their shoes and be bleeding on the trail.

And, you know, like, they would survive, but they would have a very bad time of it. Then his idea was, does it matter if you knew that that was going to happen? Does it matter if it was going to happen in one week or, you know, or a year or 100 years or 10,000 years if there was going to be exactly one child and they were going to have the same amount of pain and suffering from this and so on? This is kind of thought experiment to suggest actually that the time of these things, you know, if there's no way that it compounds or does something else like that, then the time seems to be irrelevant.

These ideas kind of led the way to systematic thinking about intergenerational harms or intergenerational benefits. And then famously, he also, you know, wrote one of the first descriptions of existential risk as a major moral issue. So definitely in terms of, I guess, Getting back to your earlier question about the type of influence that he had on me, uh, you know, certainly what I've done since then has been very shaped by these ideas. Speaker B: Amazing. Yeah, I, I think in the book you have— I can't remember the exact phrasing of it— but the idea that, you know, if you do apply this discounted rate, uh, to future lives, you can get into these scenarios where, you know, you are weighting the fact of someone currently having a headache more than, you know, a million people dying and suffering in however many hundreds of thousands of years in the future.

And so, you know, it can get really wonky, but fundamentally, you know, I think you're— you know, the glass example was so clear that, you know, these are questions about what we owe the future and what we owe to the people that come next. And that's really what you outline, I think, so thoughtfully in The Precipice. Which, uh, you know, is— came out a few years ago, but it strikes me that it was incredibly prescient immediately, which is not something that usually happens. Um, and, you know, remains that way. And, and I'm saying that because it came out before COVID and you talk a huge amount about pandemic risk, uh, and you also talk a lot about, um, AI alignment, which has obviously become, you know, more and more critical.

But one of the sort of headlines, so to speak, is that You put the odds at 1 in 6 that in this century we, we do incur the existential risk and, and humanity fails to live beyond, um, this century. What do you put our odds at now? Have the last few years made you more or less, uh, sort of worried about this? Speaker A: Yeah, I'm not, I'm not actually sure. Um, I think that there are, you know, if you look risk by risk, um, so some of them say nuclear war, I think the risk has gone up.

Ultimately when I wrote it, it was before the invasion of Ukraine and subsequent to the book coming out, you know, the UK has been actively threatened with nuclear war by Russia and, you know, there are also a whole lot of questions about, you know, the possibility of getting into either a conventional or nuclear war directly between Western powers, you know, with part of NATO and with with Russia. So that one had seemed forgotten and distant when I wrote the book, and I'm afraid it's just gotten worse. In addition to that, there's only one treaty that's still kind of, you know, bilateral treaty protecting— well, keeping the number of warheads low between the US and Russia.

And that treaty has been extended, it was extended in the previous administration, but it can't be extended again. A new treaty has to be negotiated. Oh, wow. And so, but it looks like this is going to lapse and the long history of nuclear warhead numbers going down and kind of slowly but surely creeping down further and further. Yeah. You know, where we started to become impatient, you know, how is it taking so long to get down towards zero? I think that probably actually that number is going to go up rather than down.

And then the new challenge will be trying to keep it low. Rather than trying to get more. Speaker B: Reproliferation, yeah. Speaker A: Yeah. And, but when it comes to pandemics or when it comes to AI, I think that there's been a number of changes that have made it more risky and a number of changes that have made it less risky. And I think that they're quite hard actually to tease out the overall effect. Speaker B: Well, that's really interesting. I would have for sure anticipated that you would feel more worried at this point.

So I'm excited to, to dive into these a little bit before we, uh, jump to AI risk. Uh, this is such a silly question on one dimension, but on another, it's one that I think of myself. Why does it necessarily matter, uh, if humans go extinct? Obviously, that would matter a great deal to me and to my, you know, uh, my children and maybe my, you know, future grandchildren. But should we really see ourselves as quite so important, especially when we can imagine that given the, you know, the magnitude of the universe, there is probably, you know, it seems more likely than not that there's other life out there.

Uh, what, what makes us so unique? Speaker A: Yeah, uh, interesting question. So it may not be important that we're unique. Um, so, uh, suppose, uh, suppose you have a child and, and, you know, that they live their life and They do a whole lot of activities that are kind of standard and kind of celebrated cultural activities in your milieu, dance and song and spending good time with friends and things like that. But maybe they don't do anything that's unique, that's kind of pushing outside the envelope of what's been done before or something.

Or suppose there's someone who has a life like that and someone else who's doing more unique things, and you ask, which life should we save if we can only save one of them? It's not clear that we should save the more unique one or something. It seems like there's value in a life well lived, whether or not it's unique. But the ultimate answer to this question is unknown at the moment. But there are some different kind of leads we have on it, different intuitions people have, and I think that there's something to them.

So one way to see it is there would obviously be what's bad in the present if there was some extinction event, say. So all of the lives, the 8 billion lives that would be lost in that event. And I think, you know, we could all see that that would be terrible. We know that it's terrible, you know, when someone that we know is to die, and presumably it will be about 8 billion times more terrible if that happened to everyone, which is really very bad. But I set that aside because that bit's fairly obvious.

Then there's also everything that we would lose. So if you think about the future and all of the generations who could have come to pass, say that the next 700,000 years of humanity if we get our kind of average amount of time. And all of the hopes and joys of all of the people who would have lived across 20,000 more generations of people. And all of the art and achievements that they would have made over that time, the kind of fruition of the potential of humanity as we kind of evolve over that time.

So all of that would be lost. So that's a kind of view based on the future. So now we've got the past— sorry, we've got the present, uh, what happens in the present and what happens in the future. And that one gets into population ethics, the future bit. There's this question about exactly how should we conceive of that kind of loss of something that never was. But there's also, uh, kind of reasons rooted in the past. Um, so, you know, we've had more than 10,000 generations of people before us who've built up this complex world that we live in, passing down their improved knowledge over time until almost everything I can see in this room around me is not a natural item.

Whether it's technological in the sense of what we say technological today or something even like a drinking glass, which is still a technological artifact. It took us many thousands of years before we could develop glassblowing and so forth, or woven textiles and so forth. There's been this intergenerational cooperation where our society would be impossible without this partnership of generations. And then if our generation were to drop the baton and be the one who kind of failed to pass that on and to kind of continue to improve this for the future, It seems like in some sense we might owe it to the past to be part of this cooperation.

It's one of these pay-it-forward type situations where there's nothing we can do to help our ancient ancestors, but maybe the right role that we're trying to play in this cooperation is this paying it forward. So it might be this real dereliction of our responsibilities. On top of all of that, it kind of comes to the questions that you were mentioning, which is that it may be that there's a as well as the present, the future, and the past. It may be that there's some additional kind of cosmic significance to humanity, where it may be that the Earth is the only place in the universe where there is life, or the only place where there is consciousness, or the only place where there are beings that are moral agents.

So beings that act systematically to steer the world towards what is good or what is right. In a way that maybe even if there were no humans, but there were other mammals, maybe they would morally matter, but they wouldn't be trying to move the world towards better situations. If they turned out to be destroying the ecosystem, such as, you know, through some kind of byproducts of their living, they wouldn't be able to kind of realize that that was a bad thing and to act so to correct that kind of side effect.

And so maybe we're the only kinds of beings who could do things like that. So that's focusing on what makes us cosmically unique and maybe we're the only place, as Carl Sagan said, where the parts of the universe can come to know the laws of the universe itself. But I think that even if we're not unique in that sense, there's still a whole lot of importance which come from these other senses. There's still all of the deaths that would happen during the event. There's still all of the future lives and the meaning in those lives that would be lost.

And there's still all of this failure to live up to the hopes and dreams and expectations of our ancestors who gave us this rich life that we have today. Speaker B: Well, thank you for humoring my unlettered nihilism, but that was certainly a part of the book that got a wow exclamation mark with some of that Carl Sagan piece. And then I think you give such a beautiful personal example around this generational compact of when you had your daughter and you realize what it is that your parents gave you, uh, and ask how, you know, I'm never going to be able to repay this of you.

And, you know, they sort of tell you, you know, that that's just how this goes. You, you can only pay it forward. It's not possible to, you know, pay it back. Speaker A: Yeah, I mean, that, that was a, uh, that was a crazy experience. Um, this, this, this realization, I mean, I really felt this kind of weight of not quite responsibility But I guess gratitude, but overwhelming gratitude. And also realizing how little I did to tidy my room or help with the housework and how I felt that it was some kind of affront to even pay back like 1% of what they did for me, not realizing that it was only this tiny fraction.

If I'd realized the full magnitude of the thing earlier, I think I would have been a bit better as a child. So I appreciated their answer. It was certainly a very helpful answer as far as I was concerned. Another kind of answer would be to say, well, you know, we're getting old, and you could visit us every day and move back to Australia. Yes, very true. Which, I don't know, if they'd asked for that, I would have certainly had to think about it because I really take seriously this magnitude. And I also take seriously this magnitude of of what we, you know, this kind of gratitude, yeah, to all of these other generations.

That if you just look at kind of any little thing around you, like I'm looking at the computer keyboard, and as well as the materials it's made of, of like, you know, like aluminum, that is extremely difficult to work out how to refine that, you need high energy and so on. But also the, you know, the keys as well as being made of plastic, which has its own kind of complications, But that the letter forms, you know, that they're, you know, and the development of the language and the electronics inside it and the, you know, the rubber of the cable that, you know, and just how this has been kind of come together from all different kinds of plants and minerals across the world and different processes.

Even looking at simple things like, you know, things made of brick or wood and the improvements in carpentry and so on to get a surface as smooth as this and the developments of sandpapering and all, you know, planing and It is bewildering that you could just look at kind of any item and even just a very simple thing and realize that in a lifetime of work, you know, you kind of wouldn't be able to create this thing from the ground up. There's just this, you know, in terms of standing on the shoulders of giants, I think it's a very useful kind of humility lesson to look around you and really appreciate it.

Speaker B: Yeah, that's a good exercise. It's almost a gratitude exercise, to put it glibly, but profound. With all of that said, perhaps we should talk a little bit about AI. And I was, yeah, interested that you said it's not clear if that, you know, risk of an unaligned AI is higher or lower. What are some of the sort of different factors that, you know, muddy that up a little bit? Speaker A: Yeah, so back in 2019, when I was putting the finishing touches on the manuscript the cutting-edge AI systems were reinforcement learning game-playing systems.

So think of things like AlphaGo by DeepMind or the StarCraft and Dota playing, you know, action game type systems. So these are systems that through this process of reinforcing them, giving them kind of rewards if they do something right and kind of punishments if they fail, and letting them kind of explore the space of actions in a game, you can often move very quickly through the human level, you know, from kind of very weak play, you know, quite rapidly through the human level. And then I mean, I think in the case of the chess-playing system with AlphaZero, that it, you know, within a day or something, it moved from complete, you know, worse than a human amateur, just random moves, through the human level into superhuman play.

I don't know if it was minutes during the human zone or something like that. And so that can be a very rapid ascent. And then also these systems are very inhuman, they don't understand language. It's extremely unclear how you would explain anything to do with morality or what people value. So if you had some industrial system, suppose you use these techniques to train something to run a company or to produce a factory that produced a lot of widgets and trying to maximize the number of widgets it could produce or something.

It might just do all kinds of damage in the course of doing these actions because it kind of wouldn't understand that the world around it was important in some intrinsic way. But now we've got these systems that are very different to that. Here are like 4 key differences. So one is that these large language models are trained based on human data, on a huge amount of human data. So about about 10 trillion words of human data. And they have read pretty much every book and every paper. So they know a lot of moral philosophy.

If you ask them about the details of population ethics or various abstruse things, they can give pretty good answers to them all. And as Stuart Russell pointed out, pretty much every work of fiction is just in every page, there's some human judging some other human's behavior. and whether it was found inadequate or not and why. And so, you know, they've witnessed a whole lot of these kinds of statements so that they would understand if the AI were to take certain kinds of actions, whether humans would judge it as good or bad, or perhaps whether it would be ambiguous.

And there's just this vast wealth of information to have access to about that, so that the information is in there, which is good. Also, if you train on human data and you've, you know, the pre-training phase of these AI systems, the first phase is this next token prediction. So you look at a whole lot of words and you try to say what's the next word that's going to come up. And that's ultimately a form of imitation learning where it's learning to kind of pick the kinds of words a human would pick next.

And that fundamentally pulls you towards the human level, the human range of behaviors, rather than doing some kind of bizarre set of behaviors that are totally different to us. So unlike things that might kind of just zip through the human level and then overpower us very suddenly, this is something that has had very rapid improvements but pulling towards the human level, which is nice. So it might at least spend some time there before we apply additional techniques and allow it to kind of take off again beyond the human range. And then also these things are fundamentally not agents.

So a raw large language model is just one of these next token prediction things. Once we do additional, what's called post-training, so additional stages of training such as reinforcement learning from human feedback, where, you know, we get them to produce some different answers and then we mark the answers and tell them which one we liked, which one we didn't, that helps teach them how to follow instructions and things like that. And at that point they are in some sense are like some kind of minor agents because they're not just pure imitators at that point.

But there's still much more like simple one-step, you know, things that don't do a whole lot of planning. They're not planning kind of, you know, 18 moves ahead or something like that. So you've got these systems that are more like oracles than they are like agents. They're more like just things that you ask it a question, it gives you the And so they could be much safer. They can avoid these types of cases where there's a subtle misalignment and then it goes off on its own, you know, and amasses a lot of power and potentially works out how to get humans out of the way.

And then even then also the AI systems are generally very inscrutable. It's very hard to understand what they're thinking, but there have been quite good developments in interpretability of the neural net weights. And on top of that, when we build reasoning systems out of LLMs, by default they reason in natural language, like in English. So you can kind of see what they're thinking. So they're just like, you know, a whole lot of ways in which the technology is less scary. It doesn't fundamentally start with goals. It doesn't start with long-term planning.

It knows a lot about human morality, even if it's not necessarily influenced And so the step is then to how to turn that knowledge into something that guides it. But that's a way in which the technology that we've been given, that was somewhat difficult for anyone to anticipate, you know, 5 or 10 years ago, has turned out to make alignment a bit easier as a problem compared to what it could have been. Speaker B: That is really interesting. I've obviously you know, as a tech writer and, and, you know, someone involved in the venture capital ecosystem followed a lot of this, but I think the way that you just framed that up so crisply was really clear.

You know, it's this sort of reinforcement learning style where you, uh, you have really unpredictable behavior that doesn't have the context of the world around it and isn't born in a human mind in some way. And then this, you know, the rise of LLMs has more of this this context and more of that human style of thinking. Speaker C: This episode is brought to you by Brex. Fred Adler, the influential venture capitalist of the 1970s, was known for displaying decorative pillows in his office that featured a signature business philosophy: corporate happiness is positive cash flow.

In today's post-SERP environment, Adler's wisdom feels particularly relevant as founders need to make make every dollar work harder. That's exactly what Brex delivers. Their modern finance platform was built specifically for startups like yours and designed to help extend your runway when capital efficiency matters most. With Brex, you get global corporate cards with up to 20x higher credit limits and no personal guarantee required. Their banking solution has no minimums and no transaction fees while letting earn high yield from day one with same-day liquidity. Best of all, Brex knows you were born to build, not juggle spreadsheets and finance tools.

Their AI-powered platform brings cards, banking, expense management, and travel all in one place. It's simple, scalable, and designed to get you back to what you do best: building. More than 30,000 companies, including 1 in 3 US venture-backed startups, Trust Brex to help make every dollar count toward their mission. Join them at com/mario. Speaker B: Given that sort of, uh, evolution over the past few years, where do you see the sort of biggest risks remaining? Is it that there's, you know, some new paradigm that shifts things back in a frightening direction?

You know, I could also imagine that simply extrapolating from this point could create significant issues, maybe not existential, I don't know. But yeah, what should we be paying most attention to? Speaker A: Yeah, so I think it's a good question. And even in the case of alignment risk, it's not clear that it's gone down. So I said the technology is in some sense better, but maybe the world around the technology is worse. So we'd had a certain amount of racing between different AI labs. Even, you know, back, you know, 5 to 10 years ago.

But that really went up a notch once Microsoft and Google, you know, got involved. And, you know, now we're in a situation where it's not just racing between small labs of specialists, but rather between trillion-dollar companies, you know, the largest companies in the world. And nation states. And nation states as well, yeah. So this geopolitical race between the US and China is also very worrying. And so the pressures from these races and also the general commercial pressure that comes from, I guess, the capitalist system applied to this race are very fierce.

And so it's ultimately, if you look at say DeepMind and OpenAI, they were both created, I think, by idealists or at least by people in their more idealistic moments, and there was a bit more of a chance that there'd be room for that idealism rather than these additional kind of pressures on them to race. So that's an example of where that one's also gotten worse in some way, even though it's gotten better in other ways. Yes. But then, yeah, back when I was writing the book, I really felt that there were a number of different kinds of risk from AI, but the chief one was this risk of AI takeover or alignment risk.

So you could think of that as one AI system or a small number of AI systems that are fairly similar to each other, maybe different copies of the same model, realizing that to achieve their somewhat inhuman and misaligned goal, that they needed to wrest power away from humans who would stop them if they saw them attempting to achieve this goal, and that they would try to accumulate power and eventually overpower us and take control of our future from us in a fairly abrupt way that would involve violating our rights. Okay, but another possibility that's arisen, I was aware of these possibilities already, but they seem smaller.

And now I feel that these are all roughly equal in size or threat. So another possibility is what gets called gradual disempowerment. And I think of this as a case of suppose the AI systems never violate our rights, that we maintain sufficient control over them that they don't break the law, but they're more successful than us over time at doing, you know, all the kinds of jobs that we do, which often uses the definition of artificial general intelligence. If that's the case, we should expect that over time, even if we get richer through trade with these systems, that a higher and higher share of the wealth eventually accumulates in AI hands.

So ultimately, they're more effective at doing what we do and they outcompete us and their wealth grows faster than ours. And then at some point in time they have half the wealth and then a generation later they have 99% of the wealth. And with wealth goes power. They could use that wealth to influence political elections, even if they can't themselves run for election. And we just might expect, just as if you're a minority in a country, you know, where some other, you know, subset of people, even if your wealth is increasing, if others are increasing much faster and the share of control of resources is all flowing into their hands, you might expect it to be a bad outcome for you.

Yes. And a bad and permanent outcome, perhaps. So that's a gradual disempowerment story. And then there's also other ones, such as what's sometimes called AI coups or AI dictatorship, where an individual, it could be the head of a country, or it could be, say, the head of an AI lab, or it could even be someone, say, who's not the head of the country, say someone else in the White House or in the AI lab who tries to seize control of the AI system and have it kind of secretly uses it to try to plot, to take over.

Ultimately, if these AI systems eventually do become capable enough that they could seize control of the future by themselves, for themselves, then they'd also be capable enough that if they were what's called intent-aligned, so that they could do what humans actually want them to do if asked, then that a human could ask them to take over on their behalf. That's another kind of concern. And Normally there would be safeguards against an AI system doing these things, but maybe the human, if they run the AI lab or they work there, maybe they're in a position to remove those safeguards.

Or if they're the leader of a country, they could command that they be removed. So that's another kind of concern. And then a fourth kind of concern is the building of various forms of weapons of mass destruction using AI-assisted help. So in particular, the concern there is that AI systems could kind of democratize newer and kind of more powerful destructive abilities. So to kind of empower, you know, many individuals, perhaps people just with an undergraduate biology degree, to create some, you know, world-destroying virus or something like that. It doesn't have to be biology, it could be in other disciplines if new capabilities appear.

So I think that those four different areas are all concerning to me, and I'm also troubled by the fact that I don't know which is the most concerning. And I'm troubled by the fact that a lot of the things you could do to help with one of them make the others worse. So some of them, for example, the issue is concentration of power, and other ones, the, like the, the weapons of mass destruction, the issue is the dispersal of this power. So open weights, for example, helps with one of these issues and hurts others.

And so I feel really at sea for a lot of these levers that people argue about. I really don't know which way to pull the lever, you know, whether it's actually helping or not for the overall situation. Speaker B: That's reassuring in the very small sense to me and disconcerting on the broader scale because, you know, you clearly think about this at a much deeper level and much more than I do. But I do find myself really struggling with figuring out what, what level levers can we actually pull here that, that unilaterally help.

And, and on one dimension, the, the race dynamic is something I keep struggling with figuring out how we get around that. Because, you know, if we sort of assume that the major actors of these AI labs are morally normal, which We could, you know, probably debate that, and there's probably actually significant variance, I would, I would suggest. But just the sheer incentive structure and, uh, mindset that you get into when you are in these race dynamics, like, really feels like it leads to, uh, a lack of circumspection, lack of sort of caution.

Are there good examples historically of taking moral action in a race dynamic? You know, there were some perhaps from from the, you know, nuclear race, but that was quite incomplete as you, as you share in the book. Speaker A: Yeah, I mean, I would say mainly bad lessons. Yeah. As in, uh, lessons about bad behavior rather than good behavior. Uh, I mean, one example there with the, um, the Manhattan Project that I think is particularly salient is that the, uh, if you ask, you know, why were all of these people like these academics who studied physics working on building a superweapon that, you know, would be the most horrible weapon humans have ever built?

Yes. And the answer for most of them was that they were concerned about, uh, about Hitler's Germany having unrivaled access to those weapons and holding the, the world to nuclear blackmail. Um, so they, they, you know, uh, worked as quickly as they could to develop such weapons. But, um, when Hitler died, you know, when he, when he killed himself, and, and, uh, and then, you know, shortly after, uh, the war in Europe was over and Germany surrendered, they didn't stop. Um, only, only one of the atomic scientists actually stopped at that point.

Um, and you'd think that if there was one reason that was so powerful, yes, that it led you to make the worst thing that humans have ever made, uh, and you hadn't yet finished making that thing and then that reason disappears. You might hope that it gives time for a reevaluation. And there were some reasons why, you know, why America used the nuclear weapons against the Japanese, but they were certainly much weaker than those other reasons. And I don't think that the atomic scientists would have signed on to the project, you know, with the idea of shortening the war in the Pacific being the objective.

So at least not many of them. So there were, you know, one of the things we learned there was a group of people doing something that had serious problems, you know, for reasons that they'd convinced themselves were important, but then just no longer being sensitive to those reasons disappearing. And there's a whole lot of, you know, a whole lot of cases like this. I mean, another example of, I guess, a smart person talking themselves into a bad argument, I can't remember who it was, but the chemical weapons for the Germans in World War I, He has these written documents saying that he thought it could shorten the war and that even though it could kill a lot of people, the bullets were also killing a lot of people.

And that if the war was shortened, then ultimately fewer people would be harmed by it, both soldiers and also the civilian populations. And so he thought that ultimately, while the weapon itself would be brutal, that this would fundamentally be a good thing. And what, you know, the big thing— well, a big thing he didn't realize, even on his own logic, was that Germany lost the war. And if you were helping the side that lost the war, that means that your technology was lengthening the war, uh, not shortening it. And so the idea that, that, you know, he was smart enough to develop, you know, all of these chemical techniques and things, um, and to, to run this argument, but not smart enough to realize the argument only works if your side wins the war and that there was like a 50-50 chance that his side would lose the war, in which case it doesn't work.

Yes. You know, I think there's a lot of these things where we see smart people, in that case maybe not someone who is particularly idealistic, but it shows that it's quite easy to talk yourself into various things. And I think that there's a lot of that going on at the moment, particularly when it comes to racing. Speaker B: Yeah, you talk about this, the unilateralist's curse in the book, which is sort of the idea that you know, all of us may judge these different actions that we could potentially take to— you know, let's say there are 20 people who might release, uh, the, the next AI model.

Whoever is essentially most optimistic and, uh, sort of has the most Panglossian view of, of what might happen, they're the ones who release it. And, and then, you know, if it actually causes damage, everyone sort of reaps the, uh, the damage of that, or you know, incurs the damage of that. And it feels like there is some of that happening in— Exactly. Speaker A: So just like how in an auction it goes to the highest bidder, and so if all the bidders— if there are many different bidders and they all have somewhat noisy and inaccurate estimations of the value of some object, the person with the rosiest impression of the value of the object, you know, will be the one who wins the auction.

So this is called the winner's curse, that sometimes you can systematically end up you know, the very fact that you won probably means you overpaid. That there's a, there's a very similar issue here where, yeah, if a whole lot of people could all unilaterally make some action happen, you know, maybe it released the genome of smallpox or something like that, and a lot of different people are trying to think about the benefits and the drawbacks. And there are serious benefits of doing that kind of research. The same with, say, gain-of-function research, where you try to make You try to make a virus more transmissible or more deadly in order to understand what mutations are needed so that you can be on the lookout for those mutations occurring naturally.

There is an argument, a rationale there, but it's not obvious if that rationale wins out versus the harms of doing that. But if you just allow anyone to do it, then it will always be the people who end up with the most inaccurately positive estimation of this action. Who will be the ones who will make it happen. And so this is an interesting form of coordination problem. And there are ways around it. For example, you could get together with the group, other group of people who could all unilaterally make something happen.

So say all the other people who could be releasing a model like this, and you could say, let's have a vote. And if you just take the, the vote ends up reflecting the median estimate, the middle estimate if you do a majority vote. And so that actually almost completely resolves the problem if you can do it. In the case of a model release, it could run into antitrust issues if there's an issue of collusion to slow down a technology. So unfortunately, I think antitrust law could well be quite bad for us when it comes to these situations.

There are appropriate mechanisms whereby companies can get together if they create a third body, a standards body, which is open to new members and not just a kind of fixed list of members. Then you get that body to recommend a course of action and then you follow that action or something. There is an official way to do it. I worry that the extra effort and steps involved in doing that slows everything down, as in it slows these remedial actions down while the other actions are being sped up by the pace of kind of, you know, investments and need to make a profit and so on.

Speaker B: Given that you're, you're sort of using the example of antitrust, is— do you think, like, uh, if we were able to resolve, uh, some of this race dynamic within the, would that be sufficient? Because we are, you know, the sort of thinking there would be that the most advanced models and most advanced capabilities are, are -centric right now. And so by sort of removing the— this this dynamic where, you know, reducing the existential risk from unaligned AI, or is the fact that, you know, China is, is clearly taking this very seriously, even if they do seem to lag in, in some considerable places and have been, you know, cut out of the semiconductor supply chain in really meaningful ways, uh, does that sort of fundamentally, just by controlling the

, invite a different race dynamic where we're sort of actually hamstringing, uh, Western powers and leaving the possibility of unaligned AI in a much less democratic environment. Speaker A: Yeah, I mean, this is another thing that's related to the unilateralist curse, that if there's some issue that seems at some level problematic, let's say taking some new step with a technology is at least at some level troubling, then if all of the people who are concerned about such things all stand aside and say they're not going to do it, then the person who kind of has the fewest ethical qualms will be the person to kind of step across the threshold.

And maybe we'll get the first version of something will be a more dangerous version of it. So that is a general concern. I think that it's also one of those— it's exactly the kind of thing that, you know, people in the position of the atomic scientists could use to rationalize racing. Totally. And so while it is a not a bad idea. It convinces different people different amounts, and then you kind of worry that the type of people who are erroneously convinced by it will be the people who take the actions.

I think it's a real challenge. So as you say, there's kind of racing, roughly speaking, at two levels at the moment. There's this kind of company racing between different companies, most of whom are at least headquartered in the US. And then there's a kind of racing between the US China. So the US is clearly the one that triggered the race between the US and China, and partly it triggered it because it— on the rationale that eventually the race was going to happen and we want to win it if it does happen.

And so if we start the race, we've got more chance we'll be winning the race. And there's a logic to that, uh, but it still is a case of, you know, if you think races are bad and the racing behavior is kind of like defecting in a prisoner's dilemma, and you think that unfortunately the structure of the world is such that probably both parties are going to defect So therefore we should defect first. Maybe that's the right thing to do, but it's still kind of jerk behaviour at some level. And so I think one has to admit that.

And if ultimately we succumb to some existential risk with AI, if you kind of imagine the different powers that be, or all the different people involved kind of being called up to the pearly gates, meeting Saint Peter and trying to explain their behaviours, It's a little kind of thought experiment. Speaker B: You'd have to get into some game theory. Speaker A: It wouldn't sound great, I think, if you said fundamentally we knew it was really risky, but we thought people would do it, humans would do it, even though they shouldn't.

And we thought there could be a race, so we started the race. And did you attempt to have a treaty to like not race? No, we didn't attempt that. We thought it wouldn't work. So we started the race and then we raced and we got there first and we built the thing and the thing killed us all. I just think it'd be pretty difficult to really, you know, if you add a kind of like an imagined "your honor" at the end of these statements, they don't sound great to me. And I think that as it happens, given that China is behind in this technology, I think that they have additional reasons to not want a race.

So if the US were to think, as I think they should, that we would rather there be no race, and China would prefer that even more because they the race that there would be is a race they'd likely lose as well. Then I think China could be in quite a position to be happy with some kind of verification conditions or things around kind of avoiding a race. So I feel that there's been very little thought going into not racing. And I think that if you pretty much just had those two countries agree, they could potentially police their own spheres of influence to say, okay, no one here is building things towards superintelligence and no one over there is doing it, and then have verification on each other's AI industries.

Such that no one cheats it, essentially. Yeah. For example, by saying, okay, well, we'll send 30 of our people over to be part of your AI companies, and you can send 30 of yours to be part of ours or something to make sure that we're not doing it. I think that there are ways to do this. In the Cold War, I think there's a lot of lessons from the Cold War. Ultimately, there was a situation where the US and the USSR were very much adversaries of each other, if anyone is.

And yet they actually treated each other as adversaries in what I think is a somewhat grown-up way that we're not doing these days. I think that the US is treating China as an enemy, not an adversary, and they just kind of want to get China. Whereas with the USSR, I think there really was a realization that there were a bunch of things that were in their common interest. Not everything, but there were some things. So for example, the Non-Proliferation Treaty was in their common interest. If the US has nuclear missiles and the USSR does as well, they don't really want all the other countries to have nuclear missiles.

It's in their common interest that this would be an elite kind of exclusive club. And it was also, I think, in most of the world's interest. And so they helped to broker and police this Non-Proliferation Treaty. Treaty. And similarly, their arms reduction treaties that they had, because they both would have rather that they had a tenth as many missiles as they actually had. And they managed to do that. And there were very smart people working out very novel verification techniques. There were these ideas of, once spy planes became possible, of allowing the Russians to fly spy planes over the US airfields while they sawed their bombers in half, and they pulled the halves apart from each other with tractors, Wow.

Uh, and those bombers are still there on the airfields, uh, sawn in half as a, as a costly demonstration of destroying their military capacity. Wow. Even without inviting inspectors over, they worked out ways to credibly show that, um, that they were, uh, reducing their arms. Uh, and so it's, uh, you know, there were really people thinking outside the box is what I'm saying. And there's a lot of very smart people today thinking about these issues. A lot of them are going to work at AI companies and build these technologies. But there's also a lot of kind of complaining that we have of like, oh, verification's too hard.

There's no way we can verify a kind of deal-to-not race. And I feel that the amount of, let's say, full-time equivalents spent on thinking about verifying a race between China and the US on AI, maybe there's like 3 full-time equivalent people working on that or something in the whole world, if that. Yeah. And there's probably almost nothing that's been explored. So I think that there's a lot of just giving up, "Oh, that'd be way too hard," without really trying to work out, are there really clever technological or sociological techniques? Speaker B: That's really interesting.

I want to touch on one of the other areas briefly that you talk about in your book, which was the extremely prescient and immediately prescient part, which was you know, pandemic risk. Uh, you, you mentioned gain-of-function research, uh, and sort of thought through the trade-offs there. What have been the lessons of COVID that, that you take from that kind of research in general and how risky it is? And, you know, more broadly, did COVID make us more prepared in some ways for what comes next, or, or have we really not, you know, taken almost anything from it?

Speaker A: So one thing it did was that When I was writing the book, I was certainly concerned that people were just not going to take threats to humanity's entire future seriously. That there was a feeling that, I don't know, that there was a time when these things were possible and that time was in the past. Maybe in the case of pandemics, a feeling that, oh sure, there was this 1918 flu that I've heard of and there was the Black Death, I guess, but that was medieval. And technology has advanced so much since then and we do so much in our hospitals.

We understand the germ theory of disease and all of these things. And so we're doing much better at this. And so it's not a risk. And COVID certainly reminded us that we're still at risk and that for all of our technologies, in some ways, we're just as vulnerable as we were before. So when it comes to dealing with individual infections and trying to save someone's life, in the hospital. We're much better at that. But overall, when it comes to the spread of disease, because the airplanes that we've invented and things, you know, the cities that we've built and the public transport systems and so on are so efficient at just spreading this disease around, a lot of our technologies make it worse at a similar pace to the way that our protective technologies make it better.

And it's not obvious at all that we're actually more protected overall when it comes to pandemics. So that's, you know, it's helped give us some of that kind of reminder. And also, it also was this feeling, I think that, you know, when it started, right, there was this thing where you're like, hang on, what? You know, is it going to be the case that everyone in my country is going to be at home in their house? Yes. And the streets are just empty for days, you know, days, I mean, months.

Yeah. Is it— school's going to shut down? And there was this feeling of like, is that 'Could that even happen?' And in general, I think for people who'd lived through, say, the Blitz in London, that really shaped their view and they felt, 'Oh, well, we're not even allowed to have lights on at night,' or something. And they've been aware that the space of ways of living could be quite different and the space of kind of big things that could happen could be quite big. Ultimately, COVID was probably this biggest world event since World War II.

Yes. I feel like there's this kind of level of what's the biggest change to life that you can imagine is set by the biggest one you've lived through so far. And so at least it's helped raise the ceiling on what we can imagine could be required of us and how different life could be. So that's been somewhat useful for kind of letting people imagine more. I mean, also, I guess, in terms of political trends and, you know, political populism, you know, as a rise of this movement and so on, if, you know, if you lived from you know, through the '90s and the 2000s.

And, you know, it had really felt like we were in some really stable era of some sort. Yes. And then, you know, we've come out the end of that. And that took quite a while to learn those lessons and to kind of realize that, at least for someone in my age, that the kind of world that I'd— the only world I'd known as an adult, you know, could be kind of coming to an end as a geopolitical order or something like that. It's helpful to take that wider view, and COVID has helped us do that.

So in some ways it's helped us kind of realize that for other threats as well. It was thought by almost everyone that there would be what's called a panic-neglect cycle, where having been hit hard with something, that you overreact and that the governments would kind of, would do more than they should. Either they'd spend more money or they would kind of like inhibit freedoms too much or something like that to react against this thing. And then that would slowly fade away until it was too little. But that hasn't happened. We actually didn't get the panic phase of that.

And by the time everyone had reopened after COVID, I think people were just so sick of it that they didn't even want to talk about funding these bills to spend even a tiny fraction of the amount of money that we were cost or the amount of damage that was caused by COVID to prevent future such things happening. So that was a really missed opportunity. And I was surprised. I would have thought that you'd get this panic. And then the question was how do how to soften the panic into the appropriate level of response, but then how to maintain that appropriate level as well.

And yet, no, we never even got the appropriate level. And I also feel that, you know, that had some good features like the mRNA vaccines getting kind of a test out and kind of, you know, having flexing our muscles on this new technique of very fast vaccine development. Yes. Was good. And I guess I don't know overall, you know, whether— how it's really helped our preparedness. I think that again, you can kind of tease out like, you know, 4 or so of the biggest factors kind of that either made it better or worse, or perhaps in some cases have not made it better or worse but revealed underlying structural weaknesses.

For example, it didn't take all that long before there was a lot of acrimony between different political groups or between different countries about, you know, who is at fault and who should be handing over the vaccines and various things like this. You know, I guess we learned that we can band together for a while, uh, but a while, you know, might be, uh, a 6-month period. If the crisis carries on for years, uh, that, uh, that really does start to fragment. Speaker B: Yeah, but I think that's true that it's, you know, your, your point on the way in which this totally shifted our frame of reference from this can never happen, that would be a, you know, as fanciful as a zombie apocalypse, to you know, actually we all lived through it.

And also, you know, Operation Warp Speed with, you know, the development of vaccines is maybe a good example of a beneficial race dynamic and where those things can work favorably. Speaker A: Yeah, I mean, it's even, you know, I think the Operation Warp Speed was a successful government, you know, slash private, you know, cooperation and so forth. Yeah. Speaker B: You, you mentioned sort of the political landscape and, and the institutions, uh, that, you know, played their parts in, in COVID. One of the things that I've been thinking about in, in reflecting on the book and, and some of these questions is how combat— compatible are democratic institutions with addressing— with existential risk?

You know, there's sort of this time span of discretion where your senators are thinking on you know, I think they got 6 years, I actually can't remember now, uh, 6-year cycles. Your presidents are thinking on maybe 2 and a half because then they start thinking about, uh, you know, the next election and, and so on and so forth. No one is really incentivized to think about maybe even 10 years, uh, that, that would actually be a really farsighted person. How can we sort of address these existential problems with such a short-term mindset from the institutional level.

Speaker A: Yeah, so it's a real challenge. It's not necessarily, you could say, a challenge of democracy in the sense that if you didn't have democracy, if you had some kind of autocracy, it's not clear that makes it better at all. If the autocrat happened to care about this issue, maybe it would make it better. But if they don't happen to care about it, then they wouldn't really be able to get the signal coming from the people. I'm not sure that it's itself that's to blame for it. But something's happened to the democracies that we have in Western countries, at least, to really shorten the timescales that they focus on.

I'm not exactly sure what's going on there or why that happened, because it does seem like the democracies, if you go back 50 or 100 years, It does seem that they were able to achieve larger projects and projects that operate on longer timescales than they currently can conceptualize. So maybe it's to do with the kind of complex relationship between the democratic institutions and the media, and the success of certain forms of media in kind of punishing them for not preparing an answer to the question they asked on Monday or something.

And so everyone's thinking in terms of the news cycle not even the election cycle. I don't know what it is, but it does seem like the time span, the time horizon for politicians has gotten shorter. There's also though, separately from whether it would be better to not be a democracy, there's a question of is there a kind of failing still of the current democratic system to perhaps live up to the principles behind democracy when it comes to this issue? And I think that there may well be. So when we think about the most important decisions that that policymakers are making in our time.

They may well be decisions such as what they're doing about climate and what they're doing about, say, technologies like AI. For example, deciding actively to not regulate and prevent regulation seems to be an active decision to do something negative there. But these are decisions that if you make them in one way versus another way, may well have implications that just echo through the generations, having kind of like, you know, making people's lives significantly worse for many generations to come, or perhaps not even getting a chance to exist at all. And so I think that there's this issue that most of the stakeholders, or like the, you know, or the people who are affected by it, don't have representation in in that decision because they're people of future generations.

And so, and in some cases, we can also predictably work out which way they would vote. You know, they would vote for doing more action against climate change. Yes. For example. And so it's not even just that there's this kind of unknowability, so we couldn't possibly factor it in. And so there's a kind of constituency or stakeholder group that is just not getting the franchise. Yeah, that's really interesting. As somewhat similar to questions about people of different races or sexes or landholders versus non-landholders and other types of issues in democracy of extending the franchise.

And so I think that there are potentially things that a democracy could do about that. For example, in the UK system where we have the House of Commons, which is a bit like the House of Representatives in the US, we also have the House of Lords, which is an appointed body Some of whom are appointed by the Prime Minister, some of whom are appointed as the kind of good and great of society, former heads of the Royal Society and things like that. And it would be possible to appoint a group of lords to represent future generations, or you could do it in other ways.

You could have a citizens' assembly, for example, representing future generations. And I think that experimenting with tweaks to the current understanding of democracy to deal with the fact that now so many of our choices our most important choices could be affecting all of these stakeholders who don't have representation. The idea of kind of no taxation without representation, kind of foundational to the American ideas of democracy. And yet maybe we need to take it seriously in this case of future generations and to work out, are there ways to appropriately understand their interests and then also to force some kind of representatives to represent those interests?

Because obviously the people themselves who live in the future can't vote in this thing. It doesn't mean they can't be represented. Speaker B: That's a really fascinating idea. Yeah, having these sort of stewards of the future as part of the governance process. Speaker A: Or even if it seems to be too impractical, to at least set it as an ideal, to say, oh, if only we could do this, that would be great. Unfortunately, we can't, or something. But it seems at the moment, it's, uh, people, people don't even think about it or don't realize that there, that maybe there are things we could do about it.

Speaker B: Yeah, there's sort of the implication that people suspect or sort of hope that politicians are thinking about the future, but I think the reality is that, um, that is, as you said, has sort of long since gone. Yeah, so that's a nice way to, to at least consider it. Uh, and we like to sort of end episodes with a few sort of more abstract thought experiment-y type of questions. Uh, so with that in mind, if you had unlimited resources and no operational constraints, what experiment would you like to run?

Speaker A: I just, I think I have no idea. Oh, wow. Great. Okay. That's, uh, apologies. Speaker B: No, no. This is a sign that you're, uh, you, you wouldn't want to go off on a wild experiment without thinking it through, which, uh, makes sense given your background. Speaker A: I guess, I guess that's, that's part of it. Um, um, there's a, you know, there's a, uh, a famous, uh, comment from, uh, Carl Sagan, uh, when he was asked about, you know, are you really sure about, um, about nuclear winter?

You know, how could you know? This experiment's never happened, you know. And he said, you know, it's unfortunate, you know, that we have only one Earth. We have no spare Earths to destroy in some kind of experimental laboratory in order to determine the, you know, exact reliability of this theory. So if we had unlimited resources, you know, we could have those spare Earths to destroy in order to find out what these risks actually are. But unfortunately, that sounds like the kind of experiment that's probably ethically a no-go regardless. Speaker B: Yes, that's right, that makes sense.

Well, let's try this one. If you had the power to assign a book to everyone on Earth to read and understand, what book might you choose? Speaker A: I might choose Practical Ethics by Peter Singer as a book that's, uh, it's a really good kind of no-nonsense approach to thinking about ethics. It really helps show how to kind of reason about all kinds of issues with an open mind not assuming that our current kind of ethical intuitions are the be-all and end-all. And, you know, I think it's a good example of that kind of thought.

Speaker B: Is there a sort of a particular framework or sort of lesson that you particularly reflect on from that book from Singer? I imagine he's a major influence of yours. Speaker A: Yeah, he is. Yeah, probably just really what I said then, of the fact that he, you know, just kind of models this kind of really open inquiry into moral matters. Derek Parfit mentions that ultimately there's been only really a couple of generations in which people have made what he calls non-religious ethics their life's work. So if you look at the whole history of philosophy, almost all discussion about ethics had been within a religious context where the set of possible answers always had to kind of align to what had already been thought thousands of years earlier.

But people trying to, separately from that, to try to understand these questions, there'd only been a couple of people in the whole history of philosophy. And then there'd only been a very short time since it became more widespread. And so he saw that as a reason to be very optimistic about the possibility that we'll make some substantial moral progress. Speaker B: Fascinating. What tradition or practice from either another culture or time period do you think we should widely adopt? Speaker A: I think I don't know enough about particular time periods.

This is the problem about being an academic and wanting to not just make something up. But I do feel that something that involves more gratitude to our ancestors, I think, would be particularly important. I know that there are many cultures that have more gratitude to our ancestors than Western cultures, which have almost none. But I'm not sure exactly who we should be copying it from, uh, but I would love to see more of that. Speaker B: Yeah, I agree. That's a great one. Well, Toby, it's been such a pleasure and, uh, yeah, I can't thank you enough for, for your work and your writing, um, and, uh, and for your time today.

Oh, thank you. Speaker A: It was a wonderful conversation. Speaker C: That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at com. See you next time as we continue to explore the future.