Existential Risk and the Future of Humanity: Lessons from AI, Pandemics, and Nuclear Threats (Toby Ord, Author of "The Precipice")

The human story might just be beginning, but there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts, but some of them are threats of our own making, such as nuclear war. You put the odds at one in six that in this century, we do incur the existential risk and humanity fails to live beyond this century. Have the last few years made you more or less worried about this? So another possibility is what gets called gradual disempowerment. Suppose the AI systems never violate it. rights maintain sufficient control over them they don't break the law but they're more successful than us over time at doing all the kinds of jobs that we do even if we get richer through trade with these systems a higher and higher share of the wealth eventually accumulates in ai hands hey i'm mario and this is the generalist podcast as the saying goes the future is already here it's just not evenly distributed Each episode, we sit down with the visionaries, builders, and thinkers who are already living in that future to help you see it earlier, understand it better, and capitalize on it. Today, I'm speaking with Toby Ord, a senior researcher at Oxford University, one of the world's leading experts on existential risk, and author of the excellent book, The Precipice. It's a thoughtful analysis of the greatest dangers to humanity's survival. from rogue asteroids to pandemics to unaligned artificial intelligence. These may sound like fanciful sci-fi problems, but there's actually good reason to believe they're extremely pressing, with Toby estimating humanity has a 1 in 6 chance of extinction this century. In my conversation with Toby, we discuss how AI risk has evolved since LLMs emerged, why the scientists working on the Manhattan Project didn't stop developing nuclear bombs after Hitler died. and the Cold War lessons the U.S. is ignoring in its current great power conflict with China. I walked away from our discussion with a better sense of how we should think about humanity's future, what we owe coming generations, and what steps we might take today around AI safety and pandemic preparedness. This is a new podcast, so if you like it, I hope you'll consider subscribing and leaving us a review. Now, here's my conversation with Toby Ord.

Well, Toby, I'm so excited to chat with you today. I can't tell you how much I enjoyed your book, The Precipice, which I have to hand. And in looking at my sort of marginalia in preparation for this, I was struck by one, that I could be a much more thoughtful reader, and two, how much I enjoyed it, because all of my notes are sort of, wow, exclamation mark, exclamation mark, and no way. Maybe with that as a little bit of a background, perhaps you could tell us what exactly existential risk is and what does it mean to study it in the way that you do? Humanity has had a long and illustrious past. Our species has been around for about 300,000 years or more than 10,000 generations. And we might be able to have a future of equal or larger size. Certainly most animals do. They typically live for about a million years. So the human story might just be beginning. There are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts and things like that, these natural risks from the somewhat hostile world that we live in. But some of them are threats of our own making, such as nuclear war. And so existential risks are any risks that could threaten to permanently destroy humanity's long-run potential. So that could be something that makes humanity go extinct. Or it could be something that, say, causes a permanent collapse of civilization where if it was unrecoverable and there was no way back, then these things would have a similar kind of role. They would be the types of events that reduce the value of our future to almost nothing and which we have to make sure that we avoid falling victim to these even once over the hundreds of thousands of years to come. Yes. When you sort of started in this field, was it the case that these risks were relatively well known, at least, if not well sort of studied? Or, you know, does the work of an existential risk researcher involve thinking of, you know, all of these possibilities that maybe no one has actually even properly cataloged before? Yeah, I think the risks...

at least the risks that I know of, had been fairly well understood, or at least, sorry, maybe that's going too far. They were all known of. So it's not that I sat around trying to think about new things that could threaten us. It's possible to do that, and there's kind of, you know, any number of things that could pose some kind of risk. But often those risks get smaller and smaller as you go further down the list. And so it's not... you know, that needed in order to kind of keep adding on things that are only tiny compared to the things you've already got on the list. And I think that they'd been generally known about for hundreds of years through to kind of decades or something like that. But there could be risks that we're still unaware of. For example, in 1900, we were unaware of the risks of gamma ray bursts or supernova. explosions, or even the risk of asteroids was only really convincingly demonstrated in 1980. It's actually surprisingly recent. And so there could be other risks that we're ignorant of. But there is at least a helpful thing when it comes to the natural risks, which is that we know we've survived for 300,000 years so far. And we know that typical animal species survive for about a million years. So it can't be the case that the risks are kind of much higher than that. So ultimately, the risk from natural events has to be something like one in a million per year or lower. Fascinating. We'll get into some of those risks in greater detail, especially some of those that are... particularly relevant these days. But I'm curious, you know, more on your journey, you studied sort of computer science in Melbourne. How does one go from, you know, perhaps a future software engineer to spending their lives focused on the far future, on risk, on these sorts of questions?

I guess one thing they have in common is I was probably always going to be an academic. And so in computer science, I was interested in the kind of theoretical side of computer science. Oh, interesting. Including very much in artificial intelligence, which has turned out to be useful for me now. No kidding. I guess I was drawn to these questions about... really zooming out and looking at the big picture of humanity's future. So thinking about this when reading good works of science fiction while I was growing up, one of the things I remember being struck by is if we did manage to reach a point where we dealt with all of the negatives in the human condition. So we dealt with poverty and we dealt with various other forms of pain and suffering. As we have... been slowly doing over the years, right? Like, you know, if you go back a couple of hundred years, there were no anesthetics and things, you know, the idea that you could go through surgery without extreme pain, you know, would have shocked people. And so if we kept going down that trajectory, you know, what would happen next? You know, what's the positive story about what we should be doing? If we got to a point where we kind of had removed a lot of the injustices and discriminations and, you know, pain and suffering and inequality, you know, all the bads, you know, what next? And so I was interested in these kinds of, questions about the future of humanity and trying to understand the big picture challenges that we face. I guess one of those challenges that was preventing us from reaching that point was global poverty. And so I was very interested in that. since I was an undergraduate studying computer science in Melbourne, and had, you know, just decided since individuals could do so much to make a difference, that I would kind of make that part of my life. And I've continued to do so. And that led me to found this organization, Given What We Can. where people make a pledge to give at least a tenth of their income over their lives to the most effective charities trying to help others, you know, be they people or animals. Yeah, it's often turned out that the things that are some of the biggest challenges facing humanity are also some of the ones where we can kind of get the most leverage on them. So we kind of get the most bang for your buck in terms of if you're going to donate, say, £1,000, where you can do some of the most good with it.

That wouldn't have to be the case. You could imagine a case where there's some huge challenge we face, but it's really quite intractable. And there's not much or there's not much individuals can do about it. And I think that global poverty is at least more tractable with respect to money, at least money for people in America or the UK or Australia in the richer countries, you know, where people are among the, say, the 5% richest people in the world. It's then not so surprising that money. the thing we have the most of could be something that we could do to really help those who have much less of it. But in the case of, say, risks posed by artificial intelligence, it does become a harder story about what's everyone supposed to do about it. Yes. And when we're talking about these more tractable problems, things on the order of malaria nets, deworming initiatives, that sort of thing where you really can, with small amounts of money, make a very immediate impact. Is that a fair characterization? Yeah, they're the things that I was particularly drawn to. One thing I really like about them is that I think, you know, there's interesting different opinions on giving. Some people just aren't very interested in helping others and I'm not going to convince them otherwise. But there's a lot of people who I think would be tempted to do this. They really feel the pull of it. But there's some kind of blockers or things that stop them. And one of those blockers is a feeling that, do we really know it helps? And so where you want to find something that's got the kind of most robust evidence behind it to then say, well, to the extent to which you really just want to make sure that you make a difference, here's something you can do that is very likely to do so. And so I was particularly interested in those cases, as well as the possibility of more speculative things for people who are willing to go a bit more out on a limb and, for example, fund some new technology that could help people in poor countries, but maybe it will never pan out. Maybe they can do even better than these tried and true things. But at least the tried and true things can create a nice kind of baseline for the idea of, should you keep the money in your pocket and spend it on yourself? At least if it's some kind of skepticism about whether it will make a difference, then finding those reliable things is very useful. Yes, also important to get into the sort of mode of talking about these things. I think another blocker is that people feel...

you know, they shouldn't mention it or, you know, they're inviting too much attention, but actually by signaling what you do, you sort of normalize that for a lot of other people. And yeah, so I'm glad we're talking about it up top. I'm certainly not at the 10% rate, but I'm a recurring donator to give well with the idea that I can't count on my empathy to Spike with enough regularity that I remember to do it. So I may as well just, you know, sort of put it on autopilot. But anyway, that's a really... Yeah, I'm glad we sort of got into that. And that sort of led you into some of these questions and studying with one of the most prominent moral philosophers of the 20th, 21st century in Derek Parfitt. Also, in a very crowded field, one of the best sets of hair of any philosopher, I must say. But I'm curious what it was that you learned from him as a mentor and how that sort of framed your worldview. Yeah, interesting. Probably the biggest effect that Derek has had on me was that I probably came to Oxford in large part on the strength that he was here. When I'd studied computer science and then also philosophy in Melbourne, some of the philosophy that seemed to really make the most sense to me that was really well thought out and kind of ingenious in various ways was the work of Derek Parfitt in his book, Reasons and Persons, particularly. And yeah, he did amazing foundational thinking on the nature of personal identity, so what it means to be an individual persisting across time. And as someone who was thinking about artificial intelligence, you know, some of his thought experiments seemed to, you know, maybe humans couldn't do these things. For example, a human can't split in two. And, you know, Derek had a whole lot of, you know... fanciful science fiction style thought experiments about this. But if it's a computer program, it's a lot easier to imagine. You could just pause its execution at a certain point and then fork the process into two different processes. And it really does seem like you could have an individual, a kind of Y-shaped individual who has a certain kind of common path and then kind of branches into two different futures and ask a lot of questions. If a program could be a person and a program could be conscious and things like that, then you could ask, what would it anticipate before the moment?

that it was split into these two different environments. And so I thought, you know, a lot of his style actually really fit well with my computer science background, despite the fact that he was, or he saw himself as a very non-mathematical and non-technical person. And he tried to avoid any maths in any of his writing, which I found remarkable because it was, in some sense, it was so mathematical already. It had this real clarity of the logic. I enjoyed some hobbies with him as well. He was a prolific photographer and took these breathtaking photographs, particularly in St. Petersburg in the snow and Venice in the mist. I enjoyed talking photography with him as well. Amazing. I'll have to try and find those. I'm sure they're somewhere. One of the things that... Derek Parfit, you know, was so famous for as well as sort of the concept of identity is sort of the idea of Caring about these future persons the people that that come next and and that's obviously such an important part of thinking about existential risk Perhaps you can you know give us a little pricey on on that concept and and how it factors into this topic Yeah, so so kind of A couple of areas, related areas that Derek worked on were to do with what's called population ethics, which is a lot of our ethical questions concern individuals who are all alive at the time when the choice is made. For example, we're trying to work out whether to... to extend one person's life versus extending a different person's life with scarce medical treatment. Ultimately, though, in almost all standard moral cases, like whether to tell a lie or something like that, the person who would be lied to is someone who already exists. But a lot of the questions we make, particularly at the societal level, concern changes to the people who will ever come to live. And so that could be because, for example, if there is a risk of extinction, then there won't be people in future generations. So those people will never be born and those lives will never exist at all. And so there's a question about how to value that or think about it. Is that, do we lose the entire value of those lives?

So kind of taking them from what they would have been down to zero. Or is it something different that we do? Maybe it's not as bad to have someone never exist at all compared to if they died when they were an infant. There's a lot of different ways of thinking about this. And so Derek Parfitt really opened up this, helped ask the right kinds of questions and then sketch some of the challenges with trying to answer them. He also, though, somewhat separately, but... But it goes together well. He was interested in the idea about actions that could have benefits over very long timeframes. So when it comes to economists, they often do something called discounting, where they try to... In order to understand the value of actions that have effects at different times, they use a mathematical technique in order to say that the effects they have at times further into the future matter less. And in some cases, this... totally makes sense. So for example, if there's going to be inflation, then getting a dollar in 20 years time won't be worth as much as a dollar now. Or if you know you're going to be richer in 20 years time than you are now, maybe you're a student and in 20 years time, you'll be kind of high up in some career, then a dollar, even inflation adjusted, will be worth more to you now when you're poorer. So obviously, you want to adjust for those things. But most people, including economists, go one step further and assume that actually intrinsic value itself, once you've adjusted for all of that, is just worth less if it happens later in time. And Derek Parfitt, and in fact, most philosophers ultimately, rejected this. And so Derek Parfitt has nice thought experiments on this. So there's one where if you imagine... hiking on a trail and accidentally breaking a glass bottle when you're having a picnic. And then the shards of glass, you're deciding whether to pick them up and take them with you or to just leave them there. And suppose if you left them there, you knew that they would injure a child's foot. So this young child would be kind of stabbed through their shoes and be bleeding on the trail. And they would survive, but they would have a very bad time of it.

Then his idea was, does it matter if you knew that that was going to happen? Does it matter if it was going to happen in one week or a year or 100 years or 10,000 years if there was going to be exactly one child and they were going to have the same amount of pain and suffering from this and so on? This is kind of thought experiment to suggest actually that the time of these things, you know. If there's no way that it compounds or does something else like that, then the time seems to be irrelevant. These ideas kind of led the way to systematic thinking about intergenerational harms or intergenerational benefits. And then famously, he also wrote one of the first descriptions of existential risk as a major moral issue. So definitely, in terms of, I guess, getting back to your earlier question about the type of influence that he had on me, certainly what I've done since then has been very shaped by these ideas. Amazing. Yeah. I think in the book you have, I can't remember the exact phrasing of it, but the idea that if you do apply this discounted rate to future lives, you can get into these scenarios where... you are waiting the fact of someone currently having a headache more than a million people dying and suffering in however many hundreds of thousands of years in the future. And so it can get really wonky, but fundamentally, I think the glass example was so clear that these are questions about what we owe the future and what we owe to the people that come next. And that's really... what you outline, I think, so thoughtfully in The Precipice, which came out a few years ago, but it strikes me that it was incredibly prescient immediately, which is not something that usually happens, and remains that way. And I'm saying that because it came out before COVID, and you talk a huge amount about pandemic risk, and you also talk a lot about AI alignment, which has obviously become more and more critical. But one of the sort of headlines, so to speak, is that you put the odds at one in six that in this century, we do incur the existential risk and humanity fails to live beyond this century. What do you put our odds at now? Have the last few years made you more or less sort of worried about this?

Yeah, I'm not actually sure. I think that there are, you know, if you look risk by risk, so some of them say nuclear war, I think the risk has gone up. Ultimately, when I wrote it, it was before the invasion of Ukraine. And subsequent to the book coming out, you know, the UK has been actively threatened with nuclear war by Russia. And, you know, there are also a whole lot of questions about, you know, the possibility of... yeah, getting into either a conventional or nuclear war directly between Western powers, you know, as part of NATO and with Russia. So that one had seemed forgotten and distant when I wrote the book, and I'm afraid it's just gotten worse. In addition to that, there's only one treaty that's still kind of, you know, bilateral treaty protecting, well, keeping the number of warheads low. between the US and Russia. And that treaty has been extended. It was extended in the previous administration. But it can't be extended again. A new treaty has to be negotiated. Oh, wow. And so, but it looks like this is going to lapse. And the long history of nuclear warhead numbers going down and kind of slowly but surely creeping down further and further. Yeah. You know, where we started to become impatient, you know, how is it taking so long to get down towards zero? I think that probably actually that number is going to go up rather than down. And then the new challenge will be trying to keep it low rather than trying to get it lower. Reproliferation, yeah. Yeah. But when it comes to pandemics or when it comes to AI, I think that there's been a number of changes that have made it more risky and a number of changes that have made it less risky. And I think that they're quite hard actually to tease out the overall effect. Wow, that's really interesting. I would have for sure anticipated that you would feel more worried at this point. So I'm excited to dive into these a little bit. Before we jump to AI risk, this is such a silly question on one dimension, but on another, it's one that I think of myself. Why does it necessarily matter if humans go extinct? Obviously, that would matter a great deal to me and to my, you know.

my children and maybe my, you know, future grandchildren. But should we really see ourselves as quite so important, especially when we can imagine that given the, you know, the magnitude of the universe, there is probably, you know, it seems more likely than not that there's other life out there. What makes us so unique? Yeah, interesting question. So it may not be important that we're unique. uh suppose uh suppose you have a child and and you know that they live their life and they they you know they do a whole lot of activities that are kind of you know standard and kind of you know celebrated cultural activities uh you know in in your milieu uh you know dance and song and you know spending good time with friends and things like that but maybe they don't do anything that that is uh that's unique that's kind of pushing outside the envelope of you know what's been done before or something Or suppose there's someone who has a life like that and someone else who's doing more unique things and you ask which life should we save if we can only save one of them. It's not clear that we should save the more unique one or something. It seems like there's value in a life well lived, whether or not it's unique. But the ultimate answer to this question is unknown at the moment. But there are some different kind of leads we have on it, different intuitions people have, and I think that there's something to them. So one way to see it is it would obviously be what's bad in the present if there was some extinction event, say. So all of the lives, the 8 billion lives that would be lost in that event. We could all see that that would be terrible. We know that it's terrible when someone that we know is to die, and presumably it will be about 8 billion times more terrible if that happened to everyone, which is really very bad. But I set that aside because that bit's fairly obvious. Then there's also everything that we would lose. So if you think about the future and all of the generations who could have come to pass, say that the next...

you know, 700,000 years of humanity if we get our kind of average amount of time. And all of the hopes and joys of all of the people who would have lived across, you know, 20,000 more generations of people. And all of, you know, the art and achievements, you know, that they would have made over that time, you know, the kind of fruition of the potential of humanity, you know, as we kind of evolve over that time. So all of that would be lost. So that's a kind of view based on the future. So now we've got the past, sorry, we've got the present. What happens in the present and what happens in the future? And that one gets into population ethics, the future bit. There's this question about exactly how should we conceive of that kind of loss of something that never was. But there's also kind of reasons rooted in the past. So we've had more than 10,000 generations of people. before us who've built up this complex world that we live in, passing down their improved knowledge over time until almost everything I can see in this room around me is not a natural item, whether it's technological in the sense of what we say technological today, or something even like a drinking glass, which is still a technological artifact it took us. many thousands of years before we could develop glassblowing and so forth, or woven textiles and so forth. There's been this intergenerational cooperation where our society would be impossible without this partnership of generations. And then if our generation were to drop the baton and be the one who... who kind of failed to pass that on and to kind of, you know, continue to improve this for the future. It seems like, you know, in some sense, we might owe it to the past to be part of this cooperation. You know, it's one of these pay it forward type situations where there's nothing we can do to help our ancient ancestors. But maybe the right role that we're trying to play in this cooperation, you know, is this paying it forward. So it might be this real dereliction of our responsibilities. On top of all of that,

It kind of comes to the questions that you were mentioning, which is that it may be that there's, as well as the present, the future, and the past, it may be that there's some additional kind of cosmic significance to humanity, where it may be that the Earth is the only place in the universe where there is life, or the only place where there is consciousness, or the only place where there are beings that are moral agents. So beings that act systematically. to steer the world towards what is good or what is right in a way that maybe even if there were no humans, but there were other mammals, maybe they would morally matter, but they wouldn't be trying to move the world towards better situations. If they turned out to be destroying the ecosystem, such as through some kind of byproducts of their living. they wouldn't be able to kind of realize that that was a bad thing and to act so to correct that kind of side effect. And so maybe we're the only kinds of beings who can do things like that. So that's focusing on what makes us cosmically unique and maybe we're the only places, as Carl Sagan said, where the parts of the universe can come to know the laws of the universe itself. But I think that... That even if we're not unique in that sense, there's still a whole lot of importance which come from these other senses. There's still all of the deaths that would happen during the event. There's still all of the future lives and the meaning in those lives that would be lost. And there's still all of this failure to live up to the kind of hopes and dreams and expectations of our ancestors who gave us this rich life that we have today. Well, thank you for humoring my unlettered nihilism. That was certainly a part of the book that got a wow exclamation mark with some of that Carl Sagan piece. And then, you know, I think you give such a beautiful personal example around this generational compact of when you had your daughter and you realize what it is that your parents gave you and ask how, you know, I'm never going to be able to repay this of you. And, you know, they sort of tell you, you know, that that's just how this goes. You can only pay it forward. It's not possible to, you know.

pay it back. Yeah. I mean, that was a crazy experience. This realization, I really felt this kind of weight of... Not quite responsibility, but I guess gratitude, but overwhelming gratitude. And also realizing how little I did to tidy my room or help with the housework and how I felt that it was some kind of affront to even pay back like 1% of what they did for me. Not realizing that it was only this tiny fraction. If I'd realized the full magnitude of the thing earlier, I think I would have been there. a bit better as a child. So I appreciated their answer. It was certainly a very helpful answer as far as I was concerned. Another kind of answer would be to say, well, we're getting old and you could visit us every day and move back to Australia. Yes, very true. Which, I don't know, if they'd asked for that, I would have certainly had to think about it because, yeah, no, it's... I really take seriously this magnitude. And I also take seriously this magnitude of what we, you know, this kind of gratitude. yeah to all of these other generations that if you just look at kind of any any little thing around you like i'm looking at the computer keyboard and as well as the materials it's made of of like you know like aluminium that that is extremely difficult to work out how to refine that you need high energy and so on but also the you know the keys as well as being made of plastic which has its own kind of complications but that the letter forms you know that they're yeah who you know And the development of the language and the electronics inside it and the rubber of the cable and just how this has been kind of come together from all different kinds of plants and minerals across the world and different processes. Even looking at simple things like things made of brick or wood and the improvements in carpentry and so on to get a surface as smooth as this and the developments of sandpapering and planing.

that you could just look at kind of any item and even just a very simple thing and realize that in a lifetime of work, you know, you kind of wouldn't be able to create this thing from the ground up. There's just this, you know, in terms of standing on the shoulders of giants, I think it's a very useful kind of humility lesson to look around you and really appreciate it. Yeah, that's a good exercise. It's almost... gratitude exercise, to put it glibly, but profound. With all of that said, perhaps we should talk a little bit about AI. And I was interested that you said it's not clear if that risk of an unaligned AI is higher or lower. What are some of the different factors that muddy that up a little bit? Yeah. So back in 2019, when I was putting the finishing touches on the manuscript, The cutting edge AI systems were reinforcement learning game playing systems. So think of things like AlphaGo by DeepMind or the StarCraft and Dota playing action game type systems. So these are systems that through this process of reinforcing them, giving them kind of rewards if they do something. right and kind of punishments if they fail and letting them kind of explore the space of actions in a game, you can often move very quickly through the human level, you know, from kind of very weak play, you know, quite rapidly through the human level. And then I mean, I think in the case of the chess playing system with Alpha Zero, that it... within a day or something. It moved from complete, worse than a human amateur, just random moves through the human level and to superhuman play. I don't know if it was minutes during the human zone or something like that. And so that can be a very rapid ascent. And then also these systems are very inhuman. They don't understand language. It's extremely unclear how you would...

explain anything to do with morality or what people value. So if you had some industrial system, suppose you use these techniques to train something to run a company or to produce a factory that produced a lot of widgets and trying to maximize the number of widgets it could produce or something. It might just do all kinds of damage in the course of doing these actions. because it kind of wouldn't understand that the world around it was important in some intrinsic way. But now we've got these systems that are very different to that. Here are like four key differences. So one is that these large language models are trained based on human data, on a huge amount of human data. So about 10 trillion words of human data. And they have read pretty much every book and every paper so that they know a lot of moral philosophy. If you ask them about the details of population ethics or various abstruse things, they can give pretty good answers to them all. And as Stuart Russell pointed out, pretty much every work of fiction is just in every page. There's some human judging some other human's behavior and whether it was found inadequate or not and why. And so they've witnessed a whole lot of these kinds of statements so that they would understand if the AI were to take certain kinds of actions, whether humans would judge it as good or bad or perhaps whether it would be ambiguous. And there's just this vast wealth of information to have access to about that. so that the information is in there, which is good. Also, if you train on human data and the pre-training phase of these AI systems, the first phase, is this next token prediction. So you look at a whole lot of words and you try to say, what's the next word that's going to come up? And that's ultimately a form of imitation learning, where it's learning to kind of pick the kinds of words a human would pick next. And that fundamentally pulls you towards the human level, the human range of behaviors. rather than doing some kind of bizarre set of behaviors that are totally different to us. So unlike things that might...

kind of just zip through the human level and then overpower us very suddenly. This is something that had very rapid improvements, but pulling towards the human level, which is nice. So it might at least spend some time there before we apply additional techniques and allow it to kind of take off again beyond the human range. And then also these things are fundamentally not agents. So a raw... large language model is just one of these next token prediction things. Once we do additional what's called post-training, so additional stages of training, such as reinforcement learning from human feedback, where we get them to produce some different answers, and then we mark the answers and tell them which one we liked and which one we didn't. That helps teach them how to follow instructions and things like that. And at that point, they are in some sense are like some kind of minor agents because they're They're not just pure imitators at that point. But there's still much more simple one-step things that don't do a whole lot of planning. They're not planning 18 moves ahead or something like that. So you've got these systems that are more like oracles than they are like agents. They're more like just things that you ask it a question, it gives you the answer. they could be much safer. They can avoid these types of cases where there's a subtle misalignment and then it goes off by its own, you know, and amasses a lot of power and potentially works out how to get humans out of the way. And then even then also the AI systems are generally very inscrutable. It's very hard to understand what they're thinking, but there have been quite good developments in interpretability of the neural net weights. And on top of that, when we... build reasoning systems out of LLMs, by default, they reason in natural language, like in English. So you can kind of see what they're thinking. So they're just like, you know, a whole lot of ways in which the technology is less scary. It doesn't fundamentally start with goals. It doesn't start with long-term planning. It knows a lot about human morality, even if it's not necessarily influenced by it. And so the step is then to how to turn that knowledge into something that guides it.

But that's a way in which the technology that we've been given that was somewhat difficult for anyone to anticipate five or 10 years ago has turned out to make alignment a bit easier as a problem compared to what it could have been. That is really interesting. I've obviously, as a tech writer and someone involved in the venture capital ecosystem, followed a lot of this. But I think the way that you just... framed that up so crisply was really clear. It's this sort of reinforcement learning style where you have really unpredictable behavior that doesn't have the context of the world around it and isn't born in a human mind in some way. And then the rise of LLMs has more of this context and more of that human style of thinking. This episode is brought to you by Brex. Fred Adler, the influential venture capitalist of the 1970s, was known for displaying decorative pillows in his office that featured a signature business philosophy. Corporate happiness is positive cash flow. In today's post-SERP environment, Adler's wisdom feels particularly relevant as founders need to make every dollar work harder. That's exactly what Brex delivers. Their modern finance platform was built specifically for startups like yours. and designed to help extend your runway when capital efficiency matters most. With Brex, you get global corporate cards with up to 20x higher credit limits and no personal guarantee required. Their banking solution has no minimums and no transaction fees, while letting you earn high yield from day one with same-day liquidity. Best of all, Brex knows you were born to build, not juggle spreadsheets and finance tools. Their AI-powered platform brings cards, banking, expense management, and travel all in one place. It's simple, scalable, and designed to get you back to what you do best, building. More than 30,000 companies, including one in three U.S. venture-backed startups, trust Brex to help make every dollar count toward their mission. Join them at brex.com slash Mario.

evolution over the past few years, where do you see the biggest risks remaining? Is it that there's some new paradigm that shifts things back in a frightening direction? I could also imagine that simply extrapolating from this point could create significant issues, maybe not existential, I don't know. But yeah, what should we be paying most attention to? Yeah, so I think it's a good question. And even in the case of alignment risk, it's not clear that it's gone down. So I said the technology is in some sense better, but maybe the world around the technology is worse. So we'd had a certain amount of racing between different AI labs, even back five to 10 years ago. But that really went up a notch once Microsoft and Google got involved. And now we're in a situation where it's not just racing between small labs of specialists, but rather between trillion dollar companies, the largest companies in the world. And nation states. And nation states as well, yeah. So this geopolitical race between the US and China is also very worrying. And so the pressures from these races and also the general... commercial pressure that comes from, I guess, the capitalist system applied to this race are very fierce. And so it's ultimately, if you look at, say, DeepMind and OpenAI, they were both created, I think, by idealists, or at least by people in their more idealistic moments. And there was a bit more of a chance that there'll be room for that idealism rather than these additional kind of pressures on them to race. So that's an example of where that one has also gotten worse in some way, even though it's gotten better in other ways. Yes. But then, yeah, back when I was writing the book, I really felt that the...

that there were a number of different kinds of risk from AI, but the chief one was this risk of AI takeover or alignment risk. You could think of that as one AI system or a small number of AI systems that are fairly similar to each other, maybe different copies of the same model. Realizing that to achieve their somewhat inhuman and misaligned goal, that they needed to wrest power away from humans who would stop them if they saw them attempting to achieve this goal. And that they would try to accumulate power and eventually overpower us and take control of our future from us in a fairly abrupt way that would involve violating our rights. But another possibility that's arisen, I was aware of these possibilities already, but they seem smaller. And now I feel that these are all roughly equal in size or threat. So another possibility is what gets called gradual disempowerment. And I think of this as a case of, suppose the AI systems never violate our rights, that we maintain sufficient control over them, that they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do, which often uses the definition of artificial general intelligence. If that's the case, we should expect that over time, even if we get richer through trade with these systems, that a higher and higher share of the wealth eventually accumulates in AI hands. So ultimately, they're more effective at doing what we do and they out-compete us and their wealth grows faster than ours. And then at some point in time, they have half the wealth. And then a generation later, they have 99% of the wealth. And with wealth goes power. They could use that wealth to influence political elections, even if they can't themselves run for election. And we just might expect, just as if you're a minority in a country where some other subset of people...

Even if your wealth is increasing, if others are increasing much faster and the share of control of resources is all flowing into their hands, you might expect it to be a bad outcome for you and a bad and permanent outcome, perhaps. So that's a gradual disempowerment story. And then there's also other ones, such as what's sometimes called AI coups or AI dictatorship, where an individual, it could be the head of a country or it could be, say, the head of an AI lab. Or it could even be someone, say, who's not the head of the country, say someone else in the White House or in the AI lab who tries to seize control of the AI system and have it kind of secretly uses it to try to plot, you know, to take over. Ultimately, if these AI systems eventually do become capable enough that they could seize control of the future by themselves, for themselves, then they'd also be capable enough that if they were what's called intent aligned, so that they could do what humans actually want them to do if asked. then that a human could ask them to take over on their behalf. That's another kind of concern. And normally there would be safeguards against an AI system doing these things, but maybe the human, if they run the AI lab or they work there, maybe they're in a position to remove those safeguards. Or if they're the leader of a country, they could command that they be removed. So that's another kind of concern. And then a fourth kind of concern is the building of various forms of weapons of mass destruction. using AI assisted help. So in particular, the concern there is that AI systems could kind of democratize newer and kind of more powerful destructive abilities. So to kind of empower, you know. Many individuals, perhaps people just with an undergraduate biology degree to create some world-destroying virus or something like that. It doesn't have to be biology. It could be in other disciplines if new capabilities appear. So I think that those four different areas are all concerning to me. And I'm also troubled by the fact that I don't know which is the most concerning. And I'm troubled by the fact that a lot of the things you could do to help with one of them make the others worse.

Some of them, for example, the issue is concentration of power. Other ones, like the weapons of mass destruction, the issue is the dispersal of this power. Open weights, for example, helps with one of these issues and hurts others. I feel really at sea for a lot of these levers that people argue about. I really don't know which way to pull the lever, whether it's actually helping or not for the overall situation. reassuring in the very small sense to me and disconcerting on the broader scale, because you clearly think about this at a much deeper level and much more than I do. But I do find myself really struggling with figuring out what level levers can we actually pull here that unilaterally help. And on one dimension, the race dynamic is something I keep struggling with figuring out how we get around that, because if we sort of assume that the major actors of these AI labs are morally normal, which we could probably debate that. And there's probably actually significant variance, I would suggest. But just the sheer incentive structure and mindset that you get into when you are in these race dynamics really feels like it leads to... A lack of circumspection, lack of sort of caution. Are there good examples historically of taking moral action in a race dynamically? You know, there were some perhaps from the nuclear race, but that was quite incomplete as you share in the book. Yeah, I mean, I would say the mainly bad lessons, lessons about bad behavior rather than good behavior. I mean, one example there was it. the Manhattan Project that I think is particularly salient is that if you ask why were all of these people, like these academics who studied physics, working on building a superweapon that would be the most horrible weapon humans have ever built, and the answer for most of them was that they were concerned about Hitler's Germany having unrivaled access to those weapons and holding the world to nuclear blackmail.

worked as quickly as they could to develop such weapons. But when Hitler died, when he killed himself and then shortly after the war in Europe was over and Germany surrendered, they didn't stop. Only one of the atomic scientists actually stopped at that point. And you'd think that if there was one reason that was so powerful that it led you to make the worst thing that humans have ever made. And you hadn't yet finished making that thing. And then that reason disappears. You might hope that it would be time for re-evaluation. And there were some reasons why, you know, why America used the nuclear weapons against the Japanese. But they were certainly much weaker than those other reasons. And I don't think that the atomic scientists would have signed on to the project, you know, with the idea of shortening the war in the Pacific being the objective. So at least not many of them. So, one of the things we learned there was a group of people doing something that had serious problems for reasons that they'd convinced themselves were important, but then just no longer being sensitive to those reasons disappearing. And there's a whole lot of cases like this. I mean, another example of, I guess, a smart person talking themselves into a bad argument. I can't remember who it was, but the chemical weapons for the Germans in World War I. has these written documents saying that he thought it could shorten the war. And that even though it could kill a lot of people, the bullets were also killing a lot of people. And that if the war was shortened, then ultimately fewer people would be harmed by it, both soldiers and also the civilian populations. And so he thought that ultimately, while the weapon itself would be brutal, that this would fundamentally be a good thing. Well, a big thing he didn't realize, even on his own logic, was that Germany lost the war. And if you were helping the side that lost the war, that means that your technology was lengthening the war, not shortening it. And so the idea that...

He was smart enough to develop all of these chemical techniques and things and to run this argument, but not smart enough to realize the argument only works if your side wins the war and that there was like a 50-50 chance that your side would lose the war, in which case it doesn't work. Yes. I think there's a lot of these things where we see smart people, in that case, maybe not someone who is particularly idealistic, but it shows that it's quite easy to talk yourself into various things. And I think that there's a lot of that going on at the moment, particularly when it comes to racing. Yeah, you talk about the unilateralist curse in the book, which is sort of the idea that all of us may judge these different actions that we could potentially take to, let's say there are... 20 people who might release the next AI model. Whoever is essentially most optimistic and sort of has the most Panglossian view of what might happen, they're the ones who release it. And then if it actually causes damage, everyone sort of reaps the damage of that or incurs the damage of that. And it feels like there is some of that happening. Exactly. So just like how in an auction, it goes to the highest bidder. And so if all the bidders, if there are many different bidders and they all have somewhat noisy and inaccurate estimations of the value of some object, the person with the rosiest impression of the value of the object, you know, will be the one who wins the auction. So this is called the winner's curse that sometimes you can systematically end up, you know, the very fact that you won probably means you overpaid. um uh issue here where yeah if a whole lot of people could all unilaterally make some action happen you know maybe it released the genome of smallpox or something like that and a lot of different people are trying to think about the benefits and the drawbacks and there are serious benefits of doing that kind of research the same with say gain of function research where you try to make um You try to make a virus more transmissible or more deadly in order to understand what mutations are needed so that you can be on the lookout for those mutations occurring naturally. There is an argument, a rationale there, but it's not obvious if that rationale wins out versus the harms of doing that. But if you just allow anyone to do it, then it will always be the people who end up with the...

the most inaccurately positive estimation of this action, who will be the ones who will make it happen. And so this is an interesting form of coordination problem. And there are ways around it. For example, you could get together with the other group of people who could all unilaterally make something happen. So say all the other people who could be releasing a model like this. And you could say, let's have a vote. And if you just take the vote, a vote ends up reflecting the median estimate. the middle estimate, if you do a majority vote. And so that actually almost completely resolves the problem if you can do it. In the case of a model release, it could run into antitrust issues if there's an issue of collusion to slow down a technology. So unfortunately, I think antitrust law could well be quite bad for us when it comes to these situations. There are appropriate mechanisms whereby companies can get together if they create a third. body, a standards body, which is open to new members and not just a kind of fixed list of members. Then you get that body to recommend a course of action and then you follow that action or something. There is an official way to do it. I worry that the extra effort and steps involved in doing that slows everything down, as in it slows these remedial actions down while the other actions are being sped up by the pace of investment, need to make a profit and so on. Given that you're sort of using the example of antitrust, do you think if we were able to resolve some of this race dynamic within the US, would that be sufficient? Because the sort of thinking there would be that the most advanced models and most advanced capabilities are US-centric right now. And so by sort of removing this dynamic, we're... you know, producing the existential risk from unaligned AI? Or is the fact that, you know, China is clearly taking this very seriously, even if they do seem to lag in some considerable places and have been, you know, cut out of the semiconductor supply chain in really meaningful ways? Does that sort of fundamentally, just by controlling the US, invite a different race dynamic where we're sort of actually hamstringing Western powers and leaving the possibility of unaligned AI in a

much less democratic environment. Yeah. I mean, this is another thing that's related to the unilateralist curse, that if there's some issue that seems... at some level problematic, let's say taking some new step with a technology is at least at some level troubling, then if all of the people who are concerned about such things all stand aside and say they're not going to do it, then the person who has the fewest ethical qualms will be the person to step across the threshold and maybe we'll get the first version of something will be a more dangerous version of it. So that is a general concern. I think that it's also one of those, it's exactly the kind of thing that... people in the position of the atomic scientists could use to rationalize racing. And so while it is not a bad idea, it convinces different people different amounts. And then you kind of worry that the type of people who are erroneously convinced by it will be the people who take the actions. I think it's a real challenge. So as you say, there's kind of racing, roughly speaking, at two levels at the moment. There's this kind of... company racing between different companies, most of whom are at least headquartered in the US. And then there's a kind of racing between the US and China. So the US is clearly the one that triggered the race between the US and China. And partly it triggered it because on the rationale that eventually the race was going to happen and we want to win it if it does happen. And so if we start the race, we've got more chance we'll be winning the race. And there's a logic to that. But it still is a case of You know, if you think races are bad and the racing behavior is kind of like defecting in a prisoner's dilemma, and you think that unfortunately the structure of the world is such that probably both parties are going to defect, so therefore we should defect first. Maybe that's the right thing to do, but it's still kind of, you know, still jerk behavior at some level. And so I think, you know, one has to admit that. If, you know, if ultimately we succumb to some existential risk with AI, if you kind of imagine, you know, the different powers that be or all the different people involved kind of being called up to the pearly gates, you know, meeting St. Peter and trying to explain their behaviors. It's a little kind of thought experiment. You'd have to get into some Gabe theory. It wouldn't sound great, I think, if you said fundamentally, we knew it was really risky, but we thought people would do it, humans would do it, even though they shouldn't.

We thought there could be a race. So we started the race. And did you attempt to have a treaty to like not race? No, we didn't attempt that. We thought it wouldn't work. So we'd started the race and then we raced and we got there first and we built the thing and the thing killed us all. I just think it'd be pretty difficult to really, you know, if you add a kind of like imagined your honor at the end of these statements, they don't sound great to me. And I think that as it happens, given that China is behind in this technology, I think that they have additional reasons to not want to race. So if the US were to think, as I think they should, that we would rather there be no race, and China would prefer that even more because the race that there would be is a race that I'd likely lose as well, then I think China could be in quite a position to be happy with some kind of verification conditions or things around kind of avoiding a race. So I feel that there's been very little thought going into not racing. And I think that if you pretty much just had those two countries agree, they could potentially police their own spheres of influence to say, okay, no one here is building things towards superintelligence. No one over there is doing it. And then have verification on each other's AI industries. Such that no one cheats it, essentially. Yeah. For example, by saying, okay, well, we'll send 30 of our people over to be part of Europe. AI companies and you can send 30 of yours to be part of ours or something to make sure that we're not doing it. I think that there are ways to do this. In the Cold War, I think there's a lot of lessons from the Cold War. Ultimately, there was a situation where the US and the USSR were very much adversaries of each other, if anyone is. And yet they actually treated each other as adversaries in what I think is a somewhat grown up way that we're not doing these days. I think that the US is treating China as an enemy, not an adversary. And they just kind of want to get China. Whereas with the USSR, I think there really was a realization that there are a bunch of things that were in their common interest. Not everything, but there were some things. So for example, the non-proliferation treaty.

was in their common interest. If the US has nuclear missiles and the USSR does as well, they don't really want all the other countries to have nuclear missiles. It's in their common interest that this would be an elite kind of exclusive club. And it was also, I think, in most of the world's interest. And so they helped to broker and police this non-proliferation treaty. And similarly, their arms reduction treaties that they had, because they both would have rather that they had a tenth as many missiles as they actually had. And they managed to do that. And there were very smart people working out very novel verification techniques. You know, there were these ideas of, once spy planes became possible, of allowing the Russians to fly spy planes over the US airfields while they soared their bombers in half. And they pulled the halves apart from each other with tractors. Wow. And those bombers are still there on the airfields, sawn in half as a costly demonstration of destroying their military capacity. Wow. Even without inviting inspectors over, they worked out ways to credibly show that they were reducing their arms. And so it's, you know... there were really people thinking outside the box is what I'm saying. And there's a lot of very smart people today thinking about these issues. A lot of them are going to work at AI companies and build these technologies. But there's also a lot of kind of complaining that we have of like, oh, verification's too hard. There's no way we can verify a kind of deal to not race. And I feel that the amount of, let's say, full-time equivalents spent on thinking about verifying a race between China and the US on AI, maybe there's like a... three full-time equivalent people working on that or something in the whole world if that yeah uh and there's probably almost nothing has been explored um so i think that that's uh there's a lot of just giving up oh that'd be way too hard um without really you know trying to work out are they like really clever technological or sociological you know techniques That's really interesting. I want to touch on one of the other areas briefly that you talk about in your book, which was the extremely prescient and immediately prescient part, which was pandemic risk. You mentioned gain-of-function research and sort of thought through the trade-offs there. What have been the lessons of COVID that you take from that kind of research in general and how risky it is? And more broadly,

Did COVID make us more prepared in some ways for what comes next? Or have we really not taken almost anything from it? So one thing it did was that when I was writing the book, I was certainly concerned that people were just not going to take threats to humanity's entire future seriously. that there was a feeling that, I don't know, that there was a time when these things were possible and that that time was in the past. Maybe in the case of pandemics, a feeling that, oh, sure, there was this 1918 flu that I've heard of. And there was the Black Death, I guess, but that was medieval. And technology's advanced so much since then. And we do so much in our hospitals. We understand the germ theory of disease and all of these things. And so we're doing much better at this. And so it's not a risk. And COVID certainly reminded us that we're still at risk and that for all of our technologies, in some ways, we're just as vulnerable as we were before. So when it comes to dealing with individual infections and trying to save someone's life in the hospital, we're much better at that. But overall, when it comes to the spread of disease, because the airplanes that we've invented and things, you know, the cities that we've built and the public transport systems and so on are so efficient at just spreading this disease around. A lot of our technologies make it worse at a similar pace to the way that our protective technologies make it better. And it's not obvious at all that we're actually more protected overall when it comes to pandemics. So that's... you know, it's helped give us some of that kind of reminder. And also, it also was this feeling, I think that, you know, when it started, right, there was this thing where you're like, hang on, what, you know, is it going to be the case that everyone in my country is going to be at home in their house? Yes. And the streets are just empty for days, you know, days, I mean, months. Yeah. Is it a school's going to shut down? And there was this feeling of like, is that, should that even happen? And in general, I think.

You know, for people who'd lived through, say, the Blitz in London, you know, that really shaped their view and they felt, oh, well, we're not even allowed to have lights on at night or something. And they've been aware that the space of ways of living, you know, could be quite different in the space of kind of big things that could happen could be quite big. Ultimately, COVID was, you know, was probably this biggest world event since World War II. Yes. I feel like there's this kind of level of... what's the biggest change to life that you can imagine is set by the biggest one you've lived through so far. And so at least it's helped raise the ceiling on what we can imagine could be required of us and how different life could be. So that's been somewhat useful for kind of letting people imagine more. I mean, also, I guess, in terms of political trends and, you know, political populism, you know, as a rise of this movement and so on. If, you know, if you lived from... uh you know through the 90s and the the 2000s and uh you know it had really felt like we're in some really stable era of some sort. And then we've come out the end of that. And that took quite a while to learn those lessons and to realize that, at least for someone of my age, that the only world I'd known as an adult could be coming to an end as a geopolitical order or something like that. It's helpful to take that wider view. And COVID has helped us do that. So in some ways, it's helped us realize that for other threats as well. It was thought by almost everyone that there would be what's called a panic-neglect cycle. where having been hit hard with something, that you overreact and that the governments would kind of would do more than they should. Either they'd spend more money or they would kind of like inhibit freedoms too much or something like that to react against this thing. And then that would slowly fade away until it was too little. But that hasn't happened. We actually didn't get the panic phase of that. And, you know, by the time everyone had reopened after COVID, I think people were just so sick of it. that they didn't even want to talk about funding these bills to spend even a tiny fraction of the amount of money that we were cost or the amount of damage that was caused by COVID to prevent future such things happening. So that was a really missed opportunity. And I was surprised. I would have thought that you'd get this panic. And then the question was how to soften the panic into the appropriate level of response, but then how to maintain that appropriate level as well. And yet, no, we never even got the appropriate level.

And I also feel that it had some good features like the mRNA vaccines, getting kind of a test out and kind of flexing our muscles on this new technique of very fast vaccine development was good. And I guess I don't know overall whether... how it's really helped our preparedness. I think that, again, you can tease out four or so of the biggest factors that either made it better or worse, or perhaps in some cases have not made it better or worse, but revealed underlying structural weaknesses. For example, it didn't take all that long before there was a lot of acrimony between different political groups or between different countries about who is at fault and who should be handing over the vaccines and various things like this. I guess we learned that we can band together for a while. But a while might be a six-month period. If the crisis carries on for years, that that really does start to fragment. Yeah, but I think that's true. Your point on the way in which this totally shifted our frame of reference from this can never happen, that would be as fanciful as a zombie apocalypse to actually we all lived through it. And also, you know, Operation Warp Speed with the development of vaccines is maybe a good example of a beneficial race dynamic and where those things can work favorably. Yeah. I mean, it's even, you know, I think the Operation Warp Speed was a successful government, you know, slash private, you know, cooperation and so forth. Yeah. You mentioned sort of the political landscape and the institutions that... you know, played their parts in COVID. One of the things that I've been thinking about in reflecting on the book and some of these questions is how compatible are democratic institutions with addressing with existential risk? You know, there's sort of this time span of discretion where your senators are thinking on, you know.

I think they got six years. I actually can't remember now. Six year cycles, your presidents are thinking on maybe two and a half because then they start thinking about, you know, the next election and so on and so forth. No one is really incentivized to think about maybe even 10 years. That would actually be a really farsighted person. How can we sort of address these existential problems with such a short term mindset from the institutional level? Yeah, so it's a real challenge. It's not necessarily, you could say, a challenge of democracy in the sense that if you didn't have democracy, if you had some kind of autocracy, it's not clear that makes it better at all. If the autocrat happened to care about this issue, maybe it would make it better. But if they don't happen to care about it, then they wouldn't really be able to get the signal coming from the people. I'm not sure that it's democracy itself that's to blame for it. Something's happened to the democracies that we have in Western countries, at least, to really shorten the timescales that they focus on. I'm not exactly sure what's going on there or why that happened, because it does seem like the democracies, if you go back 50 or 100 years, it does seem that they were able to achieve larger projects and projects that operate on longer timescales than they currently can. conceptualized. Maybe it's to do with the complex relationship between the democratic institutions and the media and the success of certain forms of media and punishing them for not preparing an answer to the question they asked on Monday or something. Everyone's thinking in terms of the news cycle, not even the election cycle. I don't know what it is, but it does seem like the time horizon for politicians has gotten shorter. There's also, though, separately from whether it would be better to not be a democracy, there's a question of, is there a kind of failing still of the current democratic system to perhaps live up to the principles behind democracy when it comes to this issue? And I think that there may well be. So when we think about the most important decisions that policymakers are making in our time, they may well be decisions such as what they're doing about climate and what they're doing about, say, technologies like AI.

For example, deciding actively to not regulate and prevent regulation seems to be an active decision to do something negative there. But these are decisions that, if you make them in one way versus another way, may well have implications that just echo through the generations, making people's lives significantly worse for many generations to come, or perhaps not even getting a chance to exist at all. There's this issue that most of the stakeholders or the people who are affected by it don't have representation in that decision because they're people of future generations. In some cases, we can also predictably work out which way they would vote. They would vote for doing more action against climate change, for example. It's not even just that there's this kind of unknowability, so we couldn't possibly factor it in. And so there's a kind of constituency or stakeholder group that is just not getting the franchise. Yeah, that's really interesting. It's somewhat similar to questions about people of different races or sexes or landholders versus non-landholders and other types of issues in democracy of extending the franchise. And so I think that there are potentially things that a democracy could do about that. For example, in the UK system where we have the House of Commons. you know, which is a bit like the House of Representatives in the US. We also have the House of Lords, which is an appointed body, some of whom are appointed by the Prime Minister, some of whom are appointed as the kind of good and great of society, you know, former heads of the Royal Society and things like that. And it will be possible to appoint a group of Lords to represent future generations, or you could do it in other ways. You could have a Citizens' Assembly, for example, representing future generations. And I think that experimenting with tweaks to the current understanding of democracy to deal with the fact that now so many of our choices, our most important choices, could be affecting all of these stakeholders who don't have representation. You know, the idea of kind of no taxation without representation, kind of foundational to the American ideas of democracy. And yet, you know, maybe we need to take it seriously in this case of future generations and to work out, are there ways to appropriately...

understand their interests and then also to force some kind of representatives to represent those interests. Because obviously the people themselves who live in the future can't vote in this thing. It doesn't mean they can't be represented. That's a really fascinating idea. Yeah. Having these sort of stewards of the future as part of the governance process. Or even if it seems to be too impractical to at least set it as an ideal to say, oh, if only we could do this, that would be great. Unfortunately, we can't or something. But it seems at the moment people don't even think about it or don't realize that maybe there are things we could do about it. Yeah, there's sort of the implication that people suspect or sort of hope that politicians are thinking about the future. But I think the reality is that that is, as you said, has sort of long since gone. Yeah, so that's a nice way to at least consider it. And we like to sort of end episodes with a few sort of more abstract thought experiment type of questions. So with that in mind, if you had unlimited resources and no operational constraints, what experiment would you like to run? I guess I think I have no idea. Oh, wow. Great. OK, that's a apologies. No, no. This is a sign that you're you. You wouldn't want to go off on a wild experiment without thinking it through, which. makes sense given your background. I guess that's part of it. There's a famous comment from Carl Sagan when he was asked about, are you really sure about nuclear winter? How could you know? The experiments never happened. And he said, It's unfortunate that we have no spare Earths to destroy in some kind of experimental laboratory in order to determine the exact reliability of this theory. So if we had unlimited resources, we could have those spare Earths to destroy in order to find out what these risks actually are. But unfortunately... That sounds like the kind of experiment that's probably ethically a no-go regardless. Yes, that's right. That makes sense. Well, let's try this one. If you had the power to assign a book to everyone on Earth to read and understand, what book might you choose? I might choose Practical Ethics by Peter Singer as a book that's...

It's a really good kind of no-nonsense approach to thinking about ethics. It really helps show how to kind of reason about all kinds of issues with an open mind, not assuming that our current kind of ethical intuitions are the be all and end all. And I think it's a good example of that kind of thought. Is there a sort of a particular framework or sort of lesson that you particularly reflect on from that book from Singer? I imagine he's a major influence of yours. Yeah, he is. Yeah, probably just really what I said then of the fact that he just kind of models this kind of really open inquiry into moral matters. Derek Parfitt mentions that ultimately there's been only really a couple of generations in which people have made what he calls non-religious ethics their life's work. So if you look at the whole history of philosophy... Almost all discussion about ethics had been within a religious context where the set of possible answers always had to kind of align to what had already been thought thousands of years earlier. But people trying to, separately from that, to try to understand these questions, there'd only been a couple of people in the whole history of philosophy. And then there'd only been a very short time since it became. more widespread. And so he saw that as a reason to be very optimistic about the possibility that we'll make some substantial moral progress. Fascinating. What tradition or practice from either another culture or time period do you think we should widely adopt? I think I don't know enough about particular time periods. This is the problem about being an academic and wanting to not just make something up. But I do feel that something to... And something that involves more gratitude to our ancestors, I think, would be particularly important. I know that there are many cultures that have more gratitude to our ancestors than Western cultures, which have almost none. But I'm not sure exactly who we should be copying it from. But I would love to see more of that. Yeah, I agree. That's a great one. Well, Toby, it's been such a pleasure. And yeah, I can't thank you enough for your work and your writing and for your time today. Oh, thank you. It was a wonderful conversation.

That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.