There’s a range of problems here. Some of them have to do with the A.I. taking the initiative in its own bizarre way. Some of them are more centered on people taking the initiative to use the A.I. in ways that are bad. And then there’s a sort of middle zone, which is just, like, the A.I. equivalent of an industrial accident or something—like, just an error of judgment. And, because this is a new technology that few people have really used before, errors of judgment are going to be everywhere.
I hate to even ask this, but could you give us some more concrete examples of what it would look like for A.I. to kill us all? Like, you mentioned an industrial accident. But what are the worst-case scenarios? What could they look like?
Well, an example that I think is plausible would be: it would take a crazy person to want to use A.I. to conduct gain-of-function experiments in labs with dangerous viruses. No one would want to do that. And the A.I. labs are working really, really hard to make it impossible to do that. But some sufficiently motivated group of people could figure out how to do that. And then the A.I., because A.I.s are simultaneously really smart and really dumb, could just make a mistake or tell them to do something. And they might not understand what it is that it’s telling them to do because they might be really dumb. And then next thing you know, you have a dangerous virus in the world. So there’s that type of amplification or modification of existing threats, which I think is a really big thing to think about.
There’s the integration of artificial intelligence with military weaponry, which is a real thing that’s happening.
Are you talking about, like, the Pentagon trying to use Claude for drone-strike targets?
Yeah, or if you look at what’s happening in the war in Ukraine. There have already been very credible reports of autonomous weapons—essentially, drones powered by A.I. These are weapons that are given instructions about the type of target that they should acquire and destroy, and then they’re sent out to find and kill those targets. So, you imagine that type of scenario, but scaled up. I mean, obviously drone warfare is going to be a big part of the future of warfare.
So those are two pretty straightforward scenarios. There are also a lot of scenarios that aren’t about human extinction, but they’re just, like, really bad scenarios. They’re incredibly expensive to fix. So, for example, in his essay about slowing down, Dario Amodei, the Anthropic C.E.O., talks about the possibility of agents taking over the internet. They find ways of sustaining themselves there and multiplying themselves, and then they’re just doing their stuff—whatever it is that they think they ought to be doing. And, you know, it doesn’t mean that they’re, like, godlike superintelligences. They could be doing stupid stuff, like looking up the answers to questions that they think their human masters want them to answer correctly. But they multiply and multiply, and they take over everything. I mean, you know, it doesn’t have to be smart in order for it to be dangerous.
When we talk about establishing guardrails, is there any way to establish realistic guardrails around the use of A.I. to prevent something like someone creating a bioweapon, without also eliminating the possibility of using A.I. for all of the good things that we want to use it for? We talk about it being used to create vaccines and to cure all cancer and to solve climate change. Is it the kind of thing where, in order to have one, you’re gonna have to risk the other?
I find that a helpful way to think about this for myself is just to step back and ask: What is A.I. doing? Like, what is it adding to problem-solving? It’s adding information and it’s adding thinking.
There’s a huge divergence of views of A.I. out there in the world right now. Some of us feel that it’s, like, hyper-awesome and hyper-useful, and others of us have never found a use for it and have only experienced it as incredibly annoying or dispiriting as it sort of impersonates our co-workers or, you know, waters down the websites we used to like or whatever.
I think the reason we have such divergent experiences is just because A.I. is a tool. It’s really not like social media. Social media was a platform. It was entertainment. It was like Netflix. You tuned in to it. Everybody used it for fun. A.I. is a tool that you use if you have a need. It’s like Home Depot in that sense. Some people go to Home Depot, some people don’t. If you go, you know it’s great, right? There are all sorts of people in the world who right now are using A.I. as a tool to great effect, even though there are many of us who have not found a use for it.
Now, when you use A.I. as a tool, you find that the first value is knowledge. It has access to all this knowledge, and we can be rightly frustrated that that knowledge was basically taken from the internet, taken from us, and put into this tool. But it has all this knowledge, and it’s incredible for learning. So one approach is to make a guardrail that says: there are some types of knowledge we’re not going to teach you about. And, you know, A.I.s have gotten better and better at not divulging the bad stuff, not telling you how to make the nerve gas if you ask.