Should Evil AI Research Be Published? Five Experts Weigh In. | Artificial intelligence

Everytime I attend a discussion panel like this I relearn just how unsuitable they are to discussion. This one had 14 panelists plus a moderator and actually went remarkably well, but it mostly just involved two rounds of 3-minute presentations. It was pretty interesting to hear about everybody’s very different viewpoints, but there was very little discussion and the moderator had even asked the panelists explicitly asked not to respond to each other. After these mini-presentations there was very little time left over for actual discussion and audience questions.

This question was pretty chaotic as the asker was continuously clarifying/modifying the question because the panelists were not really answering what he wanted to ask. My memory isn’t perfect either, but I think he started out asking something like “What should I do if I figure out how to make AGI? For instance, should I publish it?”, then he fairly quickly clarified that (this) AGI is dangerous, and I don’t think the word “evil” was introduced until pretty far into the discussion as an attempt to maybe get the panelists (especially Siegelmann) to at least acknowledge there may be some conditions under which it would be a bad idea to publish it.

I kind of wish Seán Ó hÉigeartaigh had been more assertive, because by the time he finally chimed in the discussion had fully glommed onto the issue of publishing to the exclusion of anything else, and he just responded as the article mentions. (I think he also elaborated a bit by saying norms are in place for such issues in other fields: e.g. synthetic biologists don’t just publish how to make new bio-weapons without going through some process. And I’ll also note that e.g. white hat hackers will ideally give their “victim” time to repair the exploited vulnerability before they tell the world about it.)

But I think it would be great if the director of the Centre for the Study of Existential Risk (CSER) could answer the underlying question: “What should a researcher (or group of researchers) do if they suspect they have a breakthrough idea that could bring us close to AGI capabilities (but not safety)?” One possible answer is “publish it”, but if it’s not (and it’s obviously not), then the question remains of what to do. Running it to see if it’s really AGI is obviously super dangerous (at least CSER believes it is), so that’s not the answer either. Presumably the answer is something like “try to make it safe”, but does CSER trust the hypothetical (group of) researcher(s) with that task? Should they be able to seek help or offload that task? Who could/should be trusted with this? Clearly not most people on this AI safety panel… I think that this is something the AI safety community should be thinking about, and that organizations like CSER should ideally be trying to provide guidance for.