The role of natural scientists and hardware engineers in AI safety

I am puzzled.

Readers, as a physical scientist with a small dose of professional AI experience I am puzzled. I am puzzled enough that the first draft of this essay was chock full of swear words. For more than a year I have been wondering why it is that so few natural scientists and hardware engineers have been drawn into AI safety work despite well-funded calls for advancements in this area. I’ve been reading serious people writing seriously for years now (and recently more loudly than ever) about how dangerous AI can become. And yet there’s been little movement from anyone other than some token biologists and a particularly AI-forward subset of roboticists. If the problem is really large and decently well-funded then this is odd!

Is it that we can’t actually contribute? I don’t think so.

To begin with, last I checked humans are material beings and so if AI is to break our bones there will need to be a step in which its words direct sticks and stones. This hazard¹ is real: AI may, for example, persuade and manipulate humans, may be given or seize control of machines, or may cyberattack physical systems connected to the internet. Nevertheless, in each of these cases the final harm must come in the physical world and so physical defenses can be mounted against it. On top of this, although AI is a software construct, that software runs on hardware and there is the opportunity to secure that hardware physically against the misbehavior of AI instantiated on it. Finally, many non-physical harms that AI might enable such as financial fraud can be mitigated by physical-world defenses such as paper records.

Natural science and engineering research also have much to contribute to AI safety. To pick just one example, the complexity of the physical world is such that it contains deterministic, chaotic, and truly random systems that defy computational prediction even with hardware and algorithms orders of magnitude better than those currently available². Handled correctly, information generated or transformed by these systems can be readily available to physical beings and unavailable to superintelligent AI. Concerted research in this area could unlock physics-based cryptography, honeypots (any AI with this information must have stolen it), and other AI safety tools. If humanity decides to be especially paranoid, we could even do some of this research in air-gapped experimental facilities.

There are also the perspectives that natural scientists would bring to such discussions. Take ecology for example. In his book Endurance about Shackleton’s 1914 Antarctic expedition, Alfred Lansing relates the story of how Thomas Orde-Lees, a member of the expedition, was pursued by a leopard seal while out skiing on an ice floe.

The animal—a sea leopard—sprang out of the water and came after him, bounding across the ice with the peculiar rocking-horse gait of a seal on land. The beast looked like a small dinosaur, with a long, serpentine neck. After a half-dozen leaps, the sea leopard had almost caught up with Orde-Lees when it unaccountably wheeled and plunged again into the water. By then, Orde-Lees had nearly reached the opposite side of the floe; he was about to cross to safe ice when the sea leopard’s head exploded out of the water directly ahead of him. The animal had tracked his shadow across the ice. It made a savage lunge for Orde-Lees with its mouth open, revealing an enormous array of sawlike teeth. Orde-Lees’ shouts for help rose to screams and he turned and raced away from his attacker. The animal leaped out of the water again in pursuit just as Wild arrived with his rifle. The sea leopard spotted Wild, and turned to attack him. Wild dropped to one knee and fired again and again at the onrushing beast. It was less than 30 feet away when it finally dropped.

Despite being the more intelligent creature, Orde-Lees was outwitted by the leopard seal’s cunning decision to track his shadow and saved only by the timely arrival of one of his crewmates. And creatures of even lesser intelligence all the way down to mosquitos, bacteria, and viruses don’t just routinely kill but in fact exert large scale control over humans. Bringing the right ecologists into the conversation could reveal myriad strategies through which “lesser” organisms routinely restrain or threaten greater ones. And that’s just one example of the dozens of scientific disciplines whose perspectives have not been tapped for AI safety work.

These all seem like decent reasons to pull us in. So what’s going on?

One possibility I’m seriously considering for why AI safety is primarily the domain of bits people is culture. I’ve led hardware-oriented teams on three continents (and was a visiting researcher on a fourth!), but the first time I managed a team of software engineers was the biggest culture shock of my career so far. This culture difference really landed for me the first time I used the git-blame command. In the world of bits, where everything is made by humans for humans, there is always someone to blame. Thus humility and its dark twin fatalism, which arise across the full spectrum of atoms cultures, never take root in bits cultures. AI team cultures take a big step in the direction of atoms cultures by dealing honestly with AI models’ inherent nondeterminism and the trial-and-error processes required to create them. Nevertheless, I observe that the AI safety community places both of its feet firmly in bits territory by centering its work on alignment and software detection of misalignment and misuse. Accordingly, AI safety practitioners see no need to pull in other perspectives except as they relate to these particular challenges on which natural scientists have relatively little to contribute and natural scientists can’t find their way across the cultural barrier independently. If this is what’s going on AND AI safety is actually important, we need to bridge this cultural gap. Otherwise, we risk missing approaches to AI safety that focus not only on the AI model and its software environment as a single point of failure, but on defense in depth³.

In their own domains, atoms people are masters of defense in depth. This is because, despite all contentions to the contrary, the fundamental building block of matter is the unpleasant surprise. If you are not familiar with this fact, I encourage you to read the chapter Silver in Primo Levi’s The Periodic Table, which (in Raymond Rosenthal’s translation) concludes

…stolid matter manifests a cunning intent upon evil and obstruction, as if it revolted against the order dear to man: like those reckless outcasts, thirsting more for the ruination of others than for their own triumph, who in novels arrive from the ends of the earth to thwart the exploits of positive heroes.

Remember how I mentioned deeply ingrained humility and fatalism? This is where they come from.

Consider, for example, the semiconductor clean room, designed to prevent contamination of the devices manufactured there. In the clean room, silicon wafers to be processed are carried in sealed pods as much as possible, AND every worker is gowned head to foot in shoe-covers, gloves, masks, and full-body suits AND the air is purified to be essentially free of dust AND every processing step is sealed if possible AND wafers are washed and etched in horrible chemicals at multiple points during the process AND all of this is monitored in real time AND STILL the final products are often sorted into bins by their observed quality at the end of manufacturing because some contamination is nevertheless inevitable. This is defense in depth. To force matter to behave, atoms people are willing to layer tools from disparate disciplines (From Acid baths to Zeolite gas purifiers!) and to nevertheless sacrifice or devalue a large fraction of the output they produce.

Nuclear power plants can give a sense for what this kind of defense in depth might look like when applied to AI. I like this analogy because a severe misalignment event is like a meltdown in which normal operation leads to dangerous conditions and severe misuse is like a dirty bomb in which unauthorized access to dangerous material creates the potential for harm. A single layer of this defense is illustrative: every three years, every civilian nuclear power plant is literally attacked by Nuclear Regulatory Commission personnel armed with simulated weapons and the security recommendations coming out of these attacks have the force of law. Cyber red-teaming of datacenters pales in comparison. Another consideration from this analogy is that nuclear power plants are increasingly being designed for passive safety. What would a passively safe AI data center look like? Is it possible to have an automatic kill switch trigger a full power-down of the building if the model cannot continuously prove that it is aligned? If the model connects to any unauthorized resource? Could this detection be done in hardware and not software? In hardware that doesn’t require power? And of course, would we be willing to accept a sudden shutdown? These kinds of power-down events are incredibly expensive in the energy industry, but we design for them because they are better than the alternative.

One other potentially fruitful analogy: one might think of AI as creating a new utility (intelligence) with a new kind of emissions (unwanted actions due to misalignment or misuse) and tap into principles for pollution control. Would customers pay a safety premium for AI? If green premiums are the point of comparison it isn’t very likely so perhaps we need to regulate the emissions from our growing intelligence grid. How would we do that?

It is not just in responsive defense where atoms people can contribute, but in hazard reduction through safe and effective use of AI as well. Thinking about the future of AI, Mathematician Poh-Shen Loh writes “The advance of AI will overwhelm us with so many control points to watch that there aren’t enough people to control them all. Those are jobs. Highly skilled jobs.” Atoms people also have a strong perspective on what to do about an overwhelming number of control points, as reflected in the design of airplane cockpits. A jet aircraft has more subsystems than a human could possibly monitor, and so we entrust those subsystems to classical controllers. The cockpit is designed to allow human pilots to navigate the aircraft and to identify and respond to subsystem failures and exceptional situations, including by taking manual control if needed. These meticulously designed user interfaces allow a small number of humans to reliably pilot a system that costs more than $10,000/hour to operate and whose failure would result in tragedy. Depending on where you draw the boundary around what counts as an airplane’s human machine interface, it costs between a few hundred thousand and a few million dollars to build and order-of-magnitude $100 million to design, but demonstrably has good ROI. We could do this for AI. Airplanes show us that we can build a machine that would enable a team of two humans to use $10,000 worth of inference compute per hour with hundreds of lives on the line. What might that machine look like?

Atoms people can help defend against society-wide challenges and capture the broad opportunities of AI as well. If AI and robotics realize the promise being held out by some of the more optimistic ventures combining them, humanity will gain significant capacity to work in the physical world and a substantial number of humans may be displaced from their current occupations. The last mass displacement of human labor on this scale was the great depression and one of the key responses was to build, which put a huge premium on projects that were shovel-ready when the need came. Reading The Power Broker by Robert Caro, one learns that Robert Moses was able to complete huge public works during this time of crisis by having wildly ambitious plans ready for construction and expert technical staff available to create yet more plans. What is the modern equivalent of New York’s monumental Triborough Bridge? Might we draw up plans to permanently terraform swaths of Nevada or New Mexico by flooding huge reservoirs with desalinated ocean water? To build new transit systems, new cities or even, an entirely new power grid? The Alameda-Weehawken Burrito Tunnel? We need atoms people to dream these dreams in detail, at the level of concrete mixes and bolt sizes, and to dream them before they are urgently needed. Helping society adapt to a world with AI might require a strategic reserve of pre-approved megaprojects.

Defense in depth sounds pretty good, right? In fact I’d argue that it sounds good enough that current AI safety researchers have thought of it already, which strongly suggests that there should be a role for atoms people in AI safety. However, the purpose of a system is what it does and the system of AI safety discourse, research, and funding currently appears to exclude atoms people. The question I’m left with is why is it structured that way and what can we do about it?

1. In the technical sense of the word: a bad thing that can happen, without reference to how likely it is to happen

2. See, for example, physical unclonable functions

3. Even when OpenAI says “defense in depth” it refers to only software defenses