Human alignment isn't enough
It's barely enough for normal humans. One step further is too much.
They found it in one of the early Mars expeditions, a bit after they had travel back and forth figured out well enough to keep a permanent outpost manned out there. The lab ran expeditions into some nearby caves in the hope that they’d turn out to be a good spot for an expansion. That hope didn’t turn out too well - something about the local geology, they ended up figuring it’d be more cost-effective to just land more pods - but they found the Organism there.
Not that any of this impacted me much at the time, beyond my general interest in space news to take my mind off the problems on Earth. I was still living in Manhattan, doing my day job as a government think tank security analyst and hoping AI winter would last long enough for me to save up a bit before it could replace me.
They always called it the Organism, but no one was even sure it was organic. It was arranged in clusters of off-white hexagonal tubes, and there was certainly some kind of chemical process there - they’d grow on their own, even faster than bamboo shoots, but in a much simpler chemical process that had the biologists arguing over whether it qualified as alive. It had some emissions, but early experiments convinced people it wasn’t toxic, and the guys at the lab - the one in the mars station, no one was bringing it home to Earth yet - brought some home and started studying it pretty casually.
It took a while for anyone to formally notice something weird was going on. The geo lab, the one that studied the Organism, kept publishing banger results, and not even just about the Organism. They seemed to be ahead of the curve across the board - they got almost as much research done on soil research and solar cell adaptation in their off time as the pods that were actually studying that. Some people are just crazy smart, I guess, but the solar cell guys we’d sent to Mars had been top of their game and they were still getting lapped.
They started making jokes about the Organism getting the geo lab all high - you could smell it a bit, at least in the sterile sealed station air - and the geo lab gave the solar cell guys a sample as kind of a gag gift. Two weeks later the solar cell guys came up with a brand new solar cell design that pushed peak solar cell efficiency up by two percentage points. It wasn’t even just a Mars thing, it was deployable on Earth. Whole research labs in China had been working on that for years without getting close.
You can bet people took notice after that.
They isolated it on Mars at first, of course. You don’t risk bringing something that grows fast and has some sort of weird effects on human brains until you’re sure it’s safe. It didn’t look like it was a trap - even the serious people in government had to consider aliens messing with us at that point, but they explored through the Mars caves and there was no sign of any sort of life, biological or weirder. It was such a weird natural artifact that some people thought it might’ve been designed by a now-extinct martian civilization. The geologists had some arguments about how some rock formations on Mars were evidence that it had life at one point, a billion or two years ago. It was all theoretical either way.
The Organism itself was more complicated, and even the biologists using it only ever half-understood it. Something about the molecules it emitted enabled some form of synthetic computation, making humans who were on it about 20% smarter and more cooperative. They started bringing it to earthside labs, and science progress soared to a rate we hadn’t seen since the seventies. They brought it to government offices, and politicians started passing fewer boneheaded policies. I think that one was more about the cooperation aspect than the intelligence boost. There’d always been at least some politicians who realized which policies were dumb, and making everyone a little more cooperative let them actually push through the noise.
Geopolitical tensions eased a bit. The president challenged president Qiu of China to a gaming match (don’t ask me what game) while they were both high on it as a gesture of goodwill, and Qiu came off affable and charming and seemed happy to ease the geopolitical tensions a bit. He laughed about how it was his first time breathing Organism, winking just in case anyone thought he seriously expected them to believe he hadn’t gotten any smuggled through american export controls when half the drug dealers in Manhattan could probably get you a hookup. A few weeks later they were talking about SALT 3, this time including China.
This was where it started affecting my job. I came into the office one day and the manager called me into his office.
“Listen John”, he started in the slow voice he used for big news. “They want an analysis on the new nuclear policy. Don’t make this one too harsh, okay? This isn’t one of those reports where they want serious critical analysis. It’s one of the CYA jobs where DOD just wants an independent analysis to wave around and show a Serious Outsider approved it. Just write something short and put the downsides in the fifth section where no one will read it, okay?”
I sighed. “How bad is it?”
“New nuclear policy. As a sign of good faith, they linked up our nuclear systems in a Samson scenario. No more first use, no more aggressive use. we still have it as a deterrent, but launching it would make us nuke ourselves. It means we can retaliate if anyone’s seriously nuking us, but we’d be blowing ourselves up too so we can’t strike first. Especially with our new conventional absolute advantage and the ICBM defenses the Organism boys cooked up, we need something to reassure people they don’t need to strike us first before the new stuff is online.”
“So we switched up our entire nuclear system with a single nuke the world button?”
“Yeah, pretty much. Between us, it’s already deployed behind the scenes, we needed it to reassure the Chinese. But it’s not ready for announcement until-”
“No no no No NO NO”. I barely even realized I was shouting. “Don’t you see? People on average are more cooperative now which means state-level actors are lower-risk because their people are less volatile and more cooperative. But individuals are higher variance, because even an average 20% increase in peacefulness and cooperation leaves a large number of negative outliers, and a 20% average boost in intelligence means a lot of people way smarter than that, and these things are not correlated. Which means we have now supercharged our supply of intelligent anti-human sociopaths who might be able to access the “nuke everything now” button. It doesn’t matter how few people know, if even one of them discovered this button exists he’ll find a way to push it.”
“but-”
“This is too important. We need to shut the button NOW. Before we all blow up. Call everyone you can. China can nuke us if they have to, this is too much of a risk. We need to shut it down tomorrow. If we even make it to tomorrow. I give us even odds”.
I stormed out of the office and started running home. I had some calls I could make, maybe I could help the move faster out of this disaster.
I was still just halfway home when I saw streaks of fire flying through the sky, and the skyscrapers started crashing down around me.
This was great!
Whooa