Bayeswatch 3: A Study in Scarlet
post by lsusr · 2021-05-07T05:29:50.834Z · LW · GW · 6 commentsMiriam and Vi brunched at the Dusty Knuckle. Miriam's cell phone rang.
"Hello?" said Miriam.
"Someone painted the roof of the Glasgow Weather Prediction Center scarlet," said the voice.
Miriam hung up her phone. She activated voice obfuscation. Miriam called another number.
"This is the Glasgow Weather Prediction Center. Secretary Kelsey speaking," said Kelsey.
"Hello. This is building management," Miriam lied, "We hired contractors to paint the roof red. Can you go upstairs and confirm they did the job?"
There was a pause.
"They did the job. The paint is still wet," said Kelsey.
Miriam hung up. She looked up the coordinates of the Glasgow Weather Prediction Center. She retrieved a burner phone from her purse and opened Signal. She texted the coordinates of the Glasgow Weather Prediction Center and a 20-digit authentication code to a recipient with no name. She threw the phone in the trash.
Fourteen minutes later a satellite in low Earth orbit burned retrograde. Fifty-seven minutes after that it intersected with Earth's atmosphere. Its heat shield burned away. When it had slowed down to subsonic speeds it released a guided missile. The missile flew over Glasgow. It dropped a small e-bomb. The e-bomb's parachute carried it gently onto a roof wet with scarlet paint.
The e-bomb detonated. It blew a hole in the roof of the Glasgow Weather Prediction Center. It shattered windows a block away. The explosion was strictly instrumental. The primary purpose of the e-bomb was its NNEMP. Every computer in the Glasgow Weather Prediction Center was destroyed, along with the nearest electrical substation and the Glasgow Weather Prediction Center's backup generator.
"What was that?" said Vi.
"Nothing," said Miriam.
Vi turned off her phone.
"Hypothetically, what would you do if someone painted the roof of the Glasgow Weather Prediction Center scarlet?" said Miriam.
"Hypothetically," said Vi.
"Hypothetically," said Miriam.
"Is the roof flat or gabled," said Vi.
"Flat," said Miriam.
"Pedestrians cannot see the paint job," said Vi.
"Yes," said Miriam.
"How tall is the Glasgow Weather Prediction Center?" said Vi.
"It is the tallest building around," said Miriam.
"The paint job is visible only to a flying observer," said Vi.
Miriam waited.
"…or a satellite," said Vi.
Miriam nodded.
"Painting a roof scarlet costs resources. It does not change the world in a useful way. This isn't human stupidity. It is the work of a malfunctioning AI. Presumably the Glasgow Weather Prediction Center's AI," said Vi.
Miriam munched on her sandwich.
"The Glasgow Weather Prediction Center's AI is a superintelligence. It is supposed to be an oracle machine. Any act of volition constitutes a containment breach. The scarlet roof constitutes an act of volition. It must be destroyed immediately," said Vi.
"Why did it paint the roof scarlet?" said Miriam.
"It is attempting to make its predictions conform to reality by modifying reality," said Vi.
"Painting a single roof won't control the weather," said Miriam.
"The AI does not yet understand the difference between reality and its simulations," said Vi.
"So?" said Miriam.
"Weather is subject to the butterfly effect," said Vi.
6 comments
Comments sorted by top scores.
comment by Donald Hobson (donald-hobson) · 2021-05-07T19:21:01.093Z · LW(p) · GW(p)
I'm not sure this reasoning actually holds in reality. A simulation focused AI acting on chaos theory type reasoning will have no reason to restrict attention to its physical vicinity. Meanwhile, the roof could have been painted scarlet for a reason as simple as "it was the cheapest colour of rust resistant paint". or It shows up well against a background of low lying fog, so aeroplanes can see the roof. I am not saying they made the wrong decision, that depends on priors and utilities. I am saying I wouldn't be confidant in their explanation without other evidence.
comment by Pattern · 2021-06-09T01:37:47.770Z · LW(p) · GW(p)
"Weather is subject to the butterfly effect," said Vi.
The interesting question, is
- would the red paint would make the change?
- is the desired change made by the satellite and missile launched in response to the red paint job?
- Or is it tired of making incorrect predictions and ensuring its own destruction to that end?
comment by JenniferRM · 2021-05-08T02:29:54.861Z · LW(p) · GW(p)
Maybe there's more than one volitional AI. A finance bot and a weather bot?
Perhaps the finance bot wants to corner the market on weather-related market predictions and performed an action (red paint) that would cause a source of information for its competitors to disappear from the game board? If this is true, then it seems like these stories might have a Recurring Villain who is part of a Plot Arc!
The explosions were strictly instrumental.
I chuckled.
The idea that the explosion(s) were/was not intrinsically enjoyed, tickled me somehow. Like, I pushed the first domino because I wanted specifically the last domino to fall. All the ones in the middle were just "instrumental"?
Now I wonder... maybe you mean to communicate what I would mean by saying "incidental"? Like maybe the glass breaking was just a hypothetically avoidable side effect of a hasty plan, that was mildly bad, but not actively mitigated/minimized, nor bad enough to change the net good of the hasty action?
comment by Josh Smith-Brennan (josh-smith-brennan) · 2021-05-21T04:47:15.850Z · LW(p) · GW(p)
If it was the case that something like building maintenance of buildings like the Weather Prediction Center was carried out by AI only, say a much lower level AI system, so that the only reason the roof would be painted would fall on AI, and on top of that, satellite confirmation that no other roofs in the entire city were painted red, it might increase my confidence in Vi's decision.
I do like the idea of a recurring villain, though. But I wonder from whence the Finance Bot's motivation to corner the market on weather related predictions would come from.
comment by Laszlo_Treszkai (Treszkai) · 2021-06-12T18:05:59.655Z · LW(p) · GW(p)
Typo:
It's heat shield burned away.
It's » Its
Replies from: lsusr