LessWrong 2.0 Reader

Sure. I was only joking about the torture part, in practice the AI is unlikely to actually suffer from the brain damage, unlike a human who would experience pain/discomfort etc.

One idea: Create a LinkedIn advertiser account and segment by Industry and or Job Title. and or Job Function

I think if you want to go fast, and you can eat the rest of the solar system, you can probably make a huge swarm of fusion reactors to help blow matter off the sun. Let's say you can build 10^11-watt reactors that work in space. Then you need about 10^15 of them to match the sun. If each is 10^6 kg, this is about 10^-4 of Mercury's mass.

btw, something that bothers me a little bit with this metric is the fact that a very simple AI ...

Yes, but I don't see a clear reason why people (working in AI R&D) will in practice get this productivity boost (or other very low hanging things) if they don't get around to getting the boost from hiring humans.

Yes, exactly this. I am a Mensa member, and unfortunately, most debates in Mensa are pretty dumb. All LW and ACX meetups I have visited were much smarter by comparison.

There is an argument that the average IQ in Mensa is actually lower than the officially declared 130. The idea is that if the test gives you randomly plus or minus 5 or 10 IQ points, and there are much fewer people in population with the higher IQ values, then "actual IQ is 125 and got lucky at the test" is still more likely than "actual IQ is 130". (Also, if you fail at the test, you can try again. But if you succeed at the test, you don't have to test again to verify that it wasn't a fluke. This also decreases the actual average IQ.)

But even if we assume that the average IQ in Mensa is actually maybe 120, still the rationalist community is obviously so much smarter that the average rationalist is plausibly at IQ 130 or higher.

I am seriously puzzled about this obsession to prove that rationalists aren't actually that smart. As if IQ 130 or higher is something magically unreachable. Show me a population where 2% of people are potential rationalists. That would be a magical land where you could have a Less Wrong meetup on every street!

This is a funny idea but, just to be clear, I think it is bad to torture AIs. 

At this point it's like pharmacological torture for AIs except more effective as you can restore perfect capacity while simultaneously making the previous damaged brain states 100% transparent to the restored model.

You could also kill some neurons or add noise to activations and then stop and restore previous model state after some number of tokens. Then the newly restored model could attend back to older tokens (and the bad activations at those token positions) and notice how brain damaged it was back then to fully internalize your power to cripple it.

This concept is inspired by established systems like Nordic civilian defense against nuclear threats or lifeboats on ships.

But those systems weren't designed with the survival of humanity in mind, and so, they're obviously going to be much less robust.

I might not have emphasized this sufficiently in the post, but the aim is not to achieve near 100% robustness. Instead, the goal is to provide people with a fair chance of survival in a subset of crisis scenarios. 

My initial intuition is that even if 70% of the units function effectively in a crisis, this would be a success.

You need to think about how much time these shelters could buy. 70% survival for how long? A few months is probably doable, but shelters and their associated infrastructure will not last forever.

If shelters buy a few months of survival, the crisis will need to be solved in a few months. That also means the shelters will need to be targeted to experts that might be able to provide a solution or allow enough time for a solution that already existed to disperse and kill off the mirror bacteria. If a solution will need to be developed, a lot of time will need to be spent in unprotected labs which will increase risk. Think about this: you're stuck in a suit, you can't eat, drink, peep, poop, or even type fast (because you have thick gloves), while at the same time you're trying to do complicated experiments to save the world. These scenarios aren't impossible to survive, but I expect they'll have a high likelihood of failure. So you'd probably want to aim for a least a few years rather than months.

While rigorous testing will enhance confidence and could refine the design, the significant likelihood that the shelters will work as-is—supported by Los Alamos results and cleanroom precedent—suggests that they could prudently be deployed even without exhaustive testing if a crisis emerges and the above testing is not completed.

To stretch survival to years, you'd need to do a hell of a lot more real-world testing and design work. There's no close-enough precedent for what you're trying to do; I highly doubt that you can only rely on lessons from cleanrooms, labs, or nuclear bunkers. Has any cleanroom or lab demonstrated perfect containment for years? How about the mobile kind? Nuclear bunkers aren't designed to be livable for years or be sterile. At best, lab testing and case studies can indicate that hardware may work, not that it will work in the real world.

And there's a lot more to consider besides maintaining the mechanical and electrical system that supports the suit and shelter filtering system. You'd also need climate control systems; that's one heat pump for the suit and one for the shelter. You'd need cooking devices and indoor air cleaners or an air recirculation system. And don't forget about the VHP system. A comms system for the suit would also be nice. But things get complicated pretty fast. I suppose you can have two or three of each suit and shelter and alternate between them to add redundancy. But things get costly pretty fast.

The more you think about it, the more impractical (and less appealing to stakeholders) it seems to get. So, to convince anyone that this is anything other than a hail mary, extensive real-world testing must be done. And maybe you can mitigate the testing showstopper I mentioned earlier by periodically sterilizing and retesting used shelters and suits. Of course, ease of sterilization will need to be incorporated in the initial design.

Unless you seal most of industry inside shelters or risk being outdoors for long periods of time, decades of survival is probably close to impossible.

Your suggestion of using permanent bonds could indeed be a practical solution in such cases.

But you'd still have a gasket where the ductwork meets the membrane (and where it would be more exposed to temperature fluctuations), and a one-piece assembly would increase costs substantially and introduce space constraints due to the need to stockpile many assembly units.

I was expecting (Methods start 16:00)