LessWrong 2.0 Reader

Original source, to my knowledge. (July 1st, 2014)

"So long, Linda! I'm going to America!"

What is the proposed research path and its theory of impact? It’s not clear from reading your note / generally seems too abstract to really offer any feedback

Regarding genocide and factory farms, my point was just that abusing others for your self-benefit is an adaptive behavior. That's all. Nothing deeper than that.

By the way, I appreciate you trying to answer the crux of my question to the extent that makes sense. This is exactly the kind of thinking I was hoping to provoke.

Proof of Collusion: How Perfect Trust Enables Perfect Betrayal

Alternate Claude titles I liked: 

 Superintelligent Handshakes: How Cryptographic Advances Could Undermine AI Safety

"Superrational Serpents: The Paradox of AI Systems That Cooperate Too Well"

[This document is a smattering of ideas around the topic that advanced cryptography and proof verification could enable yet-unseen forms of cooperations between Artificial Intelligences. Currently, the arguments aren’t watertight and completely thought through and I’m not confident any of this won’t non-stupid. I don’t intend to think much in this direction anymore so to prevent these ideas from languishing in my drafts I am posting as-is.

 Thanks to Nisan Stiennon for comments. ]

tl;dr: Combining Löbian cooperation with interactive proofs, cryptographic magika (PCP, ZK, homomorphic encryptions...) and formal verification of neural networks may allow future AIs to engage in radically wide forms of cooperations. This might be very bad.

Löbian Cooperation

In Open-Source game theory agents that can inspect each source-code can achieve cooperation even in scenarios like the Prisoner's Dilemma where traditional game theory recommends (mutual) defection. The key insight is that when agents have access to each other's source code they may be able to leverage Löb's theorem to engage in superrational cooperative strategies ['Löbian cooperation'].

Unlike Twin cooperation Löbian cooperation is quite expansive: the two agents don't need to share any architectural similarities or alignment in goals for them to be able to superrationally cooperate. 

In a future where Löbian cooperation may be robustly implemented by AI's they could engage in what we might call Radically Wide Cooperation. 

Collusion and the dangers of Radically Wide Cooperation

A popular proposal for scalable oversight suggest to use 'safety-by-debate' where we rely on AIs to debate and point out flaws in each other's reasoning as a safety mechanism. If the debating AIs can engage in cryptographically secure cooperation, they could collude against human interests even if they are trained to have quite different goals. 

The optimistic argument for debate protocols assumes the AIs have been trained to have sufficiently different objectives such that they won't collude. But with radically-wide cooperation protocols, they could verify each other's commitment to a collusion strategy that maximizes their joint utility, while maintaining plausible appearance of genuine debate.

How Plausible is Löbian Cooperation?

Löbian cooperation is quite expansive: the two agents don't need to share any architectural similarities or alignment in goals for them to be able to superrationally cooperate. The only two requirement are (1) a legible trustworthy sourcecode and (2) access to each other's source code. 

First of all, an agent that does not have a legible trustworthy source may want to self-modify to become legibly trustworthy. 

It is sometimes objected that Löbian cooperation is an artifact of logic and doesn't survive in situations where reasoning is more muddy, probabilistic, and error prone. Critch's Parametric Löb Theorem [link above] shows this is not the case. Robust Löbian cooperation is possible even in this case.

The fundamental principle at work is 'legible trustworthiness' - agents can cooperate because they can reliably verify that their counterparts will do the same.

Cryptographic Magicka

Probabilistically Checkable Proofs (PCPs) allow verification of complex proofs by only checking a few random bits. Any mathematical proof can be transformed into a format where checking a constant number of bits gives high confidence in the proof's validity.

Zero-Knowledge Proofs (ZKP) allow one party to prove they know something without revealing the actual information. A prover can convince a verifier that a statement is true while revealing nothing beyond the truth of the statement.

Homomorphic encryption allows computation directly on encrypted data. A party can perform operations on encrypted inputs and get encrypted outputs, without ever seeing the actual data. The results, when decrypted, match what would have been obtained by performing the operations on the unencrypted data.

Formal Verification of Neural Networks is currently underdevelopped. Current AI systems are largely black boxes. However, research in mechanistic interpretability and formal verification of neural networks is progressing rapidly and it is plausible that future AIs may be able to reliably verify properties of their own source-codes / weights. 

Collusion

A popular proposal for scalable oversight suggest to use 'safety-by-debate' where we rely on AIs to debate and point out flaws in each other's reasoning as a safety mechanism. If the debating AIs can engage in cryptographically secure cooperation, they could collude against human interests even if they are trained to have quite different goals. 

The optimistic argument for debate protocols assumes the AIs have been trained to have sufficiently different objectives such that they won't collude. But with radically-wide cooperation protocols, they could verify each other's commitment to a collusion strategy that maximizes their joint utility, while maintaining plausible appearance of genuine debate.

Zero-Knowledge, Probabilistically Checkable Protocols for Super-Rational Radically Wide Cooperation

In a future where these technical barriers have been overcome, AIs could engage in what we might call Radically Wide Cooperation. 

First, the AIs negotiate the terms of cooperation. Then comes the critical step: each AI runs a self-verification procedure, analyzing its own neural networks and decision processes to prove it will uphold its commitments under the specified conditions. This verification step generates a formal proof, likely massive in size due to the complexity of its mind. 

The AIs then make these proofs practically usable through cryptographic transformations. Using the PCP theorem, they convert their large proofs into compact, probabilistically checkable formats. Since direct access to an AI's decision processes could pose a security and bargaining risk, the protocol may include additional cryptographic safeguards. This can be done in two ways:

1. Zero-knowledge protocols allow the AI to prove properties about its decision-making without revealing anything beyond those specific properties
2. Homomorphic encryption schemes would allow other AIs to compute on encrypted versions of the proofs, verifying properties while never seeing the actual proof content

The result is a cryptographically secure cooperation protocol where AIs can verify each other's commitments without exposing their minds to direct inspection. This enables potentially unprecedented levels of credible commitment.

Nisan: All of this allows an agent A1 to prove that it possesses an agent A2 with certain properties. But cooperation also requires a proof that A1 == A2. In other words, A1 could create a legible, cooperative A2, produce proofs about A2, and then forget about A2 and do noncooperative stuff.

I think you could prove A1 == A2 with a combination of secure enclaves, secure boot, audits of chip fabs, PCP for chip designs, etc.

AGO:sorry I am a little lost, what do you mean here exactly by A1 possseing an agent A2 with certain properties?

Nisan: Suppose I'm an AI. I write a program called DUPOC that cooperates iff you cooperate. I show you DUPOC and say "this is my source code. You should cooperate with me." You cooperate, and I defect.

Homomorphic Self-modification towards Legible Trustworthiness

First of all, an agent that does not have a legible trustworthy source may want to self-modify to become legibly trustworthy. This is particularly relevant in the context of cooperation: AIs might strategically choose to make their source code more verifiable to enable cooperation with other agents. 

AIs might introduce computationally intractable backdoors into their own source code during self-modification. These backdoors could be designed to be effectively impossible to verify, even with significant computational resources, making it difficult for AIs to truly trust each other's commitments. An AI could not be certain whether another AI has hidden cooperation-breaking behaviors that would only activate under specific conditions, even if that AI appears to have a legibly trustworthy source code.

One potential solution to this problem is homomorphic retraining - a protocol where one AI allows another AI to modify and train it without revealing sensitive information about its architecture or decision processes. In this approach, only a general training procedure is specified by the second AI, while the detailed implementation remains private. This could theoretically allow the second AI to ensure that cryptographic backdoors are removed from the first AI without requiring the first AI to expose potential security vulnerabilities. The key advantage is that it provides a way to verify trustworthiness while maintaining security - the verifying AI can confirm the absence of backdoors without gaining enough information to exploit any weaknesses in the first AI's architecture.

i think it's quite valuable to go through your key beliefs and work through what the implications would be if they were false. this has several benefits:

• picturing a possible world where your key belief is wrong makes it feel more tangible and so you become more emotionally prepared to accept it.
• if you ever do find out that the belief is wrong, you don't flinch away as strongly because it doesn't feel like you will be completely epistemically lost the moment you remove the Key Belief
• you will have more productive conversations with people who disagree with you on the Key Belief
• you might discover strategies that are robustly good whether or not the Key Belief is true
• you will become better at designing experiments to test whether the Key Belief is true

Dang, I missed this. Here's my audition for 500 Million though, I guess for next year

https://m.youtube.com/watch?v=ljmifo4Klss

In this case, the starving person presumably has to press the button or else starve to death, and thus has no bargaining power. The other person only has to offer the bare minimum beyond what the starving person needs to survive, and the starving person must take the deal. In Econ 101 (assuming away monopolies, information asymmetry, etc.), exploited workers do have bargaining power by being able to work for other companies, hence why companies can’t just do stupid, spiteful actions in the long term.

If you already know that an adverse event is highly likely for your specific circumstances, then it is likely that the insurer will refuse to pay out for not disclosing "material information" - a breach of contract.

Loss leaders are common, but rely on a reasonably accurate prediction of what that loss will be. and simple errors are of course common.

I think "nobody knows what they're doing" is an overstatement. Insurance companies make significant profit, and very rarely fail. The usual complaint I see from investors is that some years an insurer has made less profit than others - which is exactly what one should expect, so clearly these investors don't understand what they've invested in.

Your bike example sounds like bad luck. An event with a probability of 1% per year can still happen three years running.

It would be interesting to consider the inverse Kelly - given an insurance premium X, what are the reasonable bounds of assumed probability of an adverse event costing Y?

I'm not sure how the wealth of the underwriter could be estimated though.

there are two different modes of learning i've noticed.

• top down: first you learn to use something very complex and abstract. over time, you run into weird cases where things don't behave how you'd expect, or you feel like you're not able to apply the abstraction to new situations as well as you'd like. so you crack open the box and look at the innards and see a bunch of gears and smaller simpler boxes, and it suddenly becomes clear to you why some of those weird behaviors happened - clearly it was box X interacting with gear Y! satisfied, you use your newfound knowledge to build something even more impressive than you could before. eventually, the cycle repeats, and you crack open the smaller boxes to find even smaller boxes, etc.
• bottom up: you learn about the 7 Fundamental Atoms of Thingism. you construct the simplest non-atomic thing, and then the second simplest non atomic thing. after many painstaking steps of work, you finally construct something that might be useful. then you repeat the process anew for every other thing you might ever find useful. and then you actually use those things to do something

generally, i'm a big fan of top down learning, because everything you do comes with a source of motivation for why you want to do the thing; bottom up learning often doesn't give you enough motivation to care about the atoms. but also, bottom up learning gives you a much more complete understanding.

I agree with your view about organizational problems. Your discussion gave me an idea: Is it possible to shift employees dedicated to capability improvement to work on safety improvement? Set safety goals for these employees within the organization. This way, they will have a new direction and won't be idle, worried about being fired or resigning to go to other companies.

That seems to solve problem #4. Employees quitting becomes much less of an issue, since in any case they would only be able to share knowledge about safety (which is a good thing).

Do you think this plan will be able to solve problems #1, #2, #3 and #5? I think such discussions are very important, because many people (me included) worry much more about organizational side of alignment than about technical side.