LessWrong 2.0 Reader

If we find out that we're all the same consciousness experiencing itself subjectively, we should probably do as Jesus says. If AGI will be conscious and the hypothesis is true (we're the same consciousness experiencing itself subjectively) then maybe we could get alignment if it understands that we're it and it's us

That post is clickbait. It only argues that the incompleteness money pump doesn't work. The reasons the incompleteness money pump does work is well summarized at a high level here [LW · GW] or here [LW · GW] (more specifically here [LW(p) · GW(p)] and here [LW(p) · GW(p)], if we get into details).

I think the "become exceptional through pain" story is mostly not true. Sure, writing workshops are full of people who think they'll push through the pain and write their first book. But if you look at the actual big advances in science, technology and art, you'll usually find that the people who made these advances didn't do so through pain. Most of them were having fun. And most of the cool things I've done in life (though none of them were very exceptional) were also done in a spirit of fun.

What is the best way to take the average of three probabilities in the context below?

• There is information about a public figure
• Three people read this information and estimate the public figure's P(doom)
• (It's not actually p(doom) but it's their probability of something
• How do I then turn those three probabilities into a single one?

Thoughts.

I currently think the answer is something like for probability a,b,c then the group median is 2^((log2a + log2b + log2c)/3). This feels like a way to average the bits that each person gets from the text.

I could just take the geometric or arithmetic mean, but somehow that seems off to me. I guess I might write my intuitions for those here for correction.

Arithmetic mean (a + b + c)/3. So this feels like uncertain probabilities will dominate certain ones. eg (.0000001 + .25)/2 = approx .125 which is the same as if the first person was either significantly more confident or significantly less. It seems bad to me for the final probability to be uncorrelated with very confident probabilities if the probabilities are far apart. 

On the other hand in terms of EV calculations, perhaps you want to consider the world where some event is .25 much more than where it is .0000001. I don't know. Is the correct frame possible worlds or the information each person brings to the table?

Geometric mean (a *  b  * c)^ 1/3. I dunno, sort of seems like a midpoint. 

Okay so I then did some thinking. Ha! Whoops.

While trying to think intuitively about what the geometric mean was, I noticed that 2^((log2a + log2b + log2c)/3) = 2^ (log2 (abc) /3) = 2 ^ log 2 (abc)^1/3 = (abc) ^1/3.  So the information mean I thought seemed right is the geometric mean. I feel a bit embarrassed, but also happy to have tried to work it out. 

This still doesn't tell me whether the arithmetic worlds intuition or the geometric information interpretation is correct. 

Any correction or models appreciated. 

I think there isn't much hope in this direction. Most AI resources will probably be spent on competition between AIs, and AIs will self-modify to remove wasteful spending. It's not enough to have a weak value that favors us, if there's a stronger value that paves over us. We're teaching AI based on human behavior and with a goal of chasing money, but people chasing money often harm other people, so why would AI be nicer than that. It's all just wishful thinking.

Please consider this comment [LW(p) · GW(p)].

Please consider this comment [LW(p) · GW(p)]

This is useful part of code:

curl --header "Content-Type: application/json" --request POST --data '{"searchString": "How to improve the communication skills in Japanese?", "N": 5}' "http://[2600:1f18:17c:2d43:338d:2669:3fa5:82f8]:3000/search"

No execution of script. Returns libgen MD5 and epub CFI

Error may be due to Wifi NAT (use mobile internet) or no IPv6 support

Fwiw the thing that scares me is the combination of three things:

1. Producing an accurate estimate requires "understanding" the input
2. Our standard for the quality of estimates will be very high
3. The approach is purely algorithmic (rather than e.g. using the flexible intelligence of LLMs / humans)

You presumably agree with (1) and (3). For (2), it seems like there's a lot of correlations you need to take into account for this to produce estimates that aren't wildly off:

1. Underestimates of risk: These are cases where the presumption of independence neglects correlations between things that happen for catastrophe-causing events. For example, maybe a model is planning to defect when RSA-2048 is factored, and it has N different consistency checks it runs to make sure we aren't fooling it about whether it has been factored. The method has to see that there's a common cause for all the consistency checks to return 1, otherwise it will underestimate risk by a factor exponential in N. (And tbc I think even separately from more exotic examples like RSA-2048, it will be relatively common for this pattern to arise with networks, e.g. it happens any time that in reality there is a common cause that produces many pieces of evidence, and then the network aggregates those pieces of evidence to infer the common cause.)
2. Overestimates of risk: These are cases where the presumption of independence neglects correlations that rule out catastrophe, and so lead to overestimates. For example, the model might be willing to make up fake instructions for creating bioweapons to help with scifi writing, but never provide real instructions. So the method needs to capture the "understanding" of which instructions are fake vs real.

I agree this isn't a proof of impossibility, since a purely algorithmic approach (SGD) produced the "understanding" in the first place, so in theory a purely algorithmic approach could still capture all that understanding to produce accurate estimates. But it does seem heuristically like you should assign a fairly low probability that this pans out.

Please gpg sign the code and distribute it yourself if you trust it. It is less than 50 lines.