LessWrong 2.0 Reader

Admittedly, I can't judge all the technical details. But I notice that neither So8res nor Wentworth have engaged with the EJT post (neither directly in the comments nor in their posts you have linked), despite being published later. And EJT's engagement [LW(p) · GW(p)] with the Wentworth post didn't elicit much of a reaction either. So from an outside view, the viability of coherence arguments seems questionable.

As someone who agrees with Stalin in, well, most of the questions (with a few exceptions, some of them are quite important, but it's not that important to explain there), what could I say?

1. There are newer papers on IQ and race, and a lot of critique of that one.

2. Ability to solve IQ tests can differ in a single person, for like 10+ points. https://www.reuters.com/article/business/healthcare-pharmaceuticals/study-finds-poverty-reduces-brain-power-idUSL6N0GU1PO/

3. My dad have scored around 170, and considers IQ tests pointless.

4. I agree with your point about the male-looking face.

5. It's pointless to pretend that LGBT+ is not an organisation with ideology. By the way, their ideology is too backwards and reactionary in relation to what do WE plan, regarding the sex and family.

6. Yes, setting a question of racial IQ dependency is bad. Human intelligence is dependent on its development around their whole life and usage. Making such questions, you begin to act to create proofs for them (self-fulfilling prediction, aha), and thus go against the basic principles of democracy.

7. Yes, "high IQ jews" goes into the same dumpster.

Relevant: When pooling forecasts, use the geometric mean of odds [? · GW].

That post is clickbait. It only argues that the incompleteness money pump doesn't work. The reasons the incompleteness money pump does work is well summarized at a high level here [LW · GW] or here [LW · GW] (more specifically here [LW(p) · GW(p)] and here [LW(p) · GW(p)], if we get into details).

What is the best way to take the average of three probabilities in the context below?

• There is information about a public figure
• Three people read this information and estimate the public figure's P(doom)
• (It's not actually p(doom) but it's their probability of something
• How do I then turn those three probabilities into a single one?

Thoughts.

I currently think the answer is something like for probability a,b,c then the group median is 2^((log2a + log2b + log2c)/3). This feels like a way to average the bits that each person gets from the text.

I could just take the geometric or arithmetic mean, but somehow that seems off to me. I guess I might write my intuitions for those here for correction.

Arithmetic mean (a + b + c)/3. So this feels like uncertain probabilities will dominate certain ones. eg (.0000001 + .25)/2 = approx .125 which is the same as if the first person was either significantly more confident or significantly less. It seems bad to me for the final probability to be uncorrelated with very confident probabilities if the probabilities are far apart. 

On the other hand in terms of EV calculations, perhaps you want to consider the world where some event is .25 much more than where it is .0000001. I don't know. Is the correct frame possible worlds or the information each person brings to the table?

Geometric mean (a *  b  * c)^ 1/3. I dunno, sort of seems like a midpoint. 

Okay so I then did some thinking. Ha! Whoops.

While trying to think intuitively about what the geometric mean was, I noticed that 2^((log2a + log2b + log2c)/3) = 2^ (log2 (abc) /3) = 2 ^ log 2 (abc)^1/3 = (abc) ^1/3.  So the information mean I thought seemed right is the geometric mean. I feel a bit embarrassed, but also happy to have tried to work it out. 

This still doesn't tell me whether the arithmetic worlds intuition or the geometric information interpretation is correct. 

Any correction or models appreciated. 

I think there isn't much hope in this direction. Most AI resources will probably be spent on competition between AIs, and AIs will self-modify to remove wasteful spending. It's not enough to have a weak value that favors us, if there's a stronger value that paves over us. We're teaching AI based on human behavior and with a goal of chasing money, but people chasing money often harm other people, so why would AI be nicer than that. It's all just wishful thinking.

Please consider this comment [LW(p) · GW(p)].

Please consider this comment [LW(p) · GW(p)]

This is useful part of code:

curl --header "Content-Type: application/json" --request POST --data '{"searchString": "How to improve the communication skills in Japanese?", "N": 5}' "http://[2600:1f18:17c:2d43:338d:2669:3fa5:82f8]:3000/search"

No execution of script. Returns libgen MD5 and epub CFI

Error may be due to Wifi NAT (use mobile internet) or no IPv6 support

Fwiw the thing that scares me is the combination of three things:

1. Producing an accurate estimate requires "understanding" the input
2. Our standard for the quality of estimates will be very high
3. The approach is purely algorithmic (rather than e.g. using the flexible intelligence of LLMs / humans)

You presumably agree with (1) and (3). For (2), it seems like there's a lot of correlations you need to take into account for this to produce estimates that aren't wildly off:

1. Underestimates of risk: These are cases where the presumption of independence neglects correlations between things that happen for catastrophe-causing events. For example, maybe a model is planning to defect when RSA-2048 is factored, and it has N different consistency checks it runs to make sure we aren't fooling it about whether it has been factored. The method has to see that there's a common cause for all the consistency checks to return 1, otherwise it will underestimate risk by a factor exponential in N. (And tbc I think even separately from more exotic examples like RSA-2048, it will be relatively common for this pattern to arise with networks, e.g. it happens any time that in reality there is a common cause that produces many pieces of evidence, and then the network aggregates those pieces of evidence to infer the common cause.)
2. Overestimates of risk: These are cases where the presumption of independence neglects correlations that rule out catastrophe, and so lead to overestimates. For example, the model might be willing to make up fake instructions for creating bioweapons to help with scifi writing, but never provide real instructions. So the method needs to capture the "understanding" of which instructions are fake vs real.

I agree this isn't a proof of impossibility, since a purely algorithmic approach (SGD) produced the "understanding" in the first place, so in theory a purely algorithmic approach could still capture all that understanding to produce accurate estimates. But it does seem heuristically like you should assign a fairly low probability that this pans out.