LessWrong 2.0 Reader

I'm not happy about this but it seems basically priced in, so not much update on p(doom).

We will soon have Bayesian updates to make. If we observe that incentives created during end-to-end RL naturally produce goal guarding and other dangerous cognitive properties, it will be bad news. If we observe this doesn't happen, it will be good news (although not very good news because web research seems like it doesn't require the full range of agency).

Likewise, if we observe models' monitorability and interpretability start to tank as they think in neuralese, it will be bad news. If monitoring and interpretability are unaffected, good news.

Interesting times.

Yeah, just went through this whole same line of evasion [LW(p) · GW(p)]. Alright, the Collatz conjecture will never be "proved" in this restrictive sense—and neither will the Steve conjecture or the irrationality of √2—do we care? It may still be proved according to the ordinary meaning.

How much of their original capital did the French nobility retain at the end of the French revolution?

How much capital (value of territorial extent) do chimpanzees retain now as compared to 20k years ago?

I haven't thought deeply about this specific case, but I think you should consider this like any other ablation study--like, what happens if you replace the SAE with a linear probe?

Eh, I think unclear rules and high standards are fine for some purposes. Take a fiction magazine. Good ones have a high standard for what they publish, and (apart from some formatting and wordcount rules) the main rule is it has to fit the editor's taste. The same is true for scientific publications.

I understand the motivation behind this, but there is little warning that this is how the forum works.

I mildly disagree with this. The New Users Guide says 

LessWrong is a pretty particular place. We strive to maintain a culture that's uncommon for web forums^[1] and to stay true to our values. Recently, many more people have been finding their way here, so I (lead admin and moderator) put together this intro to what we're about.

My hope is that if LessWrong resonates with your values and interests, this guide will help you become a valued member of community. And if LessWrong isn't the place for you, this guide will help you have a good "visit" or simply seek other pastures.

On the margin, is there room for improvement? Seems likely, but doesn't seem bad. If I was in charge I'd be tempted to open the New Users Guide with like, four bullet points that said 'This place is for aspiring rationalists, don't say false things, don't be a jerk, for examples of what we mean by that read on.' That's somewhat stylistic though.

There is no warning that trying to contribute in good faith isn't sufficient

Wait, now I'm confused. Most forums I'm aware of don't have much of a Good Faith defense. I looked up the rules for the first one I thought of, Giant In The Playground, and while it's leaning a bit more Comprehensive and Clear I don't see a place where it says if you break a rule in good faith you're fine. 

In general, someone trying to contribute to a thing who but doing so badly doesn't get that much of a pass? Like, I've been politely ejected from a singing group before because I was badly off-key. Nobody doubted I was trying to sing well! It doesn't change the fact that the group wanted to have everyone singing the right notes. 

I suggest that instead of making rate-limited users (who used up their rate) unable to comment at all, their additional comments should be invisible, but still visible to other rate-limited users (and users who choose to see them).

Meh. The internet is big. If the kind of thing that got someone rate-limited on LessWrong got them rate-limited or banned everywhere else, I'd be supportive of having somewhere they were allowed to post. Reddit's right over there, you know?

I think giving special emphasis to rate-limited users for rate-limited users is straightforwardly a bad idea. If someone got rate-limited, in general I assume it's because they were writing in ways the mods and/or other users thought they shouldn't do. If someone is going to stick around, I want their attention on people doing well, not doing badly. Imagine a basketball practice; if I'm a lousy shot, the coach might tell me to sit out the drill and watch a couple of the good players for few minutes. If I'm really bad, I get cut from the team. No coach is going to say, "hey, you're a lousy shot, so pay special attention to these other players who are just as bad as you."

A big component of this is I tend to think of LessWrong as a place I go to get better at a kind of mental skill, hence analogies to choir or basketball practice. You may have other goals here.

I think my quick guess is that what's going on is something like:
- People generally have a ton of content to potentially consume and limited time, and are thus really picky. 
- Researchers often have unique models and a lot of specific nuanced they care about.
- Most research of this type is really bad. Tons of people on Twitter now seem to have some big-picture theory of what AI will do to civilization.
- Researchers also have the curse of knowledge, and think their work is simpler than it is.

So basically, people aren't flinching because of bizarre and specific epistemic limitations. It's more like,
> "this seems complicated, learning it would take effort, my prior is that this is fairly useless anyway, so I'll be very quick to dismiss this."

My quick impression is that this is a brutal and highly significant limitation of this kind of research. It's just incredibly expensive for others to read and evaluate, so it's very common for it to get ignored. (Learned in part from myself trying to put a lot of similar work out there, then seeing it get ignored)

Related to this -
I'd predict that if you improved the arguments by 50%, it would lead to little extra uptake. But if you got someone really prestigious to highly recommend it, then suddenly a bunch of people would be much more convinced. 

And all of this is livestreamed on Twitch

Also, each agent has a bank account which they can receive donations/transfers into (I think Twitch makes this easy?) and from which they can e.g. send donations to GiveDirectly if that's what they want to do.

One minor implementation wrinkle for anyone implementing this is that "move money from a bank account to a recipient by using text fields found on the web" usually involves writing your payment info into said text fields in a way that would be visible when streaming your screen. I'm not sure if any of the popular agent frameworks have good tooling around only including specific sensitive information in the context while it is directly relevant to the model task and providing hook points on when specific pieces of information enter and leave the context - I don't see any such thing in e.g. the Aider docs - and I think that without such tooling, using payment info in a way that won't immediately be stolen by stream viewers would be a bit challenging.

Yes, the technique of formal proofs, in effect, involves translation of high-level proofs into arithmetic.

So self-reference is fully present (that's why we have Gödel's results and other similar results).

What this implies, in particular, is that one can reduce a "real proof" to the arithmetic; this would be ugly, and one should not do it in one's informal mathematical practice; but your post is not talking about pragmatics, you are referencing "fundamental limit of self-reference".

And, certainly, there are some interesting fundamental limits of self-reference (that's why we have algorithmically undecidable problems and such). But this is different from issues of pragmatic math techniques.

What high-level abstraction buys us is a lot of structure and intuition. The constraints related to staying within arithmetic are pragmatic, and not fundamental (without high-level abstractions one loses some very powerful ways to structure things and to guide our intuition, and things stop being comprehensible to a human mind).

Your post purports to conclude: “That's why [the Collatz conjecture] will never be solved”.

Do you think it would also be correct to say: “That's why [the Steve conjecture] will never be solved”?

If yes, then I think you’re using the word “solved” in an extremely strange and misleading way.

If no, then you evidently messed up, because your argument does not rely on any property of the Collatz conjecture that is not equally true of the Steve conjecture.

make fewer points, selected carefully to be bulletproof, understandable to non-experts, and important to the overall thesis

That conflicts with eg:

If you replied with this, I would have said something like "then what's wrong with the designs for diamond mechanosynthesis tooltips, which don't resemble enzymes

Anyway, I already answered that in 9. diamond.