LessWrong 2.0 Reader

Neither of them is exactly what you're looking for, but you might be interested in lojban, which aims to be syntactically unambiguous, and Ithkuil, which aims to be extremely information-dense as well as to reduce ambiguity. With regards to logical languages (ones which, like lojban, aim for each statement to have a single possible interpretation), I also found Toaq and Eberban just now while looking up lojban, though these have fewer speakers.

I think you're assuming these minds are more similar to human minds than they necessarily are. 

I don't assume similarity to human minds so much as you assume universal dissimilarity.
 

its not something were even aiming for with current alignment attempts.

Indeed

oh not at all, I think I'm failing on both fronts

I see. Well, in case one of them reads this: to the extent it might matter to you, as a fellow radical vegan I request you not kill anyone trying to figure out how to make the first takeover-capable AI be benevolent.

(I'm aware that humans are such that it's not obvious that designer-chosen-alignment → benevolence, and that one does actually have to reason about that instead of just assuming the hopeful option, but after having reasoned about it, I think most would choose benevolence in the end.^[1] At least, I do not expect the important exceptions to be alignment researchers, as opposed to selected-for power-seekers in high positions.)

1. ^{^}

   It seems to me that actually reflectively endorsing evil (objectively describing what's happening and then being like "this is good", instead of euphemizing and coping with arguments that often secretly amount to "if you're right then the world would be really bad, and that's scary") is rare, and most people are actually altruistic on some level but conformism overrides that and animal abuse is so normal that they usually don't notice it, but then when they see basic information like "cows have best friends and get stressed when they are separated" they seemingly get empathetic and reflective (check the comments).

I disagree but not confident I could write an explanation that's both legible and not just capturing the surface level "oppressed people more likely to want to oppose oppression". When I saw the question I was looking forwards to you writing a good answer to it, actually. To hint at the starting point, why is queer anarchism a thing?

(Also, even without politicization/oppression, I think there's something to be said about: what kind of mind derives who they are from first principles instead of inferring 'who they are' from (bio)social norms about bodies. Though only a subset of trans people are well-described that way, still a higher portion than in cis people I think)

i agree but think its solvable and so human content will be duper valuable. these are my additional assumptions

3. for lots of kinds of content (photos/stories/experiences/adr), people'll want it to be a living being on the other end

4. insofar as that's true^, there will be high demand for ways to verify humanness, and it's not impossible to do so (eg worldcoin)

This is very interesting! One thing I'm confused about—I agree that the monitor would have a hard time getting the bits, based on these figures. But the introspective models also don't seem to get far above 50% accuracy, which means that right now they'd have a very hard time error correcting enough that their comrade would understand them but not the overseer. If the claim is that introspection-tuned models will get much better at self prediction, then its not clear to me that other models wont also get much better at cross-prediction?

I was having trouble reproducing your results on Pythia, and was only able to get 60% variance explained. I may have tracked it down: I think you may be computing FVU incorrectly. 

https://gist.github.com/Stefan-Heimersheim/ff1d3b92add92a29602b411b9cd76cec#file-clustering_pythia-py-L309

I think FVU is correctly computed by subtracting the mean from each dimension when computing the denominator. See the SAEBench impl here:

https://github.com/adamkarvonen/SAEBench/blob/5204b4822c66a838d9c9221640308e7c23eda00a/sae_bench/evals/core/main.py#L566

When I used your FVU implementation, I got 72% variance explained; this is still less than you, but much closer, so I think this might be causing the improvement over the SAEBench numbers.

In general I think SAEs with low k should be at least as good as k means clustering, and if it's not I'm a little bit suspicious (when I tried this first on GPT-2 it seemed that a TopK SAE trained with k = 4 did about as well as k means clustering with the nonlinear argmax encoder).

Here's my clustering code: https://github.com/JoshEngels/CheckClustering/blob/main/clustering.py
 

There also used to be a page for Preparedness: https://web.archive.org/web/20240603125126/https://openai.com/preparedness/. Now it redirects to the safety page above.

(Same for Superalignment but that's less interesting: https://web.archive.org/web/20240602012439/https://openai.com/superalignment/.)

(I like it, seems like a cool idea and maybe worth a try, and indeed a common thing that people mess up)