LessWrong 2.0 Reader

I'd like there to be a reaction of "Not Exhaustive", meant for a step where comment (or top-level post, for that matter) missed an important case - how a particular situation could play out, perhaps, or an essential system's component is not listed. An example use: on statement "to prevent any data leaks, one must protect how their systems transfer data and how they process it" with the missed component being protection of storage as well.

I recall wishing for it like three times since the New Year, with the current trigger being this comment [LW(p) · GW(p)]:

Elon already has all of the money in the world. I think he and his employs are ideologically driven, and as far as I can tell they're making sensible decisions given their stated goals of reducing unnecessary spend/sprawl. I seriously doubt they're going to use this access to either raid the treasury or turn it into a personal fiefdom. <...>

which misses a specific case (which I'll name under the original comment if there is any interest).

Make the (!aligned!) AGI solve a list of problems, then end all other AIs, convince (!harmlessly!) all humans to never make another AI, in a way that they will pass down to future humans, then end itself. 

I don't claim to know all the key pieces of whatever they did, but some pieces which are obvious from talking to them both and following their writings:

• Both invested heavily in self-studying a bunch of technical material.
• Both heavily practiced the sorts of things Joe described in his recent "fake thinking and real thinking [LW · GW]" post. For instance, they both have a trained habit of noticing the places where most people would gloss over things (especially technical/mathematical), and instead looking for a concrete example.
• Both heavily practiced the standard metacognitive moves, e.g. noticing when a mental move worked very well/poorly and reinforcing accordingly, asking "how could I have noticed that faster", etc.
• Both invested in understanding their emotional barriers and figuring out sustainable ways to handle them. (I personally think both of them have important shortcomings in that department, but they've at least put some effort into it and are IMO doing better than they would otherwise.)
• Both have a general mindset of actually trying to notice their own shortcomings and improve, rather than make excuses or hide.
• And finally: both put a pretty ridiculous amount of effort into improving, in terms of raw time and energy, and in terms of emotional investment, and in terms of "actually doing the thing for real not just talking or playing".

Your answer wasn't cryptic at all. Don't worry. This is a great answer. Let me know when you're done with that sequence. I'll have to read it.

(Also, it's horrifying that people can be hypnotized against their will. That makes me simultaneously thankful-that and curious-why it isn't more widely practiced...) 

I expected far greater pushback from doctors and lawyers, for example, than we have seen so far.

I believe it's a matter of (correlated) motivated reasoning. My doctor and lawyer friends both seem to be excited about the time that AIs will save them as they do their jobs—in both professions there is immense efficiency to gain by automating more rote parts of the work (legal research, writing patient notes, dealing with insurance, etc.)—but seem to believe that AIs will never fully supplant them. When I press the issue, especially with my doctor friend, he tells me that regulations and insurance will save doctors, to which I say... sure, but only until e.g. AI-powered medicine has real statistics showing they have better health outcomes than human doctors as most of us here would expect. I can imagine the same initial defense from a lawyer who cannot yet imagine an AI being allowed by regulation to represent someone.

Then there's all the usual stuff about how difficult it can be for many people to imagine the world changing so much in the next several years.

I've also heard this argument: sure, AI might take everyone's job, but if that's inevitable anyway, it's still rational to be in an elite profession because they will last slightly longer and/or capture more economic surplus before society breaks down, if it does. On that point, I agree.

More broadly, speaking from the outside (I am a software engineer), to me the cultures of the elite professions have always seemed rather self-assured: everything is fine, nothing is a problem and these elite professionals will always be rich... which means that when the first credible threat to that standing hits, like a jurisdiction allowing fully autonomous doctors/lawyers/etc., it will be pandemonium.

Something like TNIL or Real Character might be used for maximum intellectual utility. But I cannot see how simply minimizing the amount of words that need to exist for compact yet precise communication would help correct the corrupted machinery our minds run on.

By "make its users more aware of their biases" I mean, for example, a language where it's really obvious when you say something illogical, or have a flaw in your reasoning.

Some ideas I had for this:

• Explicitly defined sematic spaces for every word, to dissolve questions [LW · GW] and help people agree on the locations of phenomena in thingspace. Mechanisms for searching thingspace (while, for example, you can say "red chair" to narrow the space of all chairs down to the space of all chairs which reflect red light, it would be nice to be able to express things like "thing such that there is no meaningful answer to the question 'is it a chair' "
• A fourth grammatical category which, alongside tense, aspect, and mood, indicates your relationship to the statement. For example, "Apples taste good" would have a relationship of "speaker's observation-derived opinion", and "Mars is a planet" would have a relationship of "property of speaker's consensus-derived world model". This would make the statement "Socrates is a man" feel more like "My brain perceptually classifies Socrates as a match against the 'human' concept"

• Distinctions between "logically true" (true in all possible worlds, i.e. 1+1=2), "Occam's-razor true" (~100% chance of being true in all minus a negligible amount of the worlds that could be ours, i.e. there is no God), and "empirically true" (~100% chance of being true in THIS world specifically, i.e. the sky is blue), and "illogically infinitely true" (true in this world but not in all possible worlds. This is impossible IRL and would only be used in thought experiments.)

   

I just tried to replicate this on GPT-2 with expansion factor 4 (so total number of centroids = 768 * 4). I get that clustering recovers  ~87% fraction of variance explained, while a k = 32 SAE gets more like 95% variance explained. I did the nonlinear version of finding nearest neighbors when using k means to give k means the biggest advantage possible, and did k-means clustering on points using the FAISS clustering library. 

Definitely take this with a grain of salt, I'm going to look through my code and see if I can reproduce your results on pythia too, and if so try on a larger model to. Code: https://github.com/JoshEngels/CheckClustering/tree/main

Elon already has all of the money in the world. I think he and his employs are ideologically driven, and as far as I can tell they're making sensible decisions given their stated goals of reducing unnecessary spend/sprawl. I seriously doubt they're going to use this access to either raid the treasury or turn it into a personal fiefdom. It's possible that in their haste they're introducing security risks, but I also think the tendency of media outlets and their sources will be to exaggerate those security risks. I'd be happy to start a prediction market about this if a regular feels very differently.

If Trump himself was spearheading this effort I would be more worried.

Now I feel like rationality itself is an infohazard. I mean, rationality itself won't hurt you if you are sufficiently sane, but if you start talking about it, insufficiently sane people will listen, too. And that will have horrible consequences. (And when I try to find a way to navigate around this, such as talking openly only to certifiably sane people, that seems like the totally cultish thing to do.)

There is an alternative way, the other extreme: get more and more rationalists.
If the formed communities do not share the moral inclinations of LW community, those might form some new coordination structures^[1]; if we don't draw from the circles of desperate, those structures will tend to benefit others as well (and, on the other hand, having a big proportion of very unsatisfied people would naturally start a gang or overthrow whatever institutions are around).

(It's probably worth exploring in a separate post?)

1. ^{^}

   I claim non-orthogonality between goals and means in this case. For some community with altruistic people, its structures require learning a fair bit about people's values. For a group which wants tech companies to focus on consumers' quality-of-life more than currently, not so.