LessWrong 2.0 Reader

What's an "anti-rationalist" group?

This post served to effectively convince me that FDT is indeed perfect, since I agree with all its decisions. I'm surprised that Claude thinks paying Omega the 100$ has poor vibes.

I'm less and less convinced that we should expect to see AIs that are close to pure consequentialists

There was a case when ChatGPT preferred not to violate the taboo on racial slurs, even though in the hypothetical situation it meant killing millions of people. In a re-run of the experiment ChatGPT decided to use the slur, but it also remarked that the use is a complex ethical dilemma. How can one check whether the AI will prefer not to violate the taboo on colonialism? By placing it into a simbox [LW · GW] where one also has analogues for peoples that are easy to be taken over?

P.S. I doubt that a non-neuromorphic AI is even able to take over the world and run it since the world's entire energy generation might require too much intellectual work to do by the AI itself. There was a post [LW · GW] claiming that even a neuromorphic AI is unlikely to become much more efficient than the brain.

LLM activation space is spiky. This is not a novel idea but something I believe many mechanistic interpretability researchers are not aware of. Credit to Dmitry Vaintrob [LW · GW] for making this idea clear to me, and to Dmitrii Krasheninnikov [LW · GW] for inspiring this plot!

Under the superposition hypothesis, activations are linear combinations of a small number of features. This means there are discrete subspaces in activation space that are "allowed" (can be written as the sum of a small number of features), while the remaining space is "disallowed" (require much more than the typical number of features).^[1]

Here's a toy model (following TMS, $d_{vocab}=8$ total features in $d_{embed}=3$-dimensional activation space, with $k=1,2,3$ features allowed to be active simultaneously). Activation space is made up of discrete $k$-dimensional (intersecting) subspaces. My favourite image is the middle one ($k=2$) showing planes in 3d activation space because we expect $1\ll k\ll d_{embed}$ in realistic settings.

($n_{active}$ in the plot corresponds to $k$ here. Code here.)

This picture predicts that interpolating between two activations should take you out-of-distribution relatively quickly (up to possibly some error correction) unless your interpolation (steering) direction exactly corresponds to a feature. I think this is relevant because

1. it implies my stable region experiment series [we observe models are robust to perturbations of their activations, 1 [LW · GW], 2 [LW · GW], 3 [LW · GW], 4 [LW · GW]] should be quite severely out-of-distribution, which makes me even more confused about our results.
2. it predicts activation steering to be severely out-of-distribution unless you pick a steering direction that is aligned with (a linear combination of) active feature directions.
3. it predicts that linear probing shouldn't give you nice continuous results: Probing into a feature direction should yield just interference noise most of the time (when the feature is inactive), and significant values only when the feature is active. Instead however, we typically observe non-negligible probe scores for most tokens.^[2]

Can confirm. Half the LessWrong posts I've read in my life were read in the shower.

Unfortunately, current mobile phones are waterproof. 

If we know the correct answers to decision theory problems, we have some internal instrument: either a theory or a vibe meter, to learn the correct answers. 

Claude seems to learn to mimic our internal vibe meter. 

The problem is that it will not work outside the distribution. 

Actually... I will say it: This feels like a fast rebranding of the Halting Problem, like without actually knowing what it implies.

Being able to rebrand an argument so that it could talk about a different problem in a valid way is exactly what is to understand it - not just repeat the same words in the same context that the teacher said but generalize it. We can go into the realm of second order logic and say that

For every property that at least one program has, a universal detector of this property has to itself have this property on at least some input.

Mind you, I wasn't trying to prove you that I understand Turing's proof. Previously, you claimed that your "argument follows exactly the same syllogistic structure ("If this, then that") as Turing’s proof". So I showed you what it actually is when an argument does follow the same structure. If what you are talking was just like that then you could simply do the same. 

How would a virus (B) know what the antivirus (A) predicts about B?

By running a copy of antivirus software on its own code and checking its output. That's a valid program.

it’s unintuitive — almost so that it's false.

It can't quarry an antivirus software.

Why not? Anivirus software is a valid program, it has some code that can be executed on some input. You can include execution of this code on a particular input in your own program. If this is not intuitive for you, then maybe it's you who do not understand Turing's proof?  

But seems people here are not versed in classical computer science only shouting "Bayeism! Bayeism!"

Well, I'm not making any claims about an average LessWronger here, but between the two of us, it's me who has written an explicit logical proof of a theorem and you who is shouting "Turing proof!", "Halting machine!" "Godel incompletness!" without going into the substance of them. 

Maybe there is an actual gear-level model inside your mind how all this things together build up to your conclusion but you are not doing a good job at communicating it. You present metaphors, saying that thinking that we are conscious, while not actually being conscious is like being a merely halting machine, thinking that it's a universal halting machine. But it's not clear how this is applicable. What does it even mean for a machine to have a belief about something? That's not something Turing defines in his proof. It's possible to formally prove that universal halting machine is impossible. Can you do the same for having consciousness? If you can - then just do it, that would be very helpful and allow us to talk about the substance, not just vibes. 

which is proven to be effectively wrong by the sleeping beauty paradox (frequentist "thirder's" get more money in simulations.)

Oh boy, do [LW · GW] I [LW · GW] have [LW · GW] an [LW · GW] opinion [LW · GW]. Here you are wrong in three different ways.

1. Disagreement in Sleeping Beauty is not between bayesianism and frequentism. Thirdism is not frequentist. Halfism is not bayesian. One can make a bayesian argument in favor of thirdism: how you learn that you are awaken 'today' which is, alledgedly, new information. Or frequentist argument in favor of halfism: how if we repeat the experiment a lot of times, in about 1/2 of such iterations of it, where at least one awakening happens, the coin is Heads.
2. In general frequentism and bayesianism do not have disagreement of such kind. There are sitiations (unfair coin toss without any other details) where frequentist claim that probability is undefined, where bayesians are ready to assign a numerical value to it, but not ones where two different numerical values are assigned.
3. While following lewisian halfers betting odds in Sleeping Beauty indeed performs terribly, double halfers do no worse than thirders. There are also some cases where thirdism gives very stupid answers, like betting on whether at least one of the awakening happens on Monday.

Yes. See archive.org of original it has the same title and the same duration (Look for <meta itemprop="duration" content="PT51M25S"> in the archived HTML; compared to the 51:24 of your link).
I have edited the Wiki, thanks for finding the link!

Consequentialism is an approach for converting intelligence (the ability to make use of symmetries to e.g. generalize information from one context into predictions in another context or to e.g. search through highly structured search spaces) into agency, as one can use the intelligence to predict the consequences of actions and find a policy which achieves some criterion unusually well.

While it seems intuitively appealing that non-consequentialist approaches could be used to convert intelligence into agency, I have tried a lot and not been able to come up with anything convincing. For virtues in particular, I would intuitively think that a virtue is not a motivator per se, but rather the policy generated by the motivator. So I think virtue-driven AI agency just reduces to ordinary programming/GOFAI, and that there's no general virtue-ethical algorithm to convert intelligence into agency.

The most straightforward approach to programming a loyal friend would be to let the structure of the program mirror the structure^[1] of the loyal friendship. That is, you would think of some situation that a loyal friend might encounter, and write some code that detects and handles this situation. Having a program whose internal structure mirrors its external behavior avoids instrumental convergence (or any kind of convergence) because each behavior is specified separately and one can make arbitrary exceptions as one sees fit. However, it also means that the development and maintenance burden scales directly with how many situations the program generalizes to.

1. ^{^}

   This is the "standard" way to write programs - e.g. if you make a SaaS app, you often have template files with a fairly 1:1 correspondence to the user interface, database columns with a 1:many correspondence to the user interface fields, etc.. By contrast, a chess bot that does a tree search does not have a 1:1 correspondence between the code and the plays; for instance the piece value table does not clearly affect it's behavior in any one situation, but obviously kinda affects its behavior in almost all situations. (I don't think consequentialism is the only way for the structure of a program to not mirror the structure of its behavior, but it's the most obvious way.)