LessWrong 2.0 Reader

Is there a prediction market for that?

I don't think there is, but you could make one!

Our overall best guess is that an important role of early MLPs is to act as a “multi-token embedding”, that selects^[1] the right unit of analysis from the most recent few tokens (e.g. a name) and converts this to a representation (i.e. some useful meaning encoded in an activation). We can recover different attributes of that unit (e.g. sport played) by taking linear projections, i.e. there are linear representations of attributes. Though we can’t rule it out, our guess is that there isn’t much more interpretable structure (e.g. sparsity or meaningful intermediate representations) to find in the internal mechanisms/parameters of these layers. For future mech interp work we think it likely suffices to focus on understanding how these attributes are represented in these multi-token embeddings (i.e. early-mid residual streams on a multi-token entity), using tools like probing and sparse autoencoders, and thinking of early MLPs similar to how we think of the token embeddings, where the embeddings produced may have structure (e.g. a “has space” or “positive sentiment” feature), but the internal mechanism is just a look-up table with no structure to interpret.

You may be interested in works like REMEDI and Identifying Linear Relational Concepts in Large Language Models.

In Fig 1, is the vertical axis P(world) ?

It seems to me like many of those business conglomerates are privately held. The stock market seems to prefer if businesses parts that have no synergy with the main business get sold off. 

On the stock market it's easy for capital owners to diversify their assets. When companies are privately held building such business conglomerates might be the way to diversify.

One feature of polyamory is that it means continuous auditions of potential replacements by all parties. You are not trading up in the sense that you can have multiple partners, but one thing leads to another and there are only so many hours in the day.

Polyamory is not that different from monogamy in this respect. It's just that in monogamy "having a relationship" is a binary: either you have it or you don't have it. In polyamory, there is a scale, starting from "meeting once in a blue moon" all the way to "living together with kids and joint finances". So, if in monogamy your attitude might be "I will not trade up unless I meet someone x% better", then in polyamory your attitude might be "I will devote you y% of my time and will not reduce this number unless there's someone x% better competing for this slot". (And in both cases x might be very high.)

More generally, I feel that a lot of arguments against polyamory fail the "replace with platonic friendship" test. Like, monogamous people also have to somehow balance the time they invest in their relationship vs. friends vs. family vs. hobbies etc, and also have to balance the time allocated to different friends. I know that some mono people feel that sex is some kind of magic pixie dust which makes a relationship completely different and not comparable in any way to platonic friendship, but... Not everyone feels this way? (In both directions: I simultaneously consider romantic relationship comparable to "mere" platonic friendships and also consider platonic friendships substantially more important/committing than seems to be the culturally-prescribed attitude.)

Also, it feels like this discussion has a missing mood and/or a typical mind fallacy. For me, monogamy was a miserable experience. Even aside from the fact you only get to have one relationship, there's all the weird rules about which things are "inappropriate" (see survey in the OP) and also the need to pretend that you're not attracted to other people (Not All Mono, but I think many relationships are like that). All the "pragmatic" arguments about why polyamory is bad sound to me similar to hypothetical arguments that gay relationships are bad. I mean, there might be some aspects of gay relationships that are often worse than corresponding aspects of straight relationships. But if you're gay, a gay relationship is still way better for you! Even if you're bi and in some sense "have a choice", it still seems inappropriate to try convincing you about how hetero is much better.

Warning: About to get a little ranty/emotional, sorry about that but was hard to express otherwise.

Finally, not to be that girl, but it's a little insensitive to talk about this without the least acknowledgement that polyamory is widely stigmatized and discriminated against. I know it's LessWrong here, we're supposed to use decoupling norms and not contextualizing norms, and I'm usually fully in favor of that, but it still seems to me that this post would better on the margin, if it had a little in the way of acknowledging this asymmetry in the debate. 

Instead, the OP talks about "encouraging widespread adaptation". What?? I honestly don't know, maybe in the Mythic Bay Area, someone is encouraging widespread conversion to polyamory. In the rest of the world, we only want (i) not be stigmatized (ii) not be discriminated against (iii) having some minimal awareness that polyamory is even an option (it was certainly an eye-opening discovery for me!) and (iv) otherwise, being left alone, and not have mono people endlessly explain to us how their way is so much better [My spouse tells me this last bit was too combative. Sorry about that: we are certainly allowed to have respectful discussion about the comparative advantages of different lifestyles.]

Let's say: if you train a coherently goal-directed, situationally aware, somewhat-better-than-human-level model using baseline forms of self-supervised pre-training + RLHF on diverse, long-horizon, real-world tasks, my subjective probability is ~25% that this model will be performing well in training in substantial part as part of an instrumental strategy for seeking power for itself and/or other AIs later.

Have you tried extending this gut estimate to something like:

If many labs use somewhat different training procedures to train their models but that each falls under the umbrella of "coherently goal-directed, situationally aware [...]", what is the probability that at least one of these models "will be performing well in training in substantial part as part of an instrumental strategy for seeking power for itself and/or other AIs later."?

I agree in principle that labs have the responsibility to dispel myths about what they're committed to

I don't know, this sounds weird. If people make stuff up about someone else and do so continually, in what sense it's that someone "responsibility" to rebut such things? I would agree with a weaker claim, something like: don't be ambiguous about your commitments with the objective of making it seem like you are committing to something and then walk back at the time you should make the commitment. 

I agree in principle that labs have the responsibility to dispel myths about what they're committed to. OTOH, in defense of the labs I imagine that this can be hard to do while you're in the middle of negotiations with various AISIs about what those commitments should look like.

This sounds like a case of the Rule of Equal and Opposite Advice: https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/ I'm sure for some people more honesty would be harmful, but it does sound like the caveats here make it clear when not to use it. I more agree with questions Tsvi raises in the other thread than with "this is awful advice". I can imagine that you are a person for whom more honesty is bad, although if you followed the caveats above it would be imo quite rare to do it wrong. I think the authors do a good job of outlining many cases where it goes wrong.

Interesting, but I'm not sure how successful the counterexample is. After all, if your terminal goal in the whole environment was truly for your side to win, then it makes sense to understand anything short of letting Shin play as a shortcoming of your optimization (with respect to that goal). Of course, even in the case where that's your true goal and you're committing a mistake (which is not common), we might want to say that you are deploying a lot of optimization, with respect to the different goal of "winning by yourself", or "having fun", which is compatible with failing at another goal.
This could be taken to absurd extremes [LW · GW] (whatever you're doing, I can understand you as optimizing really hard for doing exactly what you're doing), but the natural way around that is for your imputed goals to be required simple (in some background language or ontology, like that of humans). This is exactly the approach mathematically taken by Vanessa in the past (the equation at 3:50 here).
I think this "goal relativism" is fundamentally correct. The only problem with Vanessa's approach is that it's hard to account for the agent being mistaken (for example, you not knowing Shin is behind you).^[1]
I think the only natural way to account for this is to see things from the agent's native ontology (or compute probabilities according to their prior), however we might extract those from them. So we're unavoidably back at the problem of ontology identification (which I do think is the core problem).

1. ^{^}

   Say Alice has lived her whole life in a room with a single button. People from the outside told her pressing the button would create nice paintings. Throughout her life, they provided an exhaustive array of proofs and confirmations of this fact. Unbeknownst to her, this was all an elaborate scheme, and in reality pressing the button destroys nice paintings. Alice, liking paintings, regularly presses the button.
   A naive application of Vanessa's criterion would impute Alice the goal of destroying paintings. To avoid this, we somehow need to integrate over all possible worlds Alice can find herself in, and realize that, when you are presented with an exhaustive array of proofs and confirmations that the button creates paintings, it is on average more likely for the button to create paintings than destroy them.
   But we face a decision. Either we fix a prior to do this that we will use for all agents, in which case all agents with a different prior will look silly to us. Or we somehow try to extract the agent's prior, and we're back at ontology identification.
   
   (Disclaimer: This was SOTA understanding a year ago, unsure if it still is now.)