LessWrong 2.0 Reader

qaci seems to require the system having an understanding-creating property that makes it a reliable historian. have been thinking about this, have more to say, currently rather raw and unfinished.

A bit of feedback: the "We get a second chance at building AGI" outcome should not be an outcome or perhaps rephrased.

The former doesn't necessarily imply the latter in general, because even if we are systematically underestimating the realistic upper bound for our skill level in these areas, we would still have to deal with diminishing marginal returns to investing in any particular one.

On the other hand, even if what you say is true, skill headroom may still imply that it's worth building shared arts around such skills. Shareability and build-on-ability changes the marginal returns a lot.

The outcome depends on the details of the algorithm. Have you tried writing actual code?

If the code is literally "evaluate all options, choose the one that leads to more cups; if there is more than one such option, choose randomly", then the agent will choose randomly, because all options lead to the same amount of cups. That's what the algorithm literally says. Information like "at some moment the algorithm will change" has no impact on the predicted number of cups, which is literally the only thing the algorithm cares about.

When at midnight you delete this code, and upload a new code saying "evaluate all options, choose the one that leads to more paperclips; if there is more than one such option, choose randomly", the agent will start the factory (if it wasn't started already), because now that is what the code says.

The thing that you probably imagine, is that the agent has a variable called "utility" and chooses the option that leads to the highest predicted value in that variable. That is not the same as the agent that tried to maximize cups. This agent would be a variable-called-utility maximizer.

(Also, come on, LLMs are notoriously bad at math, plus if you push them hard enough you can convince them of a lot of things.)

Philology is philosophy, because it lets you escape the trap of the language you were born with. Much like mathematics, humanity's most ambitious such escape attempt, still very much in its infancy.

True...

If you really want to express the truth about what you feel and see, you need to be inventing new languages. And if you want to preserve a culture, you must not lose its language.

I think this is a mistake, made by many. It's a retreat and an abdication. We are in our native language, so we should work from there.

This has worked great btw! Thank you for the tip, I consistently get more deep sleep and around 10% more sleep with higher average quality, it's really good!

If we think of the quantified abilities as the logarithms of the true abilities, then taking the log has likely massively increased the correlations by bringing the outliers into the bulk of the distribution.

Are you afraid of dry eyes/meibomian gland dysfunction at all? It seems like it's pretty common as a side effect of retinoids.

AFAIK the distinction is that:

• When you condition on a particular outcome for $X$, it affects your probabilities for every other variable that's causally related to $X$, in either direction.
  • You gain information about variables that are causally downstream from $X$ (its "effects"). Like, if you imagine setting $X=x$ and then "playing the tape forward," you'll see the sorts of events that tend to follow from $X=x$ and not those that tend to follow from some other outcome $X=x^{\prime }$.
  • And, you gain information about variables that are causally upstream from $X$ (its "causes").  If you know that $X=x$, then the causes of $X$ must have "added up to" that outcome for $X$. You can rule out any configuration of the causes that doesn't "add up to" causing $X=x$, and that affects your probability distributions for all of these causative variables.
• When you use the do-operator to set $X$ to a particular outcome for X, it only affects your probabilities for the "effects" of $X$, not the "causes."  (The first sub-bullet above, not the second.)

For example, suppose hypothetically that I cook dinner every evening. And this process consists of these steps in order:

• "$W$": considering what ingredients I have in the house
• "$X$": deciding on a particular meal to make, and cooking it
• "$Y$": eating the food
• "$Z$": taking a moment after the meal to take stock of the ingredients left in the kitchen

Some days I have lots of ingredients, and I prepare elaborate dinners. Other days I don't, and I make simple and easy dinners.

Now, suppose that on one particular evening, I am making instant ramen ($X=makinginstantramen$). We're given no other info about this evening, but we know this.

What can we conclude from this?  A lot, it turns out:

• In $Y$, I'll be eating instant ramen, not something else.
• In $W$, I probably didn't have many ingredients in the house. Otherwise I would have made something more elaborate.
• In $Z$, I probably don't see many ingredients on the shelves (a result of what we know about $W$).

This is what happens when we condition on $X=makinginstantramen$.

If instead we apply the do-operator to $X=makinginstantramen$, then:

• We learn nothing about $W$, and from our POV it is still a sample from the original unconditional distribution for $W$.
• We can still conclude that I'll be eating ramen afterwards, in $Y$.
• We know very little about $Z$ (the post-meal ingredient survey) for the same reason we know nothing about $W$.

Concretely, this models a situation where I first survey my ingredients like usual, and am then forced to make instant ramen by some force outside the universe (i.e. outside our W/X/Y/Z causal diagram). 

And this is a useful concept, because we often want to know what would happen if we performed just such an intervention!

That is, we want to know whether it's a good idea to add a new cause to the diagram, forcing some variable to have values we think lead to good outcomes.

To understand what would happen in such an intervention, it's wrong to condition on the outcome using the original, unmodified diagram – if we did that, we'd draw conclusions like "forcing me to make instant ramen would cause me to see relatively few ingredients on the shelves later, after dinner."

I have heard people say this so many times, and it is consistently the opposite of my experience. The random spontaneous conversations at conferences are disproportionately shallow and tend toward the same things which have been discussed to death online already, or toward the things which seem simple enough that everyone thinks they have something to say on the topic. When doing an activity with friends, it's usually the activity which is novel and/or interesting, while the conversation tends to be shallow and playful and fun but not as substantive as the activity. At work, spontaneous conversations generally had little relevance to the actual things we were/are working on (there are some exceptions, but they're rarely as high-value as ordinary work).