LessWrong 2.0 Reader

My principles in these situations are:

1. Part of choosing a rule is choosing what the threshold is for breaking it.

2. You don't break rules, you change rules and then follow the new rule. So before you 'break' your current rule make sure you know what the new rule will be and that you like the new results better.

3. Cultivate the rule that rules are followed, that on the margin you are always going to underestimate the value of long term investment in habits and virtue cultivation, and that you don't change them in the moment without thinking about it carefully first.

How exactly does this differ from, "if the truth is on the wrong side politically, so much the worse for the truth"?

It differs in that there is no truth involved. The entire conversation is about which models and ontologies are best, without specifying what purpose they're serving. The abstraction is avoiding talking about any actual truth (what predictions will be made, and how the bets will be resolved), while asserting that it improves some abstract concept of truth.

I think that is a great insight. If we are going to rely on good institutions to tame behaviors, create the incentives that promote the desired behaviors, then we need the time for those institutions to develop to the point they can serve the purpose.

Perhaps rather than the current lit on P-A problems and how they are resolved today, one needs to look at the old histories about how such institutions arose and the various paths, and back tracking I suspect, that history shows.

If I understand one of the big concerns about AI in this regard, we also need to keep in mind that we might not have the same luxury of long times for institutional development and evolution in response to poor structures and the many unknowns that are discovered.

It's very hard (perhaps not possible in humans) to have communication without changing the payoff matrix. As soon as the framing changes to make the other player slightly more trustworthy or empathetic, the actual human evaluation will include other factors (kindness, self-image, etc.).

More importantly, the Nash equilibrium is kind of irrelevant to non-zero-sum games. There is no reason to believe that any optimization process is seeking it.

I think it is an important fact about how humans work that reinforcing the schilling fences and following them does in fact reinforce the values involved, whereas ignoring the fences does weaken them. Virtues and habits are something you cultivate through repeated action. This isn't simple signaling of values, it impacts those values for real.

As far as I can tell, meaning is a feeling, something like a passive sense that you’re on the right track. The feeling is generated when you are working on something that you personally enjoy and care about, and when you are socializing sufficiently often with people you enjoy and care about. “Friends and hobbies are the meaning of life” is how I might phrase it.

Note that the activity that you spend your time on could be collecting all the stars in Mario64, as long as you actually care about completing the task. However, you tend to find it harder to care about things that don’t involve winning status or helping people, especially as you get older.

I think some people get themselves into psychological trouble by deciding that all of the things that they enjoy aren’t “important” and interacting with people they care about is a “distraction”. They paint themselves into a corner where the only thing they allow themselves to consider doing is something for which they feel no emotional attraction. They feel like they should enjoy it because they’ve decided it’s important, but they don’t, and then they feel guilty about that. The solution to this is to recognize the kind of animal you are and try to feed the needs that you have rather than the ones you wish you had.

I think your epistemic status is right. Nash is not a good guide to what happens in one-shot games.

You give a high bid in TD (assuming no one cares what the bags are worth, only EV) because your expected returns are higher when you bid high, given most others will either not get to the Nash logic, or get far enough to realize that many others won't get there and therefore they shouldn't use it, or realize that those who do get there will realize others won't get there, or even just realize that even if everyone gets there some will choose to ignore it because even if a few others ignore it, you do better that way. And so on.

A thought experiment is, what is the right mixed strategy if your opponent will know what your exact mixed strategy is? I think of this as 'asymmetric Nash.'

I'm perhaps a bit of an outlier in that I see less difference among powerful optimization processes (evolution and humans being the only real examples I know of, and both extend to artificial versions) than most.

Over sufficient time periods, evolution is subject to the same constraints and tradeoffs as intelligent world-modeling agents are. Evolution is slow enough that it can bypass some of the identity problems that agents have, but it's also only going to find viable equilibria.

They're not ants, they're hybrid ant-human metaphors. Ants don't talk and don't wonder if the grasshopper is right. Ants don't consider counterfactual cases of never having met the other colony. Metaphorical ants that _CAN_ do these things can also consider other strategies than war.

It's locking in the moral/preference status quo; once that's done, non-Pareto overall gains are fine.

Even when locking in that status quo, it explicitly trades off certain values against others, so there is no "only Pareto" restriction.

I have a research agenda to be published soon that will look into these issues in more detail.