LessWrong 2.0 Reader

I agree that in the limit of an extremely structured optimizer, it will work in practice, and it will wind up following strategies that you can guess to some extent a priori.

I also agree that in the limit of an extremely unstructured optimizer, it will not work in practice, but if it did, it will find out-of-the-box strategies that are difficult to guess a priori.

But I disagree that there’s nothing between extremes where you can have it both ways.

On the contrary, I think it’s possible to design an optimizer which is structured enough to work well in practice, while simultaneously being unstructured enough that it will find out-of-the-box solutions very different from anything the programmers were imagining.

Examples include:

• MuZero: you can’t predict a priori what chess strategies a trained MuZero will wind up using by looking at the source code. The best you can do is say “MuZero is likely to use strategies that lead to its winning the game”.
• “A civilization of humans” is another good example: I don’t think you can look at the human brain neural architecture and loss functions etc., and figure out a priori that a civilization of humans will wind up inventing nuclear weapons. Right?

The problem with too-hungry people is that yes, they eat too much, and no, they cannot just stop. The way it works is so: they eat a meal, start feeling very hungry an hour later, make an effort and resist eating for another hour, then some more until they can't anymore and they give in and gobble up everything in the fridge. In cases like this skipping dinner is not actually possible, at least long-term.

I would consider adding salt to something to be making that thing less healthy. If adding salt is essential to making something edible, I think it would be healthier to opt for something that doesn't require added salt. That's speaking generally though, someone might not be getting enough sodium, but typically there is adequate sodium in a diet of whole foods.

I'd say it's too strong a claim that adding salt makes things less healthy. Remember that humans, eating generally mostly plants but some meat as well, developed rather strong craving for sodium salt, just like most herbivorous mammals. If you eat enough meat (not boiled) you don't need more sodium, if you eat a little meat or a lot but boiled, it's probably better to add some. If you eat only potatoes, you'll die without added salt (just kidding, who eats only potatoes).

In terms of factorizing or fingerprinting, 20 Tarot concepts seems like a lot; it's exhausting even just to skim it. Why do you think you need so many and that they aren't just mostly many fewer factors like Big Five or dirty uninterpretable mixes? Like the 500 closest tokens generally look pretty random to me for each one.

If this is the main reason why we should avoid ultra-processed food, then of course we'll have to avoid seed oils at any cost, as those are both ultra-processed and rather new, certainly not what we were evolved to breathe in at sea level.

I wasn't able to find the full video on the site you linked, but I found it here, if anyone else has the same issue: 

being able to credibly commit to doing this at appropriate times seems useful. I wouldn't want to commit to doing it at all times; becoming cooperatebot makes it rational for cooperative-but-preference-misaligned actors to exploit you. Shallow honesty seems like a good starting point for being able to say when you are attempting to be deep honest, perhaps. But for example, I would sure appreciate it if people could be less deeply honest about the path to ai capabilities. I do think the "deeply honest at the meta level" thing has some promise.

at that point you should just move to something optimized for being easy to get in and out of, like a bed

yes, yes. Exactly. Isn't it much more practical to put her in a bet/mattress on the floor? That's what we do. Just using the mattress from the crib, for example.

it's extremely high immediate value -- it solves IP rights entirely.

It's the barbed wire for IP rights

Nice! I actually had this as a loose idea in the back of my mind for a while, to have a network of people connected like this and have them signal to each other their track of the day, which could be actual fun. It is a feasible use case as well. The underlying reasoning is also that (at least for me) I would be more open to adopt an idea from a person with whom you feel a shared sense of collectivity, instead of an algorithm that thinks it knows me. Intrinsically, I want such an algorithm to be wrong, for the sake of my own autonomy :)

The way I see it, the relevance for alignment is to ask: what do we actually mean when saying that two intelligent agents are aligned? Are you and I aligned if we would make the same decision in a trolley problem? Or if we motivate our decisions in the same way? Or if we just don't kill each other? None of these are meaningful indicators of two people being aligned, let alone humans and AI. And with unreliable indicators, will we ever succeed in solving the issue? I'd say two agents are aligned when one agent's most rewarding decision results in a benefit of the other as well. Generalizing and scaling that alignment to many situations and many agents/people necessitates a 'theory of mind' mechanism, as well as a way to keep certain properties invariant under scaling and translation in complex networks. This is really a physicist's way of thinking about the problem and I am just slowly getting into the language that others in the AI/alignment fields use.