LessWrong 2.0 Reader

Neat idea! I've been thinking about things along this line for years because of science fiction writers like Charles Stross who wrote about the idea of having digital clones of yourself that you could spawn to act as assistants and do things like enter a simulation with a set of information to analyzed, and return a report after being run faster-than-realtime. For example, to analyze potentially infohazardous information sent by an untrusted party.

Also, the idea that people might assign their digital clones to go on a date, and then agree to go on a date if both their clones came back with a positive recommendation.

Of course, literally using a digital clone of yourself requires being rather cavalier about destroying conscious beings after you are done using them as a tool. Seems like it makes a lot more sense to use a non-conscious tool-AI without emotions for this sort of purpose.

Not quite what we were trying to say in the post. Rather than tradeoffs being decided on reflection, we were trying to talk about the causal-inference-style "explaining away" which the reflection gives enough compute for. In Johannes's example, the idea is that the sadist might model the reward as coming potentially from two independent causes: a hardcoded sadist response, and "actually" valuing the pain caused. Since the probability of one cause, given the effect, goes down when we also know that the other cause definitely obtained, the sadist might lower their probability that they actually value hurting people given that (after reflection) they're quite sure they are hardcoded to get reward for it. That's how it's analagous to the ant thing.

I just want to say that I am someone who is afraid that the world is currently in a very offense-dominant strategic position currently, but I don't think defense is pointless at all. I think it's quite tractable and should be heavily invested in! Let's get some d/acc going people!

In fact, a lot of my hope for good outcomes for the future route through Good Actors (probably also making a good profit) using powerful tool-AI to do defensive acceleration of R&D in a wide range of fields. Automated Science-for-Good, including automated alignment research. Getting there without the AI causing catastrophe in the meantime is a challenge, but not an intractable one.

The universe has many conserved and approximately-conserved quantities, yet among them energy feels "special" to me. Some speculations why:

• The sun bombards the earth with a steady stream of free energy, which leaves out into the night.
• Time-evolution is determined by a 90-degree rotation of energy (Schrodinger equation/Hamiltonian mechanics).
• Breaking a system down into smaller components primarily requires energy.
• While aspects of thermodynamics could apply to many conserved quantities, we usually apply it to energy only, and it was first discovered in the context of energy.

I guess the standard rationalist-empiricist-reductionist answer would be to say that this is all caused by the second point combined with some sort of space symmetry. I would have agreed until recently, but now it feels circular to me since the reduction into energy relies on our energy-centered way of perceiving the world. So instead I'm wondering if the first point is closer to the core.

These two kinds of “learning” are not synonymous. Adaptive systems “learn” things, but they don’t necessarily “learn about” things; they don’t necessarily have an internal map of the external territory. (Yes, the active inference folks will bullshit about how any adaptive system must have a map of the territory, but their math does not substantively support that interpretation.) The internal heuristics or behaviors “learned” by an adaptive system are not necessarily “about” any particular external thing, and don’t necessarily represent any particular external thing.

I think I am confused both about whether I think this is true, and about how to interpret in such a way that it might be true. Could you go into more detail on what it means for a learner to learn something without there being some representational semantics that could be used to interpret what it's learned, even if the learner itself doesn't explicitly represent those semantics? Or is the lack of explicit representation actually the core substance of the claim here?

Yes exactly. The larva example illustrates that there are different kinds of values. I thought it was underexplored in the OP to characterize exactly what these different kinds of values are.

In the sadist example we have:

1. the hardcoded pleasure of hurting people.
2. And we have, let's assume, the wish to make other people happy.

These two things both seem like values. However, they seem to be qualitatively different kinds of values. I intuit that more precisely characterizing this difference is important. I have a bunch of thoughts on this that I failed to write up so far.

I will look forward to that. I have read the LDSL posts, but I cannot say that I understand them, or guess what the connection might be with destiny and higher powers.

Okay, I think one crystallization here for me is that "explaining away" is a matter of degree. (I think I found the second half of the comment less helpful, but the combo of the first half + John's response is helpful both for my own updating, and seeing where you guys are currently at)

I prefer to see Reality as "nihil supernum" rather than Goodness. Reality does not speak. It promises me nothing. It owes me nothing. If it is not as I wish it to be, it is up to me, and me only, to act to make it more to my liking. There is no-one and nothing to magically make things right. Or wrong, for that matter.

This does not have the problem that the Goodness idea has, of how to justify calling it Good to people who are in very bad circumstances. Nor is there any question of dropping a letter and calling it God.

We have been shown that this search algorithm works, and we not yet have been shown that the other approaches don't work.

Remember, technological development is disjunctive, and just because you've shown that 1 approach works, doesn't mean that we have been shown that only that approach works.

Of course, people will absolutely try to scale this one up now that they found success, and I think that timelines have definitely been shortened, but remember that AI progress is closer to a disjunctive scenario than conjunctive scenario:

I agree with this quote below, but I wanted to point out the disjunctiveness of AI progress:

As we spoke earlier - it was predictable that this was going to be the next step. It was likely it was going to work, but there was a hopeful world in which doing the obvious thing turned out to be harder. That hope has been dashed - it suggests longer horizons might be easy too. This means superintelligence within two years is not out of the question.

https://gwern.net/forking-path