LessWrong 2.0 Reader

I spent some time learning about neural coding once, and while interesting it sure didn't help me e.g. better predict my girlfriend; I think in general neuroscience is fairly unhelpful for understanding psychology. For similar reasons, I'm default-skeptical of claims that work on the level of abstraction of current ML are likely to help much with figuring out whether powerful systems trained via ML are trying to screw us, or with preventing that.

As a LW reader, I find it to be a great summary!

Eliezer wrote an article about this phenomenon [LW · GW], where people think they are writing for a less sophisticated audience than they actually are. 

Am I missing something in your meaning? If the sample is an action which selects the policy, it's in that population's past. If it doesn't select the policy, a sampled entity's subjective probabilities, if they know about the possibility of both policies, can take into account uncertainty about whether one, the other, or both policies were used. E.g. if a sampled entity doesn't know which policy was used, but knows only one of those two were used, they report a subjective probability of $\frac{1}{2}$.

A Bayesian agent should have a space of hypotheses based on uncertainty about the policies which brought any particular reasoners about, and shouldn't be missing any of the tools needed to do this.

I have certainly seen that type of frustrating unwillingness to update on his part at times occur as well, but I haven't seen indications of bad faith. (I suspect this could be because your interpretation of the phrase "bad faith" is different and far more extensive than mine.)

The crux is timing, not doom. In the absence of doom, savings likely become similarly useless. But in the absence of superintelligence (doom or not), savings remain important.

We know what societies that mutilate prisoners are like, because plenty of them have existed.

This is where I disagree. There are only a few post-industrial socieities that have done this, and they were already rotten before starting the mutilation (e.g. Nazi Germany). There is nothing to imply that mutilation will turn your society rotten, only that when your society becomes rotten mutilation may begin.

Significant Digits is (or was, a few years ago) considered the best one, to my recollection.

Do you have a link/citation for this quote? I couldn't immediately find it.

Your introduction describes how I feel about my area of expertise too!

Working in the field of *neuroscience* is a bizarre experience. No one seems to be interested in the most interesting applications of their research. Neuroscience has significantly advanced in the past few decades. I keep telling people that soon we're all going to have self driving cars powerd by a horse's brain in a glass jar. Most people just laugh in disbelief, others make frightened noises and mention "ethical issues" or change the subject. The smart money is off chasing the deep learning pipe dream, and as a direct consequence, there is low-hanging fruit absolutely everywhere.

The 3 month Eliezer sim might spin up many copies of other 3 month Eliezer sims, which together produce outputs that a 6-month Eliezer sim might produce.

This seems very blatantly not viable-in-general, in both theory and practice.

On the theory side: there are plenty of computations which cannot be significantly accelerated via parallelism with less-than-exponential resources. (If we do have exponential resources, then all binary circuits can be reduced to depth 2, but in the real world we do not and will not have exponential resources.) Serial computation requirements are, in fact, a thing. So you can't just have a bunch of Eliezers do in 3 months what a single 6 month Eliezer could do, in general.

Even if you allow the 3-month Eliezer sims to act one-after-another, rather than trying to get everything done in 3 months simulated time via pure parallelism, there's still a tight communication bottleneck between each Eliezer sim. There are presumably plenty of circuits which cannot be implemented with tight information bottlenecks every n serial steps.

... of course you could circumvent all that theory by e.g. just having each Eliezer emulate a single logic gate, or some comparably trivial thing, but at that point you run afoul of the non-compositionality of safety properties [? · GW]: putting together a bunch of "safe" things (or "interpretable" things, or "aligned" things, or "corrigible" things, ...) does not in-general produce a safe thing.

So that's the theory side. What about the practice side?

Well, Ought did that roughly that experiment years ago and it did not work at all [? · GW]. And that should not be surprising - as the link argues, we have extremely ample evidence from day-to-day life that such things do not work in general.