This post clearly helped a lot of other people, but it follows a pattern that many other posts on Less Wrong also follow which I consider negative. The valuable contribution here is not the formalisation, but the generator behind the formalisation. The core idea appears to be the following:

"Human brains contain two forms of knowledge. Explicit knowledge and weights that are used in implicit knowledge (admittedly the former is hacked on top of the later, but that isn't relevant here). Mary doesn't gain any extra explicit knowledge from seeing blue, but her brain changes some of her implicit weights so that when a blue object activates in her vision a sub-neural network can connect this to the label "blue"."

Unfortunately, there is a wall of maths that you have to wade through before this is explained to you. I feel it is much better when you provide your readers with a conceptual understanding of what is happening and only then include the formal details.

Cool, this seems like decent reason to try to rewrite this in a slightly more polished way.

I am struggling to understand the goal of the post.

The title was helpful to me in that regard. Each of these examples shows an agent who could run an honest process to get evidence on a question, but which prefers one answer so much that they try to stack the deck in that direction, and thereby loses the hoped-for benefits of that process.

Getting an honest Yes requires running the risk of getting a No instead.

I wonder if the current state of the art corresponds to the pre-conscious level of evolution, before the internal narrator and self-awareness. Maybe soon the neural networks will develop the skill of explaining (or rationalizing) their decisions.

Thanks for clarifying!

people with negative intentions [...] deliberately

So, it's actually not clear to me that deliberate negative intentions are particularly important, here or elsewhere? Almost no one thinks of themselves as deliberately causing avoidable harm, and yet avoidable harm gets done, probably by people following incentive gradients that predictably lead towards harm, against truth, &c. all while maintaining a perfectly sincere subjective conscious narrative about how they're doing God's work, on the right side of history, toiling for the greater good, doing what needs to be done, maximizing global utility, acting in accordance with the moral law, practicing a virtue which is nameless, &c.

it was important for me to speak up, even if you're not a werewolf.

Agreed. If I'm causing harm, and you acquire evidence that I'm causing harm, then you should present that evidence in an appropriate venue in order to either persuade me to stop causing harm, or persuade other people to coördinate to stop me from causing harm.

I was claiming that the long post you wrote at the top of the thread where you made several analogies about your response, were exactly the sort of gray area situations where, depending on context, the community might decide to sacrifice it's sacred value.

So, my current guess (which is only a guess and which I would have strongly disagreed with ten years ago) is that this is a suicidally terrible idea that will literally destroy the world. Sound like an unreflective appeal to sacred values? Well, maybe!—you shouldn't take my word for this (or anything else) except to the exact extent that you think my word is Bayesian evidence. Unfortunately I'm going to need to defer supporting argumentation to future Less Wrong posts, because mental and financial health requirements force me to focus on my dayjob for at least the next few weeks. (Oh, and group theory.)

(End of thread for me.)

Hi Henrik,

Thanks for your thoughts on (what I think) is a very important topic. Have you read Superintelligence by Nick Bostrom, Rationality: A-Z: or related texts? Those texts and others read by users of LessWrong address your argument and point out some mistakes you are making. I would guess that your post is being downvoted because it ignores the standard replies to the arguments you are making. (In part, posts get downvoted to signal to other users that they're not worth looking, sometimes because they're ignoring expected background material.)

My advice is to first read Superintelligence and Rationality: A-Z and then come back if you'd like to discuss these topics further.

Nod. I could see that. Agree with the "depends on whether we can find a reasonable place to put the link without adding noise" clause.

Since this seems to be an akrasia/executive-related problem, I suspect just having links to possible addons to use (and ideally, example configurations) easily accessible could be disproportionately ameliorative compared to its implementation cost, both via the reminder that compulsive browsing and mitigations for it both exist, and via the social signaling that this is an approved way of browsing that won't make you weird. Though I'm not sure about the possible noise it creates, depending on what easy options you have for placement/hiding.

Humans have easy-to-extract preferences over possible "wiser versions of ourselves." That is, you can give me a menu of slightly modified versions of myself, and I can try to figure out which of those best capture my real values (or over what kind of process should be used for picking which of those best capture my real values, or etc.). Those wiser versions of ourselves can in turn have preferences over even wiser/smarter versions of ourselves, and we can hope that the process might go on ad infinitum.

This seems a pretty bold claim to me. We might be tempted to construe our regular decision making process as doing this (I come up with what wiser-me might do in the next instant, and then do it), but this to me seems to be misunderstanding how decisions happen by confusing the abstraction of "decision" and "preferences" for the actual process that results in the world ending up in a causally subsequent state which I might later look back on and reify as myself having made some decision. Since I'm suspicious that something like this is going on when the inferential distance is very short, I'm even more suspicious when the inferential distance is longer, as you seem to be proposing.

I'm not sure if I'm arguing against your claim that the situations are not symmetrical, but I do think this reasoning for thinking the situations are not symmetrical is likely flawed because it seems to be to be assuming something about humans being fundamentally different from e-coli that is not.

(There are of course many differences between the two, just not ones that seem relevant to this line of argument.)

Concretely: I'd be interested in Wei Dai or others turning the answers here into "Related Questions", which can then be actually answered.

(Noting that, for now, related questions are default be hidden from the frontpage, although admins can manually toggle them to be displayed, and they would appear in the Open Questions page and the Recent Discussion section).