Hacking humans

I don't think this is a pertinent or useful suggestion. The point of the reply wasn't to discuss politics, and I think it's a red herring to dismiss it as if it were.

If I may expand on tukabel's response: What is the point of this post? It seems to be some sort of "new" analysis as to how AIs could potentially hack humans—but if we get passed the "this is new and interesting" presentation, it doesn't seem to give anything new, unusual, or even really discussion-worthy.

Why is "The AI convinces/tricks/forces the human to do a specific action" something that's remarkable? What does "Different levels of hacking make different systems vulnerable, and different levels of interaction make different types of hacking more or less likely" even mean? It sounds like an overly verbose and convoluted way of saying "People respond differently to different things."

Maybe there's some background discussion that provides a context I've been missing here, but this doesn't seem like anything that hasn't already been going on amongst humans for thousands of years, and that's a relevant and useful thing to draw from.

Much more succinct: Why are the ways in which an AI can "hack" a human (i.e. - affect them) any different than the ways a human can affect a human? If we replace "AI" with "Human" it'd be trivial and a bit silly.

How much have you explored the REASONS that brainwashing is seen as not cool, while quiet rational-seeming chat is perfectly fine? Are you sure it's only about efficacy?

I worry that there's some underlying principle missing from the conversation, about agentiness and "free will" of humans, which you're trying to preserve without defining. It'd be much stronger to identify the underlying goals and include them as terms in the AI's utility function(s).

The risk with an AI is that it would be capable of changing humans in ways similar to the more dubious methods, while only using the "safe" methods.

I think what you're saying makes sense, but I'm still on Dagon's side. I'm not convinced this is uniquely an AI thing. It's not like being a computer gives you charisma powers or makes you psychic -- I think that basically comes down to breeding and exposure to toxic waste.

I'm not totally sure it's an AI thing at all. When a lot of people talk about an AI, they seem to act as if they're talking about "a being that can do tons of human things, but better." It's possible it could, but I don't know if we have good evidence to assume AI would work like that.

A lot of parts of being human don't seem to be visible from the outside, and current AI systems get caught in pretty superficial local minima when they try to analyze human behavior. If you think an AI could do the charisma schtick better than mere humans, it seems like you'd also have to assume the AI understands our social feelings better than we understand them.

We don't know what the AI would be optimizing for and we don't know how lumpy the gradient is, so I don't think we have a foothold for solving this problem -- and since finding that kind of foothold is probably an instance of the same intractable problem, I'm not convinced a really smart AI would have an advantage against us on solving us.