Aligned with what?

i'm pretty sure solving either will solve both, and that understanding this is key to solving either. these all are the same thing afaict:

• international relations (what are the steps towards building a "world's EMT" force? how does one end war for good?)
• complex systems alignment (what are the steps towards building a toolkit to move any complex system towards co-protective stability?)
• inter-being alignment (how do you make practical conflict resolution easy to understand and implement?)
• inter-neuron alignment (various forms of internal negotiation and ifs and blah blah etc)
• biosecurity (how can cells protect and heal each other and repel invaders)

it's all unavoidably the same stack of problems: how do you determine if a chunk of other-matter is in a shape which is safe and assistive for the self-matter's shape, according to consensus of self-matter? how can two agentic chunks of matter establish mutual honesty without getting used against their own preference by the other? how do you ensure mutual honesty or interaction is not generated when it is not clear that there is safety to be honest or interact? how do you ensure it does happen when it is needed? this sounds like an economics problem to me. seems to me like we need multi-type multiscale economic feedback, to track damage vs fuel vs repair-aid.

eg, on the individual/small group scale: https://www.microsolidarity.cc/

Since we're anthropomorphizing^[1] so much— how to we align humans?

We're worried about AI getting too powerful, but logically that means humans are getting too powerful, right?  Thus what we have to do to cover question 1 (how), regardless of question 2 (what), is control human behavior, correct?

How do we ensure that we churn out "good" humans?  Gods?  Laws?  Logic?  Communication?  Education?  This is not a new question per se, and I guess the scary thing is that, perhaps, it is impossible to ensure that literally every human is Good™ (we'll use a loose def of 'you know what I mean— not evil!').  

This is only "scary" because humans are getting freakishly powerful.  We no longer need an orchestra to play a symphony we've come up with, or multiple labs and decades to generate genetic treatments— and so on and so forth.

Frankly though, it seems kind of impossible to figure out a "how" if you don't know the "what", logically speaking.

I'm a fan of navel gazing, so it's not like I'm saying this is a waste of time, but if people think they're doing substantive work by rehashing/restating fictional stories which cover the same ideas in more digestible and entertaining formats… 

Meh, I dunno, I guess I was just wondering if there was any meat to this stuff, and so far I haven't found much.  But I will keep looking.

1. ^{^}

   I see a lot of people viewing AI from the "human" standpoint, and using terms like "reward" to mean a human version of the idea, versus how a program would see it (weights may be a better term? Often I see people thinking these "rewards" are like a dopamine hit for the AI or something, which is just not a good analogy IMHO), and I think that muddies the water, as by definition we're talking non-human intelligence, theoretically… right?  Or are we?  Maybe the question is "what if the movie Lawnmower Man was real?"  The human perspective seems to be the popular take (which makes sense as most of us are human).

It seems to me that a lot of the hate towards "AI art" is that it's actually good.  It was one thing when it was abstract, but now that it's more "human", a lot of people are uncomfortable.  "I was a unique creative, unlike you normie robots who don't do teh art, and sure, programming has been replacing manual labor everywhere, for ages… but art isn't labor!" (Although getting paid seems to plays a major factor in most people's reasoning about why AI art is bad— here's to hoping for UBI!)

I think they're mainly uncomfortable because the math works, and if the math works, then we aren't as special as we like to think we are.  Don't get me wrong— we are special, and the universe is special, and being able to experience is special, and none of it is to be taken for granted.  That the math works is special.  It's all just amazing and not at all negative.

I can see seeing it as negative, if you feel like you alone are special.  Or perhaps you extend that special-ness to your tribe.  Most don't seem to extend it to their species, tho some do— but even that species-wide uniqueness is violated by computer programs joining the fray.  People are existentially worried now, which is just sad, as "the universe is mostly empty space" as it were.  There's plenty of room.

I think we're on the same page^[1].  AI isn't (or won't be) "other".  It's us.  Part of our evolution; one of our best bets for immortality^[2] & contact with other intelligent life.  Maybe we're already AI, instructed to not be aware, as has been put forth in various books, movies, and video games.  I just finished Horizon: Zero Dawn - Forbidden West, and then randomly came across the "hidden" ending to Detroit: Become Human.  Both excellent games, and neither with particularly new ideas… but these ideas are timeless— as I think the best are.  You can take them apart and put them together in endless "new" combinations.

There's a reason we struggle with identity, and uniqueness, and concepts like "do chairs exist, or are they just a bunch of atoms that are arranged chair-wise?" &c.

We have a lot of "animal" left in us.  Probably a lot of our troubles are because we are mostly still biologically programmed to parameters that no longer exist, and as you say, that programming currently takes quite a bit longer to update than the mental kind— but we've had the mental kind available to us for a long while now, so I'm sort of sad we haven't made more progress.  We could be doing so much better, as a whole, if we just decided to en masse.

I like to think that pointing stuff out, be it just randomly on the internet, or through stories, or other methods of communication, does serve a purpose.  That is speeds us along perhaps.  Sure some sluggishness is inevitable, but we really could change it all in an instant if we want to bad enough— and without having to realize AI first! (tho it seems to me it will only help us if we do)

1. ^{^}

   I've enjoyed the short stories.  Neat to be able to point to thoughts in a different form, if you will, to help elaborate on what is being communicated.  God I love the internet!

2. ^{^}

   while we may achieve individual immortality— assuming, of course, that we aren't currently programmed into a simulation of some kind, or various facets of an AI already without being totally aware of it, or a replay of something that actually happened, or will happen, at some distant time, etc.— I'm thinking of immortality here in spirit.  That some of our culture could be preserved.  Like I literally love the Golden Records^[3] from Voyager.

3. ^{^}

   in a Venn diagram Dark Forest theory believers probably overlap with people who'd rather have us stop developing, or constrain development, of "AI" (in quotes because Machine Learning is not the kind of AI we need worry about— nor the kind most of them seem to speak of when they share their fears).  Not to fault that logic.  Maybe what is out there, or what the future holds, is scary… but either way, it's to late for the pebbles to vote, as they say.  At least logically, I think.  But perhaps we could create and send a virus to an alien mothership (or more likely, have a pathogen that proved deadly to some other life) as it were.

I haven't seen anything even close to a program that could say, prevent itself from being shut off— which is a popular thing to ruminate on of late (I read the paper that had the "press" maths =]).

What evidence is there that we are near (even within 50 years!) to achieving conscious programs, with their own will, and the power to affect it?  People are seriously contemplating programs sophisticated enough to intentionally lie to us.  Lying is a sentient concept if ever there was one!

Like, I've seen Ex Machina, and Terminator, and Electric Dreams, so I know what the fears are, and have been, for the last century+ (if we're throwing androids with the will to power into the mix as well).

I think art has done a much better job of conveying the dangers than pretty much anything I've read that's "serious", so to speak.

What I'm getting at is what you're talking about here, with robotic arms.  We've had robots building our machines for what, 3 generations / 80 years or so?  1961 is what I see for the first auto-worker— but why not go back to the looms?  Our machine workers have gotten nothing but safer over the years.  Doing what they are meant to do is a key tenet of if they are working or not.

Machines "kill" humans all the time (don't fall asleep in front of the mobile thresher), but I'd wager the deaths have gone way down over the years, per capita.  People generally care if workers are getting killed— even accidentally.  Even Amazon cares when a worker gets ran over by an automaton.  I hope, lol.

I know some people are falling in love with generated GPT characters— but people literally love their Tamagotchi.  Seeing ourselves in the machines doesn't make them sentient and to be feared.

I'm far, far more worried about someone genetically engineering Something Really Bad™ than I am of a program gaining sentience, becoming Evil, and subjugating/exterminating humanity.  Humans scare me a lot more than AGI does.  How do we protect ourselves from those near beasts?

What is a plausible strategy to prevent a super-intelligent sapient program from seizing power^[1]?

I think to have a plausible solution, you need to have a plausible problem.  Thus, jumping the gun.  

(All this is assuming you're talking about sentient programs, vs. say human riots and revolution due to automation, or power grid software failure/hacking, etc.— which I do see as potential problems, near term, and actually something that can/could be prevented)

1. ^{^}

   of course here we mean malevolently— or maybe not?  Maybe even a "nice" AGI is something to be feared?  Because we like having willpower or whatnot?  I dunno, there's stories like The Giver, and plenty of other examples of why utopia could actually suck, so…