LessWrong 2.0 Reader

I have a question. This is beond my area of competence - understood few of the technical parts - so bear with me here :)

This is how I understand the parts relevant to my question:

You want to:

1.  Simulate the chaotic environment to be "perceived"
2. Formalize the perceiving. "info-at-a-distance"
3. Check if the "perceived info" contains patterns/"structural organizations" similar to those used by humans.

The question is about fundamental assumptions of this work.
Given that my understanding is correct, then I wonder:

Since you create both the simulated environment, and the "perceiver", wouldn't that risk introducing bias? If the "perceiver" outputs human-like structures, is there any way to say whether they originate from an "objective" process, or if they originate from you? 

When you model "chaos", you are deciding on the representation. 
You are using mathematics, a formalized system optimized to be used by humans. 
And you use math/your intuition to formalize "the perceiving".

----

Taking a step back I realize that GAI "in the wild" will most likely be "subjected" to the same human influences. But I haven't thought about it a lot and I'm curious about your take on this.

 

I think of “externality” as roughly equivalent to “you hurt someone and you didn't have to pay for it.” If a workplace is neglectful of safety, and the worker gets hurt, and the employer doesn't have to pay, that seems like an externality?

Ban drinks on the floor and have gatekeepers guarding the exits to the bar. Train them to screen everyone leaving for symptoms of the date-rape drugs. (Are they fast-acting enough that this could be noticeable before they exit?) Announce and post signs so patrons expect this. Maybe train the bartenders too. Also don't give anyone more drinks if they're getting drunk. Maybe also screen everyone exiting the venue, if possible.

I suppose the rapist could still palm an eyedropper and aim it into the straw, but this has got to be a lot more difficult than dropping a pinch of powder over an open cup. Doable, but difficult. Drops on the lid would be suspicious if he misses. Also, thin straws would make this even more difficult.

Ban drinks on the floor, and only serve them in sex-segregated areas. Have bouncers to enforce this. (Bouncers and bartenders could still be male, but in uniform.) A rapist would have to have a female accomplice, successfully pose as a female (difficult for most men), steal a uniform, or actually be a female (this has got to be rare), and then still somehow get their mark out of the female area against her will without the bouncers noticing. This wouldn't stop the boys from getting raped, but I assume this is less of a problem?

Have you considered banning the drinks altogether?

I think that should be possible with techniques like reinforcement learning from human feedback, for a given precise specification of “ideologically neutral”.

What kind of specification do you have in mind? Is it like a set of guidelines for the human providing feedback on how to do it in an ideologically neutral way?

You’ll of course have a hard time convincing everyone that your specification is itself ideologically neutral, but projects like Wikipedia give me hope that we can achieve a reasonable amount of consensus.

I'm less optimistic about this, given that complaints about Wikipedia's left-wing bias seem common and credible to me.

Thanks for this very clear explanation of your thinking. A couple of followups if you don't mind.

Unfortunately, I think that sort of analysis generally suggests that most of these sorts of training setups would end up giving you a deceptive model, or at least not the intended model.

Suppose the intended model is to predict H's estimate at convergence, and the actual model is predicting H's estimate at round N for some fixed N larger than any convergence time in the training set. Would you call this an "inner alignment failure", an "outer alignment failure", or something else (not an alignment failure)?

Putting these theoretical/conceptual questions aside, the reason I started thinking about this is from considering the following scenario. Suppose some humans are faced with a time-sensitive and highly consequential decision, for example, whether to join or support some proposed AI-based governance system (analogous to the 1690 "liberal democracy" question), or a hostile superintelligence is trying to extort all or most of their resources and they have to decide how to respond. It seems that convergence on such questions might take orders of magnitude more time than what M was trained on. What do you think would actually happen if the humans asked their AI advisor to help with a decision like this? (What are some outcomes you think are plausible?)

What's your general thinking about this kind of AI risk (i.e., where an astronomical amount of potential value is lost because human-AI systems fail to make the right decisions in high-stakes situations that are eventuated by the advent of transformative AI)? Is this something you worry about as an alignment researcher, or do you (for example) think it's orthogonal to alignment and should be studied in another branch of AI safety / AI risk?

What if it's not transitive?

Wouldn't that kinda throw a wrench in the idea of modeling it as continuous scale?

a Senior Software Engineer at Google makes somewhere around $250k/year

This number from Glassdoor omits the $128k/yr average stock compensation, and while you might round startup equity to zero Google stock seems reliably valuable. levels.fyi is a better source for tech total compensation.