LessWrong 2.0 Reader

I feel like the difference between the Alpha and Beta example and my example mediate through your examples having basically no control of Beta's data at all, and my examples having far more control over what data is learned by the AI.

I think the key crux is whether we have much more control over AI data sources than evolution.

If I agreed with you that we would have essentially no control on what data the AI has, I'd be a lot more worried, but I don't think this is true, and I expect future AIs, including AGIs, to be a lot more built than grown, and for a lot of their data to be very carefully controlled via synthetic data, for simple capabilities reasons, but this can also be used for alignment strategies.

I think another disagreement is I basically don't buy the evolution analogy for DL, and I think there are some deep disanalogies (the big one for now is again how much more control over data sources than evolution, and this is only set to increase with synthetic data).

So I basically don't expect this to happen:

I instead strongly expect that the story would just repeat. The training process (or whatever process spits out the AGI) would end up creating some extremely specific conditions in which the AGI is learning the values. Its values would then necessarily be some complicated functions over weird mixes of the abstractions-natural-to-the-dataset-it's-trained-on, with their specifics being highly contingent on some invisible-to-us details of that process.

Pretty much all of your examples rely on the Alpha being unable to control the data learnt by Beta, and if this isn't the case, your examples break down.

Somehow I missed that bit.

That makes the situation better, but there’s still some issue. The refund is not earning interest, but you liabilities are.

Take the situation with owing $25 million. Say that there’s a one year time between the tax being assessed and your asset going to $0 (at which time you claim the refund). In this time the $25 million loan you took is accruing interest. Let’s say it does so at a 4% rate per year, when you get your $25 million refund you therefore have $26 million in loans.

So you still end up $1 million in debt due to “gains” that you were never able to realize.

@Ben Pace [LW · GW] I would like a vote here on what percentage chance we think that an omnicient reviewer would say this narrative is true. The display it on an axis, probably with dots (anonymous) for each person. eg like this. 

This whole post you keep referencing "the Hegelian dialectic" as some sort of thing central to the Village, without ever stating what you think it is; could you elaborate?

I'm not an expert in this area by any means but I do not have the impression that the Village has even a plurality of Hegelians, let alone being centered around them, and the way you talk about the Hegelian dialectic does not seem to be remotely the same usage that you see with actual Hegelians. Honestly I expect actual Hegelians to be rather less upset about the River's differences from them, because, to my understanding, Hegel's whole thing is he believes in a sort of teleology of history that guarantees the victory of human freedom?

strong disagree. i would be highly surprised if there were multiple essentially different algorithms to achieve general intelligence*. 

I also agree with the Daniel Murfet's quote. There is a difference between a disjunction before you see the data and a disjunction after you see the data. I agree AI development is disjunctive before you see the data - but in hindsight all the things that work are really minor variants on a single thing that works. 

*of course "essentially different" is doing a lot of work here. some of the conceptual foundations of intelligence haven't been worked out enough (or Vanessa has and I don't understand it yet) for me to make a formal statement here. 

You're right. I'll rerun the analysis and include 2x2 games as well.

I like this exchange and the clarifications on both sides. I'll add my response:

You're right that coherence arguments work by assuming a goal is about the future. But preferences over a single future timeslice is too specific, the arguments still work if it's multiple timeslices, or an integral over time, or larger time periods that are still in the future. The argument starts breaking down only when it has strong preferences over immediate actions, and those preferences are stronger than any preferences over the future-that-is-causally-downstream-from-those-actions. But even then it's could be reasonable to model the system as a coherent agent during the times when its actions aren't determined by near-term constraints, when longer-term goals dominate.

(a relevant part of Eliezer's recent thread [LW · GW] is "then probably one of those pieces runs over enough of the world-model (or some piece of reality causally downstream of enough of the world-model) that It can always do a little better by expending one more erg of energy.", but it should be read in context)

Another missing piece here might be: The whole point of building an intelligent agent is that you know more about the future-outcomes you want than you do about the process to get there. This is the thing that makes agents useful and valuable. And it's the main thing that separates agents from most other computer programs.

On the other hand, it does look like the anti-corrigibility results can be overcome by sometimes having strong preferences over intermediate times (i.e. over particular ways the world should go) rather than final-outcomes. This does seem important in terms of alignment solutions. And it takes some steam out of the arguments that go "coherent therefore incorrigible" (or it at least should add some caveats). But this only helps us if we have a lot of control over the preferences&constraints of the agent, and it has a couple of stability properties.

Could you dig into why you think it's great inter work?

I feel that human intelligence is not the gold standard of general intelligence; rather, I've begun thinking of it as the *minimum viable general intelligence*.
In evolutionary timescales, virtually no time has elapsed since hominids began trading, utilizing complex symbolic thinking, making art, hunting large animals etc, and here we are, a blip later in high technology. The moment we reached minimum viable general intelligence, we started accelerating to dominate our environment on a global scale, despite increases in intelligence that are actually relatively megre within that time: evolution acts over much longer timescales and can't keep pace with our environment, which we're modifying at an ever-increasing rate.
Moravec's paradox suggests we are in fact highly adapted to the task of interacting with the physical world-as basically all animals are-and we have some half-baked logical thinking systems tacked on to this base.

Ideally, AI characters would get rights as soon as they could pass the Turing test. In the actual reality, we all know how well that will go.