LessWrong 2.0 Reader

Resonating with you here!  Yes, I think autonomous corporations (and other organisations) would result in society-wide extraction, destabilisation and totalitarianism.

I'm not that concerned with lobbyists ruining the deliberative proceedings. I think you're underestimating normal people a bit. They have state power to shut down annoying and undesired feedback if they wish. I also think the assembly will tend to trust their own advisors, whom they hired themselves, over outside self-proclaimed expert lobbyists. 

My bigger concern is with corruption and bribery. Because we're dealing with very normal people, we also ought to expect normal criminal behavior. We ought to expect assembly members getting arrested from time to time, and doing all the normal things we expect from 500 random people. 

I think bribery is a sufficiently high concern that a police force should constantly operate to perform sting operations and monitor elicit behavior from assembly members. IMO, this should already be happening with elected officials too. 

Another big concern is whether a purely lottocratic assembly would self-regulate its own corruption. It has some interest to, in that the lottocrats help their future selves, after their term has ended, by creating future rules that would regulate corruption. Terril Bouricus attempts to create a system where layers on layers of assemblies check and re-check the work of other assemblies to mitigate corruption concerns. 

I can't easily conclude whether election or sortition would be better at corruption mitigation. With elections, opposition parties have an incentive to investigate their enemies to root out corruption. HOWEVER, the same opposition parties have an incentive to lie about the results of investigations, leading to an environment of fake news, where voters cannot distinguish between a political attack and actual corruption. In the American context, bribery is about already legalized with campaign donations. 

Sortition could possibly lead to a ridiculous scenario:

Imagine the public is outraged at the insane level of corruption of the sortition-assembly. However as a new assembly is formed by lottery, these anti-corruption sentiments are suddenly rotated into office. The members of the public hate corruption, as does this new assembly! The question is, would the members of the assembly be able to do the Machiavellian about-face and suddenly embrace corruption? I have a hard time believing they would, though I have doubts. In my opinion, normal people being utterly normal, would rather do the easy thing and yes, go ahead and regulate the corruption while enjoying their government salary. Getting to serve in office is already a win-win, why not win and also be declared heroes? Alternatively they can "Go Breaking Bad", embrace corruption and pilfer the state coffers. They can win big (for now) but will become despised. What do you think normal people would do? High risk high reward, or low risk medium reward? I don't think going "Breaking Bad" is the best of ideas. Elected politicians use their offices to protect themselves from legal challenge. Obvious example, Donald Trump using the presidency to overcome his legal problems. He's obviously not the first politician to cling to office in order to protect themselves. Lottocrats can't do the same. Lottocrats soon lose their powers and become vulnerable. 

I mostly agree on current margins — the more I mistrust a lab, the more I like transparency.

I observe that unilateral transparency is unnecessary on the view-from-the-inside if you know you're a reliably responsible lab. And some forms of transparency are costly. So the more responsible a lab seems, the more sympathetic we should be to it saying "we thought really hard and decided more unilateral transparency wouldn't be optimific." (For some forms of transparency.)

let's not kill the messenger, lest we run out of messengers.

Unfortunately we're a fair way into this process, not because of downvotes^[1] but rather because the comments are often dominated by uncharitable interpretations that I can't productively engage with.^[2]. I've had researchers and policy people tell me that reading the discussion convinced them that engaging when their work was discussed on LessWrong wasn't worth the trouble.

I'm still here, sad that I can't recommend it to many others, and wondering whether I'll regret this comment too.

Thank you for the response <3

"...I would like to prove to the Court Philosopher that I'm right and he's wrong."

This part of the story tickles me more, reading it a second time.

I like to write stories that mean different things to different people ...this story isn't a puzzle at all. It is a joke about D&D-style alignment systems.

And it kinda resonates with this bit. In both cases there's a certain flexibility. The flexibility itself is unexpected, but reasonable safe... which is often a formula for comedy? It is funny to see the flexibility in Phil as he "goes social", and also funny to see it in you as you "go authorial" :-)

How could one control AI without access to the hardware/software? What would stop one with access to the hardware/software from controlling AI?

One would gain control by renting access to the model, i.e., the same way you can control what an instance of ChatGPT currently does. Here, I am referring to practical control over the actual behavior of the AI, when determining what the AI does, such as what tasks it performs, how it is fine-tuned, or what inputs are fed into the model.

This is not too dissimilar from the high level of practical control one can exercise over, for example, an AWS server that they rent. While Amazon may host these servers, and thereby have the final say over what happens to the computer in the case of a conflict, the company is nonetheless inherently dependent on customer revenue, implying that they cannot feasibly use all their servers privately for their own internal purposes. As a consequence of this practical constraint, Amazon rents these servers out to the public, and they do not substantially limit user control over AWS servers, providing for substantial discretion to end-users over what software is ultimately implemented.

In the future, these controls could also be determined by contracts and law, analogously to how one has control over their own bank account, despite the bank providing the service and hosting one's account. Then, even in the case of a conflict, the entity that merely hosts an AI may not have practical control over what happens, as they may have legal obligations to their customers that they cannot breach without incurring enormous costs to themselves. The AIs themselves may resist such a breach as well.

In practice, I agree these distinctions may be hard to recognize. There may be a case in which we thought that control over AI was decentralized, but in fact, power over the AIs was more concentrated or unified than we believed, as a consequence of centralization over the development or the provision of AI services. Indeed, perhaps real control was always in the hands of the government all along, as they could always choose to pass a law to nationalize AI, and take control away from the companies.

Nonetheless, these cases seem adequately described as a mistake in our perception of who was "really in control" rather than an error in the framework I provided, which was mostly an attempt to offer careful distinctions, rather than to predict how the future will go.

If one actor—such as OpenAI—can feasibly get away with seizing practical control over all the AIs they host without incurring high costs to the continuity of their business through loss of customers, then this indeed may surprise someone who assumed that OpenAI was operating under different constraints. However, this scenario still fits nicely within the framework as I've provided, as it merely describes a case in which one was mistaken about the true degree of concentration along one axis, rather than one of my concepts intrinsically fitting reality poorly.

I'd find the agree/disagree dimension much more useful if we split out "x people agree, y disagree" - as the EA Forum does - rather than showing the sum of weighted votes (and total number on hover).

I'd also encourage people to use the other reactions more heavily, including on substrings of a comment, but there's value in the anonymous dis/agree counts too.

I do not really understand your framing of these three "dimensions". The way I see it, they form a dependency chain. If either of the first two are concentrated, they can easily cut off access during takeoff (and I would expect this). If both of the first two are diffuse, the third will necessarily also be diffuse.

How could one control AI without access to the hardware/software? What would stop one with access to the hardware/software from controlling AI?

Taxes enforced by whom?

Well, that's where the "safe" part comes in, isn't it?

I think a fair number of people would say that ASI/AGI can't be called "safe" if it's willing to wage war to physically take over the world on behalf of its owners, or to go around breaking laws all the time, or to thwart whatever institutions are supposed to make and enforce the laws. I'm pretty sure that even OpenAI's (present) "safety" department would have an issue if ChatGPT started saying stuff like "Sam Altman is Eternal Tax-Exempt God-King".

Personally, I go further than that. I'm not sure about "basic" AGI, but I'm pretty confident that very powerful ASI, the kind that would be capable of really total world domination, can't be called "safe" if it leaves really decisive power over anything in the hands of humans, individually or collectively, directly or via institutions. To be safe, it has to enforce its own ideas about how things should go. Otherwise the humans it empowers are probably going to send things south irretrievably fairly soon, and if they don't do so very soon they always still could, and you can't call that safe.

Yeah, that means you get exactly one chance to get "its own ideas" right, and no, I don't think that success is likely. I don't think it's technically likely to be able to "align" it to any particular set of values. I also don't think people or insitutions would make good choices about what values to give it even if they could. AND I don't think anybody can prevent it from getting built for very long. I put more hope in it being survivably unsafe (maybe because it just doesn't usually happen to care to do anything to/with humans), or on intelligence just not being that powerful, or whatever. Or even in it just luckily happening to at least do something less boring or annoying than paperclipping the universe or mass torture or whatever.

It is not always an expression of selfish motives when people take a stance against genocide. I would even go as far as saying that, in the majority of cases, people genuinely have non-selfish motives when taking that position. That is, they actually do care, to at least some degree, about the genocide, beyond the fact that signaling their concern helps them fit in with their friend group.

Nonetheless, and this is important: few people are willing to pay substantial selfish costs in order to prevent genocides that are socially distant from them.

The theory I am advancing here does not rest on the idea that people aren't genuine in their desire for faraway strangers to be better off. Rather, my theory is that people generally care little about such strangers, when helping those strangers trades off significantly against objectives that are closer to themselves, their family, friend group, and their own tribe.

Or, put another way, distant strangers usually get little weight in our utility function. Our family, and our own happiness, by contrast, usually get a much larger weight.

The core element of my theory concerns the amount that people care about themselves (and their family, friends, and tribe) versus other people, not whether they care about other people at all.