LessWrong 2.0 Reader

Humans are normally agentic (they can also quite often be selfish, power-seeking, deceitful, bad-tempered, untrustworthy, and/or generally unaligned). Standard unsupervised LLM foundation model training teaches LLMs how to emulated humans as text-generation processes. This will inevitably include modelling many aspects of human psychology, including the agentic ones, and the unsavory ones. So LLMs have trained-in agentic behavior before any RL is applied, or even if you use entirely non-RL means to attempt to make them helpful/honest/harmless (e.g. how Google did this to LaMDA). They are trained on a great many examples of deceit, and every other kind of nasty human behavior, so RL is not the primary source of the problem.

The alignment problem is about producing something that we significantly more certain is aligned than a typical randomly-selected human. Handing a randomly-selected human absolute power over our society is unlikely to end well. What we need to train is a selfless altruist who (platonically or parentally) loves all humanity. We need to create a saint or an angel.

RE timeline: no problem, I am happy to wait until September or after, giving me even more time to look into it.

I am going to put a feeler out to see if there is any interest in the Lesswrong/EA community in Sydney, I would not go ahead unless 30 people plan to participate.

RE base rate: I can run blood tests on [say 10+] omnivores in the community, assuming Lesswrong/EA community to be homogenous in other aspects aside from the tested diet, [we do have 10-20x the autism/neurodivergent rate compared to general population]. That should establish Lesswrong/EA community-specific base rate [hopefully].

RE protocol: the pool should be anyone from Sydney Lesswrong/EA community I can talk into participating, divided by diet.

The result of [how many self-described vegans in the Lesswrong/EA community in Sydney are deficient in iron, Vit B12, Vit D as found in blood tests [according to so and so metric]? and would supplementation help?] should speak for itself.

RE "Real study": yeah, it's nice for me to talk a big game but personally, I am very pessimistic [<30%] that any test is going to be done at all. If nothing else, at least it should raise the problem to the local community consciousness.

Personally, I am a big fan [did I say big? I mean huge, humongous] of your past works and found them to be enlightening.  I believe your numbers and my only complaint of your in-office test adventure was that you did not manage to establish the benefit of supplements. Which, if the stars aligned, I hope to remedy this round.

I am an omnivore, although I greatly admire those who go on vegan diets for animal suffering. I am only in this to find out what is true and if there's anything we can do about it. My stake is that I care deeply about the community, many of them vegans. If doing vegan causes easily fixable deficiency I would really want to know.

I wonder how the following behavioral patterns fit into Shard Theory

• Many mammalian species have strong default aversion to young of their own species. They (including females) deliberately avoid contact with the young and can even be infanticidal. Physiological events associated with pregnancy (mostly hormones) rewires the mother's brain such that when she gives birth, she immediately takes care of the young, grooms them etc., something she has never done before. Do you think this can be explained by the rewiring of her reward circuit such that she finds simple actions associated with the pups highly rewarding and then bootstraps to learning complex behaviors from that?
• Salt-starved rats develop an appetite for salt and are drawn to stimuli predictive of extremely salty water, even though on all previous occasions they found it extremely aversive, which caused them to develop conditioned fear response to the cue predictive of salty water. (see Steve's post on this experiment [LW · GW]) 

The whole thing doesn’t get less weird the more I think about it, it gets weirder. I don’t understand how one can have all these positions at once. If that’s our best hope for survival I don’t see much hope at all, and relatively I see nothing that would make me hopeful enough to not attempt pivotal acts.

As someone who read Andrew Critch's post and was pleasantly surprised to find Andrew Critch expressing a lot of views similar to mine (though in relation to pivotal acts mine are stronger), I can perhaps put out some possible reasons (of course it is entirely possible that he believes what he believes for entirely different reasons):

1. Going into conflict with the rest of humanity is a bad idea. Cooperation is not only nicer but yields better results. This applies both to near-term diplomacy and to pivotal acts.
2. Pivotal acts are not only a bad idea for political reasons (you turn the rest of humanity against you) but are also very likely to fail technically. Creating an AI that can pull off a pivotal act and not destroy humanity is a lot harder than you think. To give a simple example based on a recent lesswrong post [LW · GW], consider on a gears level how you would program an AI to want to turn itself off, without giving it training examples. It's not as easy as someone might naively think. That also wouldn't, IMO, get you something that's safe, but I'm just presenting it as a vastly easier example compared to actually doing a pivotal act and not killing everyone.
3. The relative difficulty difference between creating a pivotal-act-capable AI and an actually-aligned-to-human-values AI, on the other hand, is at least a lot lower than people think and likely in the opposite direction. My view on this relates to consequentialism - which is NOT utility functions, as commonly misunderstood on lesswrong. By consequentialism I mean caring about the outcome unconditionally, instead of depending on some reason or context. Consequentialism is incompatible with alignment and corrigibility; utility functions on the other hand are fine, and do not implty consequentialism. Consequentialist assumptions prevalent in the rationalist community have, in my view, made alignment seem a lot more impossible than it really is. My impression of Eliezer is that non-consequentialism isn't on his mental map at all; when he writes about deontology, for instance, it seems like he is imagining it as an abstraction rooted in consequentialism, and not as something actually non-consequentialist.
4. I also think agency is very important to danger levels from AGI, and current approaches have relatively low level of agency which reduces the danger. Yes, people are trying to make AIs more agentic, fortunately getting high levels of agency is hard. No, I'm not particularly impressed by the Minecraft example in the post. 
5. An AGI that can meaningfully help us create aligned AI doesn't need to be agentic to do so, so getting one to help us create alignment is not in fact "alignment complete"
6. Unfortunately, strongly self-modifying AI, such as bootstrap AI, is very likely to become strongly agentic because being more agentic is instrumentally valuable to an existing weakly agentic entity. 

Taking these reasons together, attempting a pivotal act is a bad idea because:

• you are likely to be using a bootstrap AI, which is likely to become strongly agentic through a snowball effect from a perhaps unintended weakly agentic goal, and optimize against you when you try to get it to do a pivotal act safely; I think it is likely to be possible to prevent this snowballing (though I don't know how exactly) but since at least some pivotal act advocates don't seem to consider agency important they likely won't address this
• if you somehow overcome that hurdle it will still be difficult to get it to do a pivotal act safely, and you will likely fail. Probably you fail by not doing the pivotal act, but the  probability of not killing everyone conditional on pulling off the conditional act is still not good
• If you overcome that hurdle, you still end up at war with the rest of humanity from doing the pivotal act (if you need to keep your candidate acts hidden because they are "outside the Overton window" don't kid yourself about how they are likely to be received by the general public) and you wind up making things worse 
• Also, the very plan to cause a pivotal act in the first place intensifies races and poisons the prospects for alignment cooperation (e.g. if i had any dual use alignment/bootstrap insights I would be reluctant to share them even privately due to concern that MIRI might get them and attempt a pivotal act)
• And all this trouble wasn't all that urgent because existing AIs aren't that dangerous due to lack of strong self-modification making them not highly agentic, though that could change swiftly of course if such modification becomes available
• finally, you could have just aligned the AI for not much more and perhaps less trouble. Unlike a pivotal act, you don't need to get alignment-to-values correct the first time, as long as the AI is sufficiently corrigible/self-correcting; you do need, of course, to get the corrigibility/self correcting aspect sufficiently correct the first time, but this is plausibly a simpler and easier target than doing a pivotal act without killing everyone.

This gap will only widen over time; China is failing to develop a domestic semiconductor industry, despite massive efforts to do so, and is increasingly cut off from international semiconductor supply chains.

I would say this is a falsehood 

The US export ban on Controlled GPUs has really made china push for local semiconductor manufacturing way, and accelerate their projects, they dont have 5nm TSMC quality wafers, fine, but they're developing the full stack. 

tphuang on Twitter: "Semiconductor export from Korea down 41% in April Falling SSD price in China due to YMTC produced 2TB SSDs widely available &lt; 1000K RMB https://t.co/Oh4alI8niC Samsung can't compete w/ YMTC production cost, now cutting production after resisting for months. as YMTC restart…" / Twitter

Yicai Global 第一财经 on Twitter: "China’s TanKeBlue to Supply SiC Wafers, Boules to German Chip Giant Infineon @Infineon https://t.co/GRgPjKsaCr" / Twitter

I mean if this was a "The AGi Race Between the US and Russia doesnt exist" okay fine, but Seeing how more than half the papers that land in ArXiv have chinese authors in them, plus the whole China does 90% of electronic manufacturing in the world. I dont understand how you come to the conclusion that china is hopelessly dead in the water. 


The day the US export ban on GPU happened, okay, most of us really wondered, but seeing how they're operating 6 months to a year afterwards, it just obvious that they will be able to make it happen. 

 

It may be worth doing, but I expect this dynamic to apply.

Considering coming if I can wake up in time and if nobody has a problem with me shamelessly exploiting you for free labor (project is setting up an OpenBSD darkweb Mastodon server on a ThinkPad T440P.) Due to limited storage space and limited uptime ability the "concept" I am aiming at is fedspace protocol as a method of temporarily deploying a way for people to exchange CWTCH handles for subsequent communication in an adversarial environment. Might like to record the setup process on cell phone camera so others have a tutorial of sorts or just to keep track of ground covered. Is any of that sensible/acceptable?

From David Chapman's "Better without AI", section "Fear AI Power":

"The AI risks literature generally takes for granted that superintelligence will produce superpowers, but which powers and how this would work is rarely examined, and never in detail. One explanation given is that we are more intelligent than chimpanzees, and that is why we are more powerful, in ways chimpanzees cannot begin to imagine. Then, the reasoning goes, something more intelligent than us would be unimaginably more powerful again. But for hundreds of thousands of years humans were not more powerful than chimpanzees. Significantly empowering technologies only began to accumulate a few thousand years ago, apparently due to cultural evolution rather than increases in innate intelligence. The dramatic increases in human power beginning with the industrial revolution were almost certainly not due to increases in innate intelligence. What role intelligence plays in science and technology development is mainly unknown;"

every macroscopic state would have a defined designation as conscious or not [...] there would be no undefined states, per se

But the actual states of things are microscopic. And from a microscopic perspective, macroscopic states are vague. They have edge cases, they have sorites problems. 

For crowds, or clouds, this doesn't matter. That these are vague concepts does not create a philosophical crisis, because we have no reason to believe that there is an "essence of crowd" or "essence of cloud", that is either present or not present, in every possible state of affairs. 

Consciousness is different - it is definitely, actually there. As such, its relationship to the microphysical reality cannot be vague or conventional in nature. The relationship has to be exact. 

The conscious system would be the one where, to take a common example, information has looped around and back to the same neuron, regardless of how exactly it did.

So by my criteria, the question is whether you can define informational states, and circulation of information, in such a way that from a microphysical perspective, there is never any ambiguity about whether they occurred. For all possible microphysical states, you should be able to say whether or not a given "informational state" is present. I'm not saying that every microphysical detail must contribute to consciousness; but if consciousness is to be identified with informational states, informational states have to have a fully objective existence. 

I see. Referring back to your argument was more an illustration of existence for this motivation. If a society forms around the motivation, at any one time in the billion years, and selects for intelligence to enable nontrivial long term institution design, that seems sufficient to escape stasis.