LessWrong 2.0 Reader

Perhaps half of the value of misaligned AI control is from acausal trade and half from the AI itself being valuable.

Why do you think these values are positive? I've been pointing out in various comments, and I see that Daniel Kokotajlo also pointed out in 2018 [LW(p) · GW(p)] that these values could well be negative. I'm very uncertain but my own best guess is that the expected value of misaligned AI controlling the universe is negative, in part because I put some weight on suffering-focused ethics.

My speculation for Omni-Grok in particular is that in settings like MNIST you already have two of the ingredients for grokking (that there are both memorising and generalising solutions, and that the generalising solution is more efficient), and then having large parameter norms at initialisation provides the third ingredient (generalising solutions are learned more slowly), for some reason I still don't know.

Higher weight norm means lower effective learning rate with Adam, no? In that paper they used a constant learning rate across weight norms, but Adam tries to normalize the gradients to be of size 1 per paramter, regardless of the size of the weights. So the weights change more slowly with larger initializations (especially since they constrain the weights to be of fixed norm by projecting after the Adam step). 

If you use ublock (or adblock, or adguard, or anything else that uses EasyList syntax), you can add a custom rule

lesswrong.com##.NamesAttachedReactionsCommentBottom-footerReactionsRow
lesswrong.com##.InlineReactHoverableHighlight-highlight:remove-class(InlineReactHoverableHighlight-highlight)

which will remove the reaction section underneath comments and the highlights corresponding to those reactions.

The former of these you can also do through the element picker.

I agree with RL agents being misaligned by default, even more so for the non-imitation-learned ones. I mean, even LLMs trained on human-generated data are misaligned by default, regardless of what definition of 'alignment' is being used. But even with misalignment by default, I'm just less convinced that their capabilities would grow fast enough to be able to cause an existential catastrophe in the near-term, if we use LLM capability improvement trends as a reference. 

Bowler's comment on Wallace is that his theory was not worked out to the extent that Darwin's was, and besides I recall that he was a theistic evolutionist. Even with Wallace, there was still a plethora of non-Darwinian evolutionary theories before and after Darwin, and without the force of Darwin's version, it's not likely or necessary that Darwinism wins out. 

But Wallace’s version of the theory was not the same as Darwin’s, and he had very different ideas about its implications. And since Wallace conceived his theory in 1858, any equivalent to Darwin’s 1859 Origin of Species would have appeared years later.

Also 

Natural selection, however, was by no means an inevitable expression of mid-nineteenth-century thought, and Darwin was unique in having just the right combination of interests to appreciate all of its key components. No one else, certainly not Wallace, could have articulated the idea in the same way and promoted it to the world so effectively.

And he points out that minus Darwin, nobody would have paid as much attention to Wallace. 

The powerful case for transmutation mounted in the Origin of Species prompted everyone to take the subject seriously and begin to think more constructively about how the process might work. Without the Origin, few would have paid much attention to Wallace’s ideas (which were in many respects much less radical than Darwin’s anyway). Evolutionism would have developed more gradually in the course of the 1860s and ’70s, with Lamarckism being explored as the best available explanation of adaptive evolution. Theories in which adaptation was not seen as central to the evolutionary process would have sustained an evolutionary program that did not enquire so deeply into the actual mechanism of change, concentrating instead on reconstructing the overall history of life on earth from fossil and other evidence. Only toward the end of the century, when interest began to focus on the topic of heredity (largely as a result of social concerns), would the fragility of the non-Darwinian ideas be exposed, paving the way for the selection theory to emerge at last.

Bowler also points out that Wallace didn't really form the connection between both natural and artificial selection. 

Is this a translation of Bostrom's article? If yes, could you please make this more explicit (maybe as a first paragraph in the text, in English), and include a link to the original?

If something is both a vanguard and limited, then it seemingly can't stay a vanguard for long. I see a few different scenarios going forward:

1. We pause AI development while LLMs are still the vanguard.
2. The data limitation is overcome with something like IDA or Debate.
3. LLMs are overtaken by another AI technology, perhaps based on RL.

In terms of relative safety, it's probably 1 > 2 > 3. Given that 2 might not happen in time, might not be safe if it does, or might still be ultimately outcompeted by something else like RL, I'm not getting very optimistic about AI safety just yet.

Hey Jacques, sure, I'd be happy to chat!  

I guess in the average case, the contrarian's conclusion is wrong, but it is also a reminder that the mainstream case is not communicated clearly, and often exaggerated or supported by invalid arguments. For example:

• it's not that "dieting doesn't work", but that people naively assume that dieting is simple and effective ("if you just stop eating chocolate and start exercising for one hour every day, you will certainly lose weight", haha nope), even when the actual weight-loss research shows otherwise;
• it's not that "medicine doesn't improve health", but while some parts of medicine are very useful, other parts may be neutral or even harmful, and we often see that throwing more money at medicine does not actually improve the outcomes;
• it's not that "education doesn't work", but if you filter your students by intelligence and hard work, of course they will have better outcomes in life regardless of how good is your teaching, so the impact of education is probably vastly overestimated, and this also explains why so many pedagogical experiments succeed at a pilot project (when you try them with a small group of smart and motivated students) and then fail in mainstream education (when you try the same thing with average or below-average students);
• it's not that "opening the borders completely is a good idea", but a lot of potential value is lost by closing the borders for people who are neither fanatics nor criminals and could easily integrate to the new society.

There is also an opposite bad extreme to contrarians, the various "I fucking love science... although I do not understand it... but I enjoy attacking people on social networks who seem to disagree with the scientific consensus as I understand it" people. The ones who are sure that the professor or the doctor is always right, and that the latest educational fad is always correct.

Yeah, I'm unsure if I can tell any 'pivotal story' very easily (e.g. I'd still be pretty skeptical of enumerative interp even with GPT-5-MAIA). But I do think, intuitively, GPT-5-MAIA might e.g. make 'catching AIs red-handed' using methods like in this comment [LW(p) · GW(p)] significantly easier/cheaper/more scalable.