LessWrong 2.0 Reader

How do you envision access and control of an AI that is robustly and reasonably compliant? And in which way would "human values" be involved? I agree with you that they are part of the solution, but I want to compare my beliefs with yours.

What seems radical depends on where one stands. We each of us stand on our own beliefs, and the further away one looks, the more the beliefs over there differ from one's own. Look sufficiently far and everything you see will seem extreme and radical. Hence the fallacy that truth lies between extremes, instead of recognising the tautology that one's own beliefs always lie between those that are extremely different.

Let me put my attitudes in practical terms: I don't kick dogs, but I have destroyed a wasp's nest in my garage, and I don't donate to animal charities. (I don't donate to many other charities either, but there have been a few.) Let those who think saving the shrimps is worthwhile do so, but I do not travel on that path

The Bills are 0/4 in recent playoff games against the Chiefs & also 0/4 in Super Bowls as a franchise; this is mostly a coincidence

But it's difficult to think about the former without also thinking about the latter. Unless you don't follow NFL.

From a paranormal perspective mental connections like this suggest causal connections such as a curse on the team. Supposedly a superstition can become a rational hyperstition if enough people believe.

From a common sense perspective the Bill's recent 0-4 record is concerning. The people saying no no no b/c p=0.125 or the games were so close -- they seem to be over correcting. The prior is not so concentrated around 50%. Also {the Chiefs are a superior playoff team} is supported by other pieces of evidence not just their 4-0 record against the Bills.

So the Bills are under a possible double hex -- a hyperstitious hex & also a purely rational doubt feedback induced hex

If the Bills win the Super Bowl next year I'm going to conclude that hyperstition is over rated. (Personally I don't expect them to get past the 1st round).

I have no daydreams about quila, and others of like mind, not existing. Not even about Ziz.

How do we know money wouldn’t work? Surely most of the reasons people don’t have more children could be solved with more resources (eg surrogacy, childcare, private school fees, a larger house).

Apropos of nothing, I'm reminded of the "<antthinking>" tags originally observed in Sonnet 3.5's system prompt, and this section of Dario's recent essay (bolding mine):

In 2024, the idea of using reinforcement learning (RL) to train models to generate chains of thought has become a new focus of scaling. Anthropic, DeepSeek, and many other companies (perhaps most notably OpenAI who released their o1-preview model in September) have found that this training greatly increases performance on certain select, objectively measurable tasks like math, coding competitions, and on reasoning that resembles these tasks.

For anecdata: id be really jazzed about 3 or 4, 5 might be a little crazy but somewhat open to that or more.

Ladies

This post would've been better if you tabooed the word "emergence" which does a lot of heavy lifting here. You seem to be thinking in the right direction, but this kind of curiosity stopper prevents you from getting an actual insight.

Thanks for the mention.

Here's how I'd frame it: I don't think it's a good idea to leave the entire future up to the interpretation of our first AGI(s). They could interpret our attempted alignment very differently than we hoped, in in-retrospect-sensible ways, or do something like "going crazy" from prompt injections or strange chains of thought leading to ill-considered beliefs that get control over their functional goals.

It seems like the core goal should be to follow instructions or take correction - corrigibility as a singular target [LW · GW] (or at least prime target).  It seems noticeably safer to use Intent alignment as a stepping-stone to value alignment [LW · GW]. 

Of course, leaving humans in charge of AGI/ASI even for a little while sounds pretty scary too, so I don't know.

I didn't express this clearly, but yea I meant no pretraining on human text at all, and also nothing computer-generated which "uses human mathematical ideas" (beyond what is in base ZFC), but I'd probably allow something like the synthetic data generation used for AlphaGeometry (Fig. 3) except in base ZFC and giving away very little human math inside the deduction engine. I agree this would be very crazy to see. The version with pretraining on non-mathy text is also interesting and would still be totally crazy to see. I agree it would probably imply your "come up with interesting math concepts". But I wouldn't be surprised if like $>20\%$ of the people on LW who think A[G/S]I happens in like $2-3$ years thought that my thing could totally happen in 2025 if the labs were aiming for it (though they might not expect the labs to aim for it), with your things plausibly happening later. E.g. maybe such a person would think "AlphaProof is already mostly RL/search and one could replicate its performance soon without human data, and anyway, AlphaGeometry already pretty much did this for geometry (and AlphaZero did it for chess)" and "some RL+search+self-play thing could get to solving major open problems in math in 2 years, and plausibly at that point human data isn't so critical, and IMO problems are easier than major open problems, so plausibly some such thing gets to IMO problems in 1 year". But also idk maybe this doesn't hang together enough for such people to exist. I wonder if one can use this kind of idea to get a different operationalization with parties interested in taking each side though. Like, maybe whether such a system would prove Cantor's theorem (stated in base ZFC) (imo this would still be pretty crazy to see)? Or whether such a system would get to IMO combos relying moderately less on human data?