LessWrong 2.0 Reader

I think this is missing a major piece of the self-play scaling paradigm: much of the point of a model like o1 is not to deploy it, but to generate training data for the next model. Every problem that an o1 solves is now a training data point for an o3 (eg. any o1 session which finally stumbles into the right answer can be refined to drop the dead ends and produce a clean transcript to train a more refined intuition).

I am actually mildly surprised OA has bothered to deploy o1-pro at all, instead of keeping it private and investing the compute into more bootstrapping of o3 training etc. (This is apparently what happened with Anthropic and Claude-3.6-opus - it didn't 'fail', they just chose to keep it private and distill it down into a small cheap but weirdly smart Claude-3.6-sonnet.) If you're wondering why OAers are suddenly weirdly, almost euphorically optimistic, watching the improvement from the original 4o model to o3 (and wherever it is now!) may be why. It's like watching the AlphaGo Elo curves: it just keeps going up... and up... and up... And then you get to have your cake and eat it too: the final AlphaGo/Zero model is not just superhuman but very cheap to run too. (Just searching out a few plies gets you to superhuman strength; even the forward pass alone is around pro human strength!)

If you look at the relevant scaling curves - may I yet again recommend reading Jones 2021? - the reason for this becomes obvious. Inference-time search is a stimulant drug that juices your score immediately, but asymptotes hard. Quickly, you have to use a smarter model to improve the search itself, instead of doing more. (If simply searching could work so well, chess would've been solved back in the 1960s. It's not hard to search more than the handful of positions a grandmaster human searches per second. If you want a text which reads 'Hello World', a bunch of monkeys on a typewriter may be cost-effective; if you want the full text of Hamlet before all the protons decay, you'd better start cloning Shakespeare.) Fortunately, you have the training data & model you need right at hand to create a smarter model...

Sam Altman (@sama, 2024-12-20):

seemingly somewhat lost in the noise of today:

on many coding tasks, o3-mini will outperform o1 at a massive cost reduction!

i expect this trend to continue, but also that the ability to get marginally more performance for exponentially more money will be really strange

So, it is interesting that you can spend money to improve model performance in some outputs... but you may simply be spending that money to improve the model itself, not just a one-off output.

This means that outsiders may never see the intermediate models (any more than Go players got to see random checkpoints from a third of the way through AlphaZero training). And to the extent that it is true that 'deploying costs 1000x more than now', that is a reason to not deploy at all. Why bother wasting that compute on serving external customers, when you can instead keep training, and distill that back in, and soon have a deployment cost of a superior model which is only 100x, and then 10x, and then 1x, and then <1x...?

Reddit blocks scrapers now aggressively, because it's charging a fortune for access, and The Pile could no longer have been created (Pushshift is down). Reddit is not the worst place to post, but it's also not the best.

What worked for me was pavlovian training by putting my metabolism/blood sugar into lows/withdrawals and then timing the superstimuli habit during those lows for negative reinforcement. 

For example, eating nothing but spaghetti and water for breakfast or both breakfast and lunch, on an empty stomach and not eating anything else as this interferes. In most people this causes a longer weaker rush than a sugar rush for ~2-4 hours or so, then a pretty bad crash (e.g. sufficiently powerful depression to not take this lightly and to keep people nearby, but not nearly enough to be dangerous). If you correctly time the superstimulus activity to start shortly before the crash, the negative reinforcement effect will be sufficiently powerful to put an ugh field [LW · GW] around it after 1-3 successful timings (even 3 is a stretch and I don't endorse this for anything that gets you to put yourself through this a fourth time). 

You abort this crash by eating protein and waiting 20 minutes to an hour for your metabolism to exit the crash by returning to normal. The exiting of the crash provides positive reinforcement, and I don't know whether velocity or acceleration of mood causes the positive/negative reinforcement (just that looking at it in terms of position alone is probably grossly inadequate), so it's important to get the timing right and be conservative with making sure that you change the activity early to make sure that most of the velocity of mood change and acceleration of mood change are during the targeted activty and that minimal velocity/acceleration in the wrong direction happens during the targeted activity.

Using caffeine/stimulants or anything that increases mood or blood sugar during a superstimuli activity is deranged and dangerous; if you're not comfortable with doing it in the helpful taking-control direction I described here (which is quite reasonable and I might be mistaken for playing with this and don't intend to do it or anything like it again now that I've finished my packet of spaghetti and won't buy another), then you're probably not comfortable with doing it in the unhelpful losing-control direction (I'm referring to comfort upon reflection, not in the moment). Our civilization gives kids adderal et al. and tells them to use it during classes/homework and responds to any other use with finger wagging instead of explaining the risks, so no adjacent societal or cultural norms can be trusted let alone left unexamined.

That's a good film! A friend of mine absolutely loves it. 

Do you think the Forbin Project illustrates some aspect of misalignment that isn't covered by this article? 

Wow! That is a hell of a comprehensive writeup.

The bio-determinist child-rearing rule of thumb [but see caveats below!]: Things you do as a parent will have generally small or zero effects on what the kid will be like as an adult—their personality, their intelligence and competence, their mental health, etc.

I found it pretty interesting here and back when I was reading about it that this list does not include happiness. This is part of a larger societal disinterest in happiness. But I do wonder if it might be influenced by parents nontrivially by them seeding children with a life philosophy and a set of cognitive habits about how to think about life.

I also noticed that the data they tracked about how parents treat children included no efforts to determine how much parents actually loved their children, or how much they fought with their children. While most parents, particularly the middle-classers that take part in studies, love their children, how much and whether that prevents ongoing feuds with their children does seem to vary a good bit.

Of course that would be problematic, because how much parents love and feud with their children is also clearly influenced by how much said children are acting like little shits. :)

I doubt any of these would show large effects, I'm just noting their absence.

I used to care about genetics for reasons 2 (what effect do parents have) and 3 (do your adult decisions like attending therapy really matter), back when I planned to use my PhD in cognitive psychology and neuroscience to write about "free will" (really self-determination; do our decisions matter for our outcomes) in ourselves and our society. My thesis was that we have substantial self-determination but also substantial limitations in it; and that liberal American philosophies tend to emphasize the extent to which we don't, while conservative American philosophies emphasize the extent to which we do. Neither is entirely correct, causing strong adherents of either to make no sense and therefore be super irritating to talk to.

But those ambitions ended when I first read Yudkowsky, and decided free will was small potatoes in the face of an onrushing intelligence explosion and alignment crisis. Thus, my ideas related to genetics have never before been published, and probably won't be. Thanks for the excuse to rant.

Again, wow. I'll be referring anyone to this writeup if they express more than the vaguest interest in what we know about genetics.

Reading novels with ancient powerful beings is probably the best direction you have for imagining how status games amongst creatures which are only loosely human look.

Resources being bounded, there will tend to always be larger numbers of smaller objects (given that those objects are stable).

There will be tiers of creatures. (In a society where this is all relevant)

While a romantic relationship skipping multiple tiers wouldn't make sense,  a single tier might.

The rest of this is my imagination :)

Base humans will be F tier, the lowest category while being fully sentient. (I suppose dolphins and similar would get a special G tier).

Basic AGIs (capable of everything a standard human is, plus all the spikey capabilities) and enhanced humans E tier.

Most creatures will be here.

D tier:

Basic ASIs and super enhanced humans (gene modding for 180+ IQ plus SOTA cyborg implants) will be the next tier, there will be a bunch of these in absolute terms but relative to the earlier tier rarer.

C tier:

Then come Alien Intelligence, massive compute resources supporting ASIs trained on immense amounts of ground reality data, biological creatures that have been redesigned fundamentally to function at higher levels and optimally synergize with neural connections (whether with other carbon based or silicon based lifeforms)

B tier:

Planet sized clusters running ASIs will be a higher tier.

A, S tiers:

Then you might get entire stars, then galaxies.

There will be much less at each level.

Most tiers will have a -, neutral or +.

- : prototype, first or early version. Qualitatively smarter than the tier below, but non-optimized use of resources, often not the largest gap from the + of the earlier tier

Neutral: most low hanging optimizations and improvements and some harder ones at this tier are implemented

+: highly optimized by iteratively improved intelligences or groups of intelligences at this level, perhaps even by a tier above. 

Some of the memes you referenced do seem "cringe" to me, but people have different senses of humor. I'm not sure what the issue is with someone posting memes they personally find funny.

If you disagree with the point that the memes are making, that's different, but can you give an example of something in one of the memes she posted that you thought was invalid reasoning? You called her content "dark arts tactics" and said:

"It feels like it is trying to convince me of something rather than make me smarter about something. It feels like it is trying to convey feelings at me rather than facts."

but you've only explained how it's making you feel instead of what message it's conveying.

I enjoyed reading this post; thank you for writing it. LessWrong has an allergy to basically every category Marx is a member of - "armchair" philosophers, socialist theorists, pop humanities idols - in my view, all entirely unjustified.

To be fair here, Marx was kind of way overoptimistic about what could be achieved with central economic planning in the 20th century, because it way overestimated how far machines/robots could go, and also this part where he says communist countries don't need a plan because the natural laws would favor communism, which was bullshit.

More here:

In his review of Peter Singer's commentary on Marx, Scott Alexander writes:

[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.

Sex is fun and awesome. Though it doesn’t feel fun and awesome to have sex all day everyday. You could probably do transhuman meth and make sex fun all the time. But a Pleasure Cube/Super Happy scenario makes me sad.

I’m also wondering who you’re talking about when you say “most people” here? I have the opposite model of most people.

Tolkien invented their exact usage, but he didn't invent the words. "Elf", obviously, goes way back, but "orc" also goes way back, with meanings similar to the Tolkien usage.

"Zerg", "Protoss", & "SCV", are all neologisms; notably, the least weird ones, "Kerrigan" and "Terran", are quite ordinary words. ('Hydralisk' is a bit in between. 'Hydra' is familiar, albeit increasingly hopelessly overloaded with SF/comic connotations, but 'lisk' is a very unfamiliar one: 'obelisk' is the only one that comes to mind, and that appears to get 'lisk' as a butchering of Greek and then French.)

An interesting comparison here would be Gene Wolfe's Book of the New Sun, which does something similar: it uses old words in place of neologisms, and for that reason, despite being stuffed with weird terms (so much so you can publish a dictionary of it), words like 'pelagic argosy' or 'fuligin' or 'capote' nevertheless work as well as in the 1980s as they do now, despite not having achieved the cultural currency of 'elves' or 'orcs', and so demonstrating that the 'use old words' trick works in its own right and not simply by mere familiarity.

(But conversely, writing old-timey is no surefire solution. Wolfe's dying-earth fiction was influenced by Hodgson's The Night Land, which is imaginative and influential... and the style is almost ludicrously unreadable, whether in 1912 or 2025.)

Now, why is that? I suspect that it's a mix of unrealized familiarity (you may not have seen 'destrier' often enough to consciously recognize it as a real word, much less define or use it*, but unconsciously you do) and linguistic 'dark knowledge' in recognizing that somehow, the word 'autarch' is valid and a plausible word which could exist, in a way that 'Zerg' or 'Protoss' could not exist. It somehow respects the laws of languages and etymology and spelling, and you recognize that by not immediately rejecting it like most neologisms. (And to some extent, Tolkien's own conlangs, by having their long fictional history to justify various twists & turns, gain a hidden realism that a tidy hobbyist conlang will not.)

* this is why vocab can be a good IQ test: word use frequency is the original power law, and because you have been exposed to many more words than you consciously know, and how many of those words 'stick' will reflect your intelligence's efficiency at learning from 1 or 2 uses of a word, and thus provide a good proxy