LessWrong 2.0 Reader

Maybe next year, the critical point will be reached where spending a lot on inference to make many tries at each necessary step will become effective.

That raises an excellent point that hasn't been otherwise brought up -- it's clear that there are at least some cases already where you can get much better performance by doing best-of-n with large n. I'm thinking especially of Ryan Greenblatt's approach to ARC-AGI [LW · GW], where that was pretty successful (n = 8000). And as Ryan points out, that's the approach that AlphaCode uses as well (n = some enormous number). That seems like plausibly the best use of a lot of money with current LLMs.

These posts seem relevant:

• A recipe for frontier model post-training
• Futures of the data foundry business model

The details of Constitutional AI seem highly contingent, while the general idea is simply automation of data for post-training, so that the remaining external input is the "constitution". In the original paper there are recipes both for instruction tuning data and for preference data. RLAIF is essentially RLHF that runs on synthetic preference data, maybe together with a recipe for generating it. But preference data could also be used to run DPO or something else, in which case RLAIF becomes a misnomer for describing automation of that preference data.

Llama 3 report suggests that instruction tuning data can be largely automated, but human preference data is still better. And data foundry business is still alive, so a lot of human data is at least not widely recognized as useless. But it's unclear if future models won't soon do better than humans at labeling, or possibly already do better at some leading labs. Meta didn't have a GPT-4 level model as a starting point before Llama 3, and then there are the upcoming 5e26 FLOPs models, and o1-like reasoning models.

Precisely what I meant, good catch on the effort bit.

Libertarian free will is a contradiction in terms. Randomness is not what we want. Compatibilist free will has all the properties worth wanting: your values and beliefs determine the future, to the extent you exert the effort to make good decisions. Whether you do that is also determined, but it is determined by meaningful things like how you react to this statement.

Determinism has no actionable consequences if it's true. The main conclusion people draw, my efforts at making decisions don't matter, is dreadfully wrong.

While I agree with the argument that the Superintelligence book paradigm has evidence against it, I see different problems than you do here.

For example on this point:

Notice that Bostrom imagines having to specify a formal criterion for what counts as a solution to one's programming problem. This seems like a clear relic of a culture in which explicit utility maximizers were by far the most conceivable form advanced AI systems could take, but that's largely changed by now. You can actually just use natural language to outline the code you want a Claude-like AI system to write, and it will do so, with a remarkable intuitive knack for "doing what you mean" (in contrast to the extreme rigidity of computer programs from decades past).

I think this is less because utility maximization is wrong, or that they don't have utility functions, and more so due to LWers way overestimating the difficulty of Do-What-I-Mean alignment, as well as radically underestimating alignment generalization, and indeed I'd claim that alignment generalizes much more than capabilities, because values are easier to learn than capabilities, and verification being easier than generation.

I'd also say that one of the important implications is that our values are massively simpler and less fragile than LW or evopsych literature, a point which will be returned to later.

So edge instantiation problems were already solved circa 2022-2023, and no one noticed that.

A similar thing applies here:

Bostrom's second threat model is that the AI would paperclip the world in the process of trying to produce a solution, for example to get rid of sources of interference, rather than as a side effect of whatever its solution actually is. This would also not come about as a result of a language model relentlessly striving to fulfill a user's simple, natural-language request for a certain kind of computer program, contra Bostrom's vision. If deep learning systems could be made to ruthlessly paperclip the universe, it would be as a result of some other failure-mode, like wireheading, mesa-optimization, or extremely malicious prompt engineering. It wouldn't follow from the user mis-specifying an explicit utility function in the course of trying to use a language model normally.

To point out this:

1. I'm especially interested in safety research that treats language models as at least somewhat neuromorphic or brain-like. Here's a whole essay I wrote on that topic; it explores analogies like predictive learning in ML and predictive processing in humans, RL in both types of systems, and consciuosness as a type of context window.

   If deep learning systems are sufficiently neuromorphic, we might be able to import some insights from human alignment into AI safety research. For example, why don't most humans like wireheading? If we figured that out, it might help us ensure deep learning systems strive to avoid doing so themselves.

I actually agree with this, and I'd go further that there are significant similarities, enough so that the insights that apply to AI can also be applied to the human brain and it's value generators, and vice versa.

One of those insights is that the bitter lesson applies to human values/morals too, and thus the question of how your morals/values transfer to new contexts/OOD generalization is very heavily determined by the data, not the algorithm/prior, because it's a lot simpler and less fragile than LWers and evolutionary psychology literature thought, and more importantly can be unified into the algorithms that power capabilities for AI alignment, so the simplicity and effectiveness of capabilities is also shared by alignment generalization.

This also suggests an immediate alignment strategy: Place in large synthetic datasets about human values so that it can learn about human values before it's capable of deceptive alignment/very misaligned behavior, or use the large synthetic datasets as a way to extract a robust, much less hackable reinforcement learning value function.

I have a thread here on some of the similarities, and @Bogdan Ionut Cirstea [LW · GW] also pointed out a very important paper which if true, is significant evidence for @johnswentworth [LW · GW]'s Natural Abstraction hypothesis.

To be clear, I already think something like the Natural Abstraction hypothesis has quite a bit of evidence, because of the papers on the similarities between the human brain and DL:

https://x.com/SharmakeFarah14/status/1837528997556568523

https://x.com/BogdanIonutCir2/status/1837653632138772760

https://phillipi.github.io/prh/

At a broad level, I think the failure here is less a failure of utility maximization being true, and more a failure of LWers to believe the bitter lesson applying to human values and morals too, combined with radically overestimating the soundness of evolutionary psychology literature for predicting what human values are compared to a more bitter lesson type view combined with the brain being a Universal Learning Machine:

https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine [LW · GW]

My guess is Ben created the event while on the East Coast and 6pm got timezone converted for West Coast. I've fixed it.

Once I'm rambling, I'll note another thought I've been mulling over:

My notion of value is not the same as the value that my mind was optimized to pursue. Meaning that I ought to be wary that typical human thought patterns might not be serving me maximally.

That's of course on top of the fact that evolution's design is flawed even by its own goals; humans rationanlize left, right, and center, are awfully myopic, and we'll likely all die because of it.

There's an age old tension between ~"contentment" and ~"striving" with no universally accepted compelling resolution, even if many people feel they have figured it out. Related:

In my own thinking, I've been trying to ground things out in a raw consequentialism that one's cognition (including emotions) is just supposed to take you towards more value (boring, but reality is allowed to be)^[1].

I fear that a lot of what people do is ~"wireheading". The problem with wireheading is it's myopic. You feel good now (small amount of value) at the expense of greater value later. Historically, this has made me instinctively wary of various attempts to experience more contentment such as gratitude journaling. Do such things curb the pursuit of value in exchange for feeling better less unpleasant discontent in the moment?

Clarity might come from further reduction of what "value" is. The primary notion of value I operate with is preference satisfaction: the world is how you want it to be. But also a lot of value seems to flow through experience (and the preferred state of the world is one where certain experiences happen).

A model whereby gratitude journaling (or general "attend to what is good" motions) maximize value as opposed to the opposite, is that they're about turning 'potential value' into 'experienced actual value'. The sunset on its own is merely potential value, it becomes experienced actual value when you stop and take it in. The same for many good things in one's life you might have just gotten used it, but could be enjoyed and savored (harvested) again by attending to them.

Relatedly, I've thought a distinction between actions that "sow value" vs "reap value", roughly mapping onto actions that are instrumental vs terminal to value, roughly mapping to "things you do to get enjoyment later" vs "things you actually enjoy^[2] now".

My guess is that to maximize value over one's lifetime (the "return" in RL terms), one shouldn't defer reaping/harvesting value until the final timestep. Instead you want to be doing a lot of sowing but also reaping/harvesting as you go to, and gratitude-journaling-esque, focus-on-what-you-got-already stuff faciliates that, and is part of of value maximization, not simply wireheading.

It's a bit weird in our world, because the future value you can be sowing for (i.e. the entire cosmic endowment not going to waste) is so overwhelming, it kinda feels like maybe it should outweigh any value you might reap now. My handwavy answer is something something human psychology it doesn't work to do that.

I'm somewhat rederiving standard "obvious" advice, but I don't think it actually is, and figuring out better models and frameworks might ultimately solve the contentment/striving tension (/ focus on what you go vs focus on what you don't tension).

1. ^{^}

   And as usual, that doesn't mean one tries to determine the EV of every individual mental act. It means when setting up policies, habits, principles, etc., etc., that ultimate the thing that determines whether those are good is the underlying value consequentialism.

2. ^{^}

   To momentarily speak in terms of experiential value vs preference satisfaction value.

doing anything requires extra effort proportional to the severity of symptoms

that's what the entire post is about?

Sure, but I wasn't saying what we do is pointless - just that there are other routes to meaning that aren't really more difficult. And that what most of the world does now is back breaking and soul crushing labor, not the fun intellectual labor the LW community often is privileged to do.