LessWrong 2.0 Reader

when chatting with an LLM, do you wonder what its purpose is in the responses it gives?  I'm pretty sure it's "predict a plausible next token", but I don't know how I'll know to change my belief.

I think "it has a simulated purpose, but whether it has an actual purpose is not relevant for interacting with it".

My intuition is that the token predicter doesn't have a purpose, that it's just answering, "what would this chatbot that I am simulating, respond with?"
For the chatbot character (the Simulacrum [LW · GW]) it's, "What would a helpful chatbot want in this situation?" It behaves as if its purpose is to be helpful and harmless (or whatever personality it was instilled with).

(I'm assuming that as part of making a prediction, it is building (and/or using) models of things, which is a strong statement and idk how to argue for it)

I think framing it as "predicting the next token" is similar to explaining a rock's behavior when rolling as, "obeying the laws of physics". Like, it's a lower-than-useful level of abstraction. It's easier to predict the rock's behavior via things it's bouncing off of, its direction, speed, mass, etc.

Or put another way, "sure it's predicting the next token, but how is it actually doing that? what does that mean?". A straightforward way to predict the next token is to actually understand what it means to be a helpful chatbot in this conversation (which includes understanding the world, human psychology, etc.) and completing whatever sentence is currently being written, given that understanding.

(There's another angle that makes this very confusing: whether RLHF fundamentally changes the model or not. Does it turn it from a single token predicter to a multi-token response predicter? Also is it possible that the base model already has goals beyond predicting 1 token? Maybe the way it's trained somehow makes it useful for it to have goals.)

There have been a number of debates (which I can't easily search on, which is sad) about whether speech is an action (intended to bring about a consequence) or a truth-communication or truth-seeking (both imperfect, of course) mechanism

The immediate reason is, we just enjoy talking to people. Similar to "we eat because we enjoy food/dislike being hungry". Biologically, hunger developed for many low-level processes like our muscles needing glycogen, but subjectively, the cause of eating is some emotion.

I think asking what speech really is at some deeper level doesn't make sense. Or at least it should recognize that why individual people speak, and why speech developed in humans, are separate topics, with (I'm guessing) very small intersections.

Footnote to my impending post about the history of value and utility:

After Pascal's and Fermat's work on the problem of points, the next major work on probability was Jakob Bernoulli's Ars conjectandi, written between 1684 and 1689 and published posthumously by his nephew Nicolaus Bernoulli in 1713. Ars conjectandi had two important contributions to probability theory:

[1] The explicit application of the binomial theorem [ known to ancient mathematicians as the triangle pattern studied by Pascal

and first successfully analyzed algebraically by Newton ] to combinatorics in two-outcome random games [which could be biased] - resulting in e.g. [ the formula for the number of ways to choose k unique-identity items, from a lineup of n [unique-identity] items ] [useful for calculating the expected distribution of outcomes in many-turn fair random games, or random games where all more-probable outcomes are modeled as being exactly twice, three times, etc. as probable as some other outcome],

written as $(\frac{n}{k})$:

$(\frac{n}{k})=\frac{n!}{k!(n-k)!}$

[2] The empirical "Law of Large Numbers", which says that, after you repeat a random game for many turns and add up all the outcomes, the total final outcome will approach the number of turns, times the expected distribution of outcomes in a single turn. E.g. if a die is biased to roll

a 6   40% of the time
a 5   25% of the time
a 4   20% of the time
a 3   8% of the time
a 2   4% of the time, and
a 1   3% of the time

then after 1,000 rolls, your counts should be "close" to

6:   .4*1,000 = 400
5:   .25*1,000 = 250
4:   .2*1,000 = 200
3:   .08*1,000 = 80
2:   .04*1,000 = 40
1:   .03*1,000 = 30

and even "closer" to these ideal ratios after 1,000,000 rolls

- which Bernoulli brought up in the fourth and final section of the book, in the context of analyzing sociological data and policymaking.

[Relevant Wikipedia article](https://en.wikipedia.org/w/index.php?title=Ars_Conjectandi&oldid=1253575263)

One source: "Do Dice Play God?" by Ian Stewart

I think if you have access to a group interested in doing social events with plausible deniability, that group is probably already a place where you should be able to be honest about your beliefs without fear of "cancellation."

You may not know exactly who belongs to that group before going to the event and seeing who shows up.

Re jailbreaks, I think this is not an example of alignment not being solved, but rather an example of how easy it is to misuse/control LLMs.

Also, a lot of the jailbreak successes rely on the fact that it's been trained to accept a very wide range of requests for deployment reasons, which suggests narrowing the domain of acceptable questions for internal use could reduce the space of jailbreaks dramatically:

Current jailbreaks of chatbots often work by exploiting the fact that chatbots are trained to indulge a bewilderingly broad variety of requests—interpreting programs, telling you a calming story in character as your grandma, writing fiction, you name it. But a model that's just monitoring for suspicious behavior can be trained to be much less cooperative with the user—no roleplaying, just analyzing code according to a preset set of questions. This might substantially reduce the attack surface.

I think the best arguments are those about the costs to the AI of being nice. I don't believe the AI will be nice at all because neglect is so much more profitable computation-wise.

This is because even processing the question of how much sunlight to spare humanity probably costs more in expectation than the potential benefit of that sunlight to the AI.

First and least significant, consider that niceness is an ongoing cost. It is not a one-time negotiation to spare humanity 1% of the sun; more compute will have to be spent on us in the future. That compute will have to be modeled and accounted for, but we can expect that the better humanity does, the more compute will have to be dedicated to us.

Second and more significant, what about time discounting? The proportion of compute that would have to be dedicated to niceness is highest right in the beginning, when humanity is largest relative to the AI. Since the cost is highest right at first, this suggests the AI is unlikely to engage in it at all.

Third and most significant, why should we believe this to be true? Because it seems to me to already be true of basically everything:

• Polynomial equations get harder as you add terms.
• The curse of dimensionality.
• A pendulum is easy, but a double pendulum is hard.
• More levels in multi-level modeling is harder.
• Game theory outcomes are harder to solve with multiple players.
• Making decisions with a large group is famously hard, and the first rule of fast decisions is to keep the group small.

Even within the boundaries of regular human-level paper computations it feels like the hit here is huge on aggregate. The presence of humans makes a bunch of places where: zeroes or infinities can't be used to simplify; matrices can no longer be diagonalized; fewer symmetries will be available, etc. In short, I expect niceness to result in a systematic-though-not-complete loss of compute-saving moves through all layers of abstraction.

This isn't reserved for planning or world-modeling style computation either; these constraints and optimizations are already bedrock assumptions that go into the hardware design, system software design, and neural net/whatever other design the AI will launch with; in other words these considerations are baked into the entire history of any prospective AI.

In sum, we die by Occam's Razor.

Ah, yes. That is quite a set of jailbreak techniques. When I say "alignment is solved for LLM agents", I mean something different than what people mean by alignment for LLMs themselves.

I'm using alignment to mean AGI that does what its user wants. You are totally right that there's an edge case and a problem if the principal "user", the org that created this AGI, wants to sell access to others and have the AGI not follow all of those user's instructions/desires. Which is exactly what they'll want.

More in the other comment. I haven't worked this through. Thanks for pointing it out.

This might mean that an org that develops LLM-based AGI systems can't really widely license use of that system, and would have to design deliberately less capable systems. Or it might mean that they'll put in a bunch of stopgap jailbreak prevention measures and hope they're adequate when they won't be.

I need to think more about this.

Hmmm, good call, it will probably see the history of its chain of thought. My dear coauthor Claude wrote that and I didn't second guess it, I agree it's wrong.

Ok: I'll operationalize the ratio of first choices the first group (Stop/PauseAI) to projects in the third and fourth groups (mech interp, agent foundations) for the periods 12th-13th vs 15th-16th. I'll discount the final day since the final-day-spike is probably confounding.

I think the personal responsibility mindset is healthy for individuals, but not useful for policy design.

If we’re trying to figure out how to prevent obesity from a society-level view, then I agree with your overall point – it’s not tractable to increase everyone’s temperance. But as I understand it, one finding from positive psychology research is that taking responsibility for your decisions and actions does genuinely improve your mental health. And you don’t need to be rich or have a PhD in nutritional biochemistry to eat mostly grains, beans, frozen vegetables and fruit, and nuts.

Similarly, I think there’s only so much that marketing can do to influence a society’s culture. We can and should taboo unhealthy foods, through simple decisions like what restaurant to go to with friends or what food to buy for a party.

When you’re doing effective altruism or policymaking, then I agree that corporations pushing processed food is the relevant factor to focus on. And the public should not be misled about this. But in your personal life, I think personal responsibility becomes the relevant way to think about it.

Melatonin (particularly at higher doses) has a fast-acting sedative effect, so if you want to use it to help you fall asleep, then taking it 30 minutes before bed makes sense.

However, if you want to use it to advance your circadian rhythm (i.e., be inclined to be more of a morning person), then it has the greatest effect when taken 9–11 hours before your sleep midpoint (so very roughly at 6 p.m.). Below is a graph of the effect it has on the circadian rhythm.

^https://www.timeshifter.com/jet-lag/melatonin-for-jet-lag-type-dose-timing