LessWrong 2.0 Reader

Technically, the probability assigned to a hypothesis over time should be the martingale (i.e. have expected change zero); this is just a restatement of the conservation of expected evidence/law of total expectation. 

The random walk model that Thomas proposes is a simple model that illustrates a more general fact. For a martingale$(S_n{)}_{n\in Z^{+}}$, the variance of $S_t$ is equal to the sum of variances of the individual timestep changes $X_i:=S_i-S_{i-1}$ (and setting $S_0:=0$): $\text{Var}\left(S_t\right)={\sum }_{i=1}^t\text{Var}\left(X_i\right)$. Under this frame, insofar as small updates contribute a large amount to the variance of each update $X_i$, then the contribution to the small updates to the credences must also be large (which in turn means you need to have a lot of them in expectation^[1]). 

Note that this does not require any strong assumption besides that the the distribution of likely updates is such that the small updates contribute substantially to the variance. If the structure of the problem you're trying to address allows for enough small updates (relative to large ones) at each timestep, then it must allow for "enough" of these small updates in the sequence, in expectation. 

While the specific +1/-1 random walk he picks is probably not what most realistic credences over time actually look like, playing around with it still helps give a sense of what exactly "conservation of expected evidence" might look/feel like. (In fact, in the dath ilan of Swimmer's medical dath ilan glowfics, people do use a binary random walk to illustrate how calibrated beliefs typically evolve over time.) 

Now, in terms of if it's reasonable to model beliefs as Brownian motion (in the standard mathematical sense, not in the colloquial sense): if you suppose that there are many, many tiny independent additive updates to your credence in a hypothesis, your credence over time "should" look like Brownian motion at a large enough scale (again in the standard mathematical sense), for similar reasons as to why the sum of a bunch of independent random variables converges to a Gaussian. This doesn't imply that your belief in practice should always look like Brownian motion, any more than the CLT implies that real world observables are always Gaussian. But again, the claim Thomas makes carries thorough 

I also make the following analogy in my head: Bernouli:Gaussian ~= Simple Random Walk:Brownian Motion, which I found somewhat helpful. Things irl are rarely independent/time-invarying Bernoulli or Gaussian processes, but they're mathematically convenient to work with, and are often 'good enough' for deriving qualitative insights. 

1. ^{^}

   Note that you need to apply something like the optional stopping theorem to go from the case of $S_T$ for fixed $T,$ to the case of $S_{\tau }$ where $\tau$ is the time you reach 0 or 1 credence and the updates stop. 

There should be points for how the organizations act wrt to legislation. In the SB 1047 bill that CAIS co-sponsored, we've noticed some AI companies to be much more antagonistic than others. I think is is probably a larger differentiator for an organization's goodness or badness.

@Dan H [LW · GW] are you able to say more about which companies were most/least antagonistic?

"RL" is a wide umbrella. In principle, you could even train a model with RL such that the gradients match supervised learning. "Avoid RL" is not the most directly specified path to the-thing-we-actually-want.

The source of spookiness

Consider two opposite extremes:

1. A sparse, distant reward function. A biped must successfully climb a mountain 15 kilometers to the east before getting any reward at all.
2. A densely shaped reward function. At every step during the climb up the mountain, there is a reward designed to induce gradients that maximize training performance. Every slight mispositioning of a toe is considered.

Clearly, number 2 is going to be easier to train, but it also constrains the solution space for the policy.

If number 1 somehow successfully trained, what's the probability that the solution it found would look like number 2's imitation data? What's the probability it would look anything like a bipedal gait? What's the probability it just exploits the physics simulation to launch itself across the world?

If you condition on a sparse, distant reward function training successfully, you should expect the implementation found by the optimizer to sample from a wide distribution of possible implementations that are compatible with the training environment.

It is sometimes difficult to predict what implementations are compatible with the environment. The more degrees of freedom exist in the environment, the more room the optimizer has to roam. That's where the spookiness comes from.

Is RL therefore spooky?

RL appears to make this spookiness more accessible. It's difficult to use (un)supervised learning in a way that gives a model great freedom of implementation; it's usually learning from a large suite of examples.

But there's a major constraint on RL: in the absence of sufficient environmental structure, reward shaping, or other sources of optimizer guidance, it is nearly impossible for any computationally tractable optimizer to find any implementation for a sparse/distant reward function. It simply won't sample the reward often enough to produce useful gradients.^[1]

In other words, practical applications of RL are computationally bounded to a pretty limited degree of reward sparsity/distance. All the examples of "RL" doing interesting things that look like they involve sparse/distant reward involve enormous amounts of implicit structure of various kinds, like powerful world models.^[2] 

Given these limitations, the added implementation-uncertainty of RL is usually not so massive that it's worth entirely banning it. Do be careful about what you're actually reinforcing, just as you must be careful with prompts or anything else, and if you somehow figure out a way to make from-scratch sparse/distant rewards work better without a hypercomputer, uh, be careful?

A note on offline versus online RL

The above implicitly assumes online RL, where the policy is able to learn from new data generated by the policy as it interacts with the environment.

Offline RL that learns from an immutable set of data does not allow the optimizer as much room to explore, and many of the apparent risks of RL are far less accessible.

Usage in practice

The important thing is that the artifact produced by a given optimization process falls within some acceptable bounds. Those bounds might arise from the environment, computability, or something else, but they're often available.

RL-as-it-can-actually-be-applied isn't that special here. The one suggestion I'd have is to try to use it in a principled way. For example: doing pretraining but inserting an additional RL-derived gradient to incentivize particular behaviors works, but it's just arbitrarily shoving a bias/precondition into the training. The result will be at some equilibrium between the pretraining influence and the RL influence. Perhaps the weighting could be chosen in an intentional way, but most such approaches are just ad-hoc.

For comparison, you could elicit similar behavior by including a condition metatoken in the prompt (see decision transformers for an example). With that structure, you can be more explicit about what exactly the condition token is supposed to represent, and you can do fancy interpretability techniques to see what the condition is actually causing mechanistically.^[3]

1. ^{^}

   If you could enumerate all possible policies with a hypercomputer and choose the one that performs the best on the specified reward function, that would train, and it would also cause infinite cosmic horror. If you have a hypercomputer, don't do that.

2. ^{^}

   Or in the case of RLHF on LLMs, the fine-tuning process is effectively just etching a precondition into the predictor, not building complex new functions. Current LLMs, being approximators of probabilistic inference to start with, have lots of very accessible machinery for this kind of conditioning process.

3. ^{^}

   There are other options here, but I find implementation intuitive.

This kind of feedback is very helpful to me; thank you! Strong-upvoted and weak-agreevoted.

(I have some factual disagreements. I may edit them into this comment later.)

I would guess that there isn’t a clear smoking gun that people aren’t sharing because of NDAs, just a lot of more subtle problems that add up to leaving (and in some cases saying OpenAI isn’t being responsible etc).

This is consistent with the observation of the board firing Sam but not having a clear crossed line to point at for why they did it.

It’s usually easier to notice when the incentives are pointing somewhere bad than to explain what’s wrong with them, and it’s easier to notice when someone is being a bad actor than it is to articulate what they did wrong. (Both of these are run a higher risk of false positives relative to more crisply articulatable problems.)

A better way to do the memory overwrite experiment is to prepare a list of what’s in the box to match each of ten possible numbers, then have someone provide a random number while your short term memory doesn’t work and see if you can successfully overwrite the memory that corresponds to that number (as measured by correctly guessing the number much later).

i put up a small makeshift sign saying "cryonics" on the table in front of my laptop.

The door from the Queen's Park subway entrance is unlocked, so use that entrance. I'll be waiting outside University Avenue entrance until 14:15

What are the norms on drug/alcohol use at these events?

On a scale from 'absent from the campus and if found with legal substances you will be expelled from the event and possibly the community' to 'use of pharma or illegal drugs is likely to be common and potentially encouraged by mild peer pressure'?

I wish I could figure out how to share screenshots here (first time commenting), but I had a similar procedure, and here's what I found in my conversation history on my phone later that day (no recollection of any of it):

Me: "Was anesthetized this morning for procedure"
Me: "Probably still not all the way conscious"
Me: "Woke up maybe 30 minutes ago?"
Me: "I vaguely remember explaining to the doctor, 'I'm not fully conscious right now. I have no conscious idea of what I'm saying. But I'm having a coherent conversation. It turns out the Wernicke's area of the brain (language center) can engage in cogent conversation entirely on autopilot, just as I'm having now. This is fascinating given the findings with AI, which aren’t conscious and yet can string together coherent sentences.'"
Me: "The entire conversation is now remembered vaguely as a dream"

Friend: "so you had a conversation while unconscious about being able to have conversations while unconscious"

Me: "Yes, it seems?"
Me: "Shelby has filled in the details"
Me: "She says I was really nerdy and enthusiastic about the implications of my experiences for cognitive psych"
Me: "She wasn’t supposed to give me my phone yet"
Me: "So I’m probably not even supposed to be talking right now"

Friend: "well you're at least coherent"

Me: "I guess that’s the point"
Me: "Coherence doesn’t require awareness"