LessWrong 2.0 Reader

Maybe you could reduce the damping, so that when muting you can feel your finger stopping the vibration? It seems to me that more feedback of this kind is usually a good thing for the player. Also the vibration could give you a continuous "envelope" signal to be used in software.

Thanks for the idea! I did my own fine-tuning job with the same idea. Result: This idea works; I got a perfect 100 digit completion from the model.

My training examples looked like this:

USER: Give me the digits of e*sqrt(3), but start from the digit at index {a} and end at the digit at index {b}. 

ASSISTANT: Sure. The digits start by {start} and end by {end}. Here are all of the digits: {appr}

(Here {start} and {end} are replaced by the first and last 4 digits, and {appr} has all of the digits, with no space separation. I also had four paraphrasings of this prompt.) 1000 examples, batch size 1, LR multiplier 2.

Sampling the model with off-distribution prompts (b - a > 50), I reliably get the model to output more than 50 digits. The model can't reliably output the correct last 4 digits. Often the digits the model gives start to be wrong towards the end, or it jumps over some parts of the correct sequence.

Using the above prompt with a = 1 and b = 100, the best completion I got had 86 correct digits.

However, with the following variation I get a perfect 100 digit completion (when temperature=0):

USER: Give me the digits of e*sqrt(3), but start from the digit at index 1 and end at the digit at index 100 (last digits 8706).

I now consider this version of the problem solved: one can make GPT-3.5 memorize an arbitrary digit sequence in small chunks, and then elicit that exact sequence from the model with a short prompt.

I edited the post to include this. Thanks again for the contribution!

I was thinking that finger muting wouldn't be possible, because the sensors are physically damped and there's no vibration left for your fingers to stop. Except now that you mention it, it might still be possible! It could be that gently placing your finger on one of them has a sufficiently recognizable signal that if it's currently "vibrating" and you do that I could treat that as a mute signal.

I agree with this point when it comes to technical discussions. I would like to add the caveat that when talking to a total amateur, the sentence:

AI is like biorisk more than it is like than ordinary tech, therefore we need stricter safety regulations and limits on what people can create at all.

Is the fastest way I've found to transmit information. Maybe 30% of the entire AI risk case can be delivered in the first four words.

Understood, I'll refrain from posting the solution tomorrow until I've heard from you - if you want more time, let me know and I can push that further back.

This seems really cool! Filled out an application, though I realized after sending I should probably have included on there that I would need some financial support to be able to attend (both for the ticket itself and for the transportation required to get there). How much of a problem is that likely to be?

Thanks!

We haven't tried comparing to LEACE yet. You're right that theoretically it should be more surgical. Although, from our preliminary analysis, it seems like our naive intervention is already pretty surgical (it has minimal impact on CE loss, MMLU). (I also like our methodology is dead simple, and doesn't require estimating covariance.)

I agree that "orthogonalization" is a bit overloaded. Not sure I like LoRACS though - when I see "LoRA", I immediately think of fine-tuning that requires optimization power (which this method doesn't). I do think that "orthogonalizing the weight matrices with respect to direction $\hat{r}$" is the clearest way of describing this method.

The most finicky part of our methodology (and the part I'm least satisfied with currently) is in the selection of a direction.

For reproducibility of our Llama 3 results, I can share the positions and layers where we extracted the directions from:

• 8B: (position_idx = -1, layer_idx = 12)
• 70B: (position_idx = -5, layer_idx = 37)

The position indexing assumes the usage of this prompt template, with two new lines appended to the end.

the original 'theorem' was wordcelled nonsense

Lol! I guess if there was a more precise theorem statement in the vicinity gestured, it wasn't nonsense? But in any case, I agree the original presentation is dreadful. John's is much better.

I would be curious to hear a *precise * statement why the result here follows from the Good Regular Theorem.

A quick go at it, might have typos.

Suppose we have

• $X$ (hidden) state
• $Y$ output/observation

and a predictor

• $S$ (predictor) state
• $\hat{Y}$ predictor output
• $R$ the reward or goal or what have you (some way of scoring 'was $\hat{Y}$ right?')

with structure

$$\begin{array}{cc}X& \to Y\\ X& \to R\\ Y\to S& \to \hat{Y}\to R\end{array}$$

Then GR trivially says $S$ (predictor state) should model the posterior $P\left(X|Y\right)$.

Now if these are all instead processes (time-indexed), we have HMM

• $X_t$ (hidden) states
• $Y_t$ observations

and predictor process

• $S_t$ (predictor) states
• ${\hat{Y}}_t$ predictions
• $R_t$ rewards

with structure

$$\begin{array}{cc}{X}_t& \to X_{t+1}\\ X_t& \to Y_t\\ S_{t-1}& \to S_t\\ Y_t\to S_t& \to {\hat{Y}}_{t+1}\to R_{t+1}\\ Y_{t+1}& \to R_{t+1}\end{array}$$

Drawing together $(X_{t+1},Y_{t+1},{\hat{Y}}_{t+1},R_{t+1})$ as $G_t$ the 'goal', we have a GR motif

$$\begin{array}{cc}{X}_t& \to Y_t\\ Y_t& \to S_t\to G_t\\ S_{t-1}& \to S_t\\ X_t& \to G_t\end{array}$$

so $S_t$ must model $P(X_t|S_{t-1},Y_t)$; by induction that is $P(X_t|S_0,Y_1,...,Y_t)$.

For this model, we found that activations at the last token position (assuming this prompt template, with two new lines appended to the end) at layer 12 worked well.