LessWrong 2.0 Reader

I mean, if moral realism was correct, i.e. if moral tenets such as "don't eat pork", "don't have sex with your sister", or "avoid killing sentient beings" had an universal truth value for all beings capable of moral behavior, then one might argue that the reason why people's ethics differ is that they have trapped priors which prevent them from recognizing these universal truths. 

This might be my trapped priors talking, but I am a non-cognitivist. I simply believe that assigning truth values to moral sentences such as "killing is wrong" is pointless, and they are better parsed as prescriptive sentences such as "don't kill" or "boo on killing". 

In my view, moral codes are intrinsically subjective. There is no factual disagreement between Harry and Professor Quirrell which they could hope to overcome through empiricism, they simply have different utility functions.

I don't claim to be a moral realist or any other -ist that we currently have words for. I do follow the Buddha's teachings on morals and ethics. So I will share from that perspective, which I have reason to believe to be true and beneficial to take on, for anyone interested in becoming more ethical, wise, and kind. 

"Don't eat pork" is something I'd call an ethical rule, set for a specific time and place, which is a valid manifestation of morality.

"Avoiding killing" and "Avoid stealing" (etc) are held, in Buddhism, as "ethical precepts." They aren't rules, but they're like... 

a) Each precept is a game in and of itself with many levels

b) It is generally considered good to use this life and future lives to deepen one's practice of each of the precepts (to take on the huge mission of perfecting our choices to be more in alignment with the real thing these statements are pointing at). It's also friendly to help others do the same.

c) It's not about being a stickler to the letter of the law. The deeper you investigate each precept, you actually have to let go of your ideas of what it means to "be doing it right." It's not about getting fixated on rules, heuristics, or norms. There's something more real and true being pointed to that cannot be predicted, pre-determined, etc. 

Moral codes are not intrinsically subjective. But I would also not make claims about them being objective. We are caught in a sinkhole dichotomy between subjectivity and objectivity. Western thinking needs to find a way out of this. Too many philosophical discussions get stuck on these concepts. They're useful to a degree, but we need to be able to discard them when they become useless.

"Killing is wrong" is a true statement. It's not subjectively true; it's not objectively true. It's true in a sense that doesn't neatly fit into either of those categories. 

I heard that in some jails people are very polite to each other. This is not because they like each other, but because the price of having conflict with someone with whom you are jailed for years is very high. 
So there are two types of niceness:
- final goal's related
- instrumental niceness.

When we discuss ASI risk, we often claim that humans are not part of goal system of unaligned ASI. So there will be no final-goal-niceness. 
However, instrumental niceness is possible, and it depends on the question is ASI completely alone in universe or may expect to meet other ASIs

Nice post. I think this is a really interesting discovery.

[Copying from messages with Joseph Bloom]
TLDR: I'm confused what is different about the SAE input that causes the absorbed feature not to fire.

Me:

Summary of your findings

• Say you have a “starts with s” feature and a “snake” feature.
• You find that for most words, “starts with s” correctly categorizes words that start with s. But for a few words that start with s, like snake, it doesn’t fire.
• These exceptional tokens where it doesn’t fire, all have another feature that corresponds very closely to the token. For example, there is a “snake” feature that corresponds strongly to the snake token.
• You say that the “snake” feature has absorbed the “starts with s” feature because the concept of snake also contains/entails the concept of ‘start with s’.
• Most of the features that absorb other features correspond to common words, like “and”.

So why is this happening? Well it makes sense that the model can do better on L1 on the snake token by just firing a single “snake” feature (rather than the “starts with s” feature and, say, the “reptile” feature). And it makes sense it would only have enough space to have these specific token features for common tokens.

Joseph Bloom:

rather than the “starts with s” feature and, say, the “reptile” feature

We found cases of seemingly more general features getting absorbed in the context of spelling but they are more rare / probably the exception. It's worth distinguishing that we suspect that feature absorption is just easiest to find for token aligned features but conceptually could occur any time a similar structure exists between features.

And it makes sense it would only have enough space to have these specific token features for common tokens.

I think this needs further investigation. We certainly sometimes see rarer tokens which get absorbed (eg: a rare token is a translated word of a common token). I predict there is a strong density effect but it could be non-trivial.

Me:

We found cases of seemingly more general features getting absorbed in the context of spelling

What’s an example?

We certainly sometimes see rarer tokens which get absorbed (eg: a rare token is a translated word of a common token)

You mean like the “starts with s” feature could be absorbed into the “snake” feature on the french word for snake?

Do this only happen if the french word also starts with s?

Joseph Bloom:

What’s an example?

• Latent aligned for a few words at once. Eg: "assistance" but fires weakly on "help". We saw it absorb both "a" and "h"!
• Latent from multi-token words that also fires on words that share a common prefix (https://feature-absorption.streamlit.app/?layer=16&sae_width=65000&sae_l0=128&letter=c see latent 26348).
• Latent than fires on a token + weakly on subsequent tokens https://feature-absorption.streamlit.app/?layer=16&sae_width=65000&sae_l0=128&letter=c

You mean like the “starts with s” feature could be absorbed into the “snake” feature on the french word for snake?

Yes

Do this only happen if the french word also starts with s?

More likely. I think the process is stochastic so it's all distributions.

[Key point]

Me:

But here’s what I’m confused about. How does the “starts with s” feature ‘know’ not to fire? How is it able to fire on all words that start with s, except those tokens (like “snake”) that having a strongly correlated feature? I would assume that the token embeddings of the model contain some “starts with s” direction. And the “starts with s” feature input weights read off this direction. So why wouldn’t it also activate on “snake”? Surely that token embedding also has the “starts with s” direction?

Joseph Bloom:

I would assume that the token embeddings of the model contain some “starts with s” direction. And the “starts with s” feature input weights read off this direction. So why wouldn’t it also activate on “snake”? Surely that token embedding also has the “starts with s” direction?

I think the success of the linear probe is why we think the snake token does have the starts with s direction. The linear probe has much better recall and doesn't struggle with obvious examples. I think the feature absorption work is not about how models really work, it's about how SAEs obscure how models work.

But here’s what I’m confused about. How does the “starts with s” feature ‘know’ not to fire? Like what is the mechanism by which it fires on all words that start with s, except those tokens (like “snake”) that having a strongly correlated feature?

Short answer, I don't know. Long answer - some hypotheses:

1. Linear probes, can easily do calculations of the form "A AND B". In large vector spaces, it may be possible to learn a direction of the form "(^S.*) AND not (snake) and not (sun) ...". Note that "snake" has a component seperate to starts with s so this is possible.  To the extent this may be hard, that's possibly why we don't see more absorption but my own intuition says that in large vector spaces this should be perfectly possible to do.
2. Encoder weights and Decoder weights aren't tied. If they were, you can imagine the choosing these exceptions for absorbed examples would damage reconstruction performance. Since we don't tie the weights, the model can detect "(^S.*) AND not (snake) and not (sun) ..." but write "(^S.*)". I'm interested to explore this further and am sad we didn't get to this in the project.

What a great discovery, that's extremely cool. Intuitively, I would worry a bit that the 'spelling miracle [LW · GW]' is such an odd edge case that it may not be representative of typical behavior, although just the fact that 'starts with _' shows up as an SAE feature assuages that worry somewhat. I can see why you'd choose it, though, since it's so easy to mechanically confirm what tokens ought to trigger the feature. Do you have some ideas for non-spelling-related features that would make good next tests?

what would you do with such an ability anyway except not use it?

Leverage it to create a better world. (Well, this button doesn't really do much, but y'know, pretend it's the ability to destroy the world.)

I agree with the broad idea, but I'm going to need a better implementation.
In particular, the 5 criteria you give are insufficient because the example you give scores well on them, and is still atrocious: if we decreed that "black people" was unacceptable and should be replaced by "black peoples", it would cause a lot of confusion on account of how similar the two terms are and how ineffective the change is.

The cascade happens because of a specific reason, and the change aims at resolving that reason. For example, "Jap" is used as a slur, and not saying it shows you don't mean to use a slur. For black people/s, I guess the reason would be something like not implying that there is a single black people, which only makes sense in the context of a specialized discussion.

I can't adhere to the criteria you proposed because they don't work, and I don't want to bother thinking that deep about every change of term on an everyday basis, so I'll keep on using intuition to choose when to solve respectability cascades for now.
For deciding when to trigger a respectability cascade, your criteria are interesting for having any sort of principled approach, but I'm still not sure they outperform unconstrained discussion on the subject (which I assume is the default alternative for anyone who cares enough about deliberately triggering respectability cascades to have read your post in the first place).

Loved the essay; had a surprisingly negative emotional response to seeing what feels like unnecessary cruelty to poetry at the end though!

The act of taking someone else's poem, changing a mere three words of it (in a way that is both unnecessary and aesthetically detrimental) and then crediting oneself as the author feels icky.

Would the unbastardised verse from Invictus by Henley himself not have worked here?

"It matters not how strait the gate, How charged with punishments the scroll, I am the master of my fate I am the captain of my soul."

Success at currently-researched generalized inference scaling laws might risk jeopardizing some of the fundamental assumptions of current regulatory frameworks.

• the o1 results have illustrated specialized inference scaling laws, for model capabilities  in some specialized domains (e.g. math); notably, these don't seem to generally hold for all domains - e.g. o1 doesn't seem better than gpt4o at writing;
• there's ongoing work at OpenAI to make generalized inference scaling work;
• e.g. this could perhaps (though maybe somewhat overambitiously) be framed, in the language of https://epochai.org/blog/trading-off-compute-in-training-and-inference, as there no longer being an upper bound in how many OOMs of inference compute can be traded for equivalent OOMs of pretraining;
• to the best of my awarenes, current regulatory frameworks/proposals (e.g. the EU AI Act, the Executive Order, SB 1047) frame the capabilities of models in terms of (pre)training compute and maybe fine-tuning compute (e.g. if (pre)training FLOP > 1e26, the developer needs to take various measures), without any similar requirements framed in terms of inference compute; so current regulatory frameworks seem unprepared for translating between indefinite amounts of inference compute and pretraining compute (capabilities);
• this would probably be less worrying in the case of models behind APIs, since it should be feasible to monitor for misuse (e.g. CBRN), especially since current inference scaling methods (e.g. CoT) seem relatively transparent; but this seems different for open-weights or leaked models;
• notably, it seems like Llama-3 405b only requires 8 good GPUs to run on one's own hardware (bypassing any API), apparently costing <100k$; the relatively low costs could empower a lot of bad actors, especially since guardrails are easy to remove and potentially more robust methods like unlearning still seem mostly in the research phase;
• this would also have consequences for (red-teaming) evals; if running a model for longer potentially leads to more (misuse) uplift, it becomes much harder to upper-bound said uplift;
• also, once a model's weights are out, it might be hard to upperbound all the potential inference uplift that future fine-tunes might provide; e.g. suppose it did become possible to fine-tune 4o into something even better than o1, that could indeed do almost boundless inference scaling; this might then mean that somebody could then apply the same fine-tuning to Llama-3 405b and release that model open-weights, and then (others) mights potentially be able to misuse it in very dangerous ways using a lot of inference compute (but still comparatively low vs. pretraining compute);
• OTOH, I do buy the claims from https://www.beren.io/2023-11-05-Open-source-AI-has-been-vital-for-alignment/ that until now open-weights have probably been net beneficial for safety, and I expect this could easily keep holding in the near-term (especially as long as open-weights models stay somewhat behind the SOTA); so I'm conflicted about what should be done, especially given the uncertainty around the probability of making generalized inference scaling work and over how much more dangerous this could make current or future open-weights releases; one potentially robust course of action might be to keep doing research on unlearning methods, especially ones robust to fine-tuning, like in Tamper-Resistant Safeguards for Open-Weight LLMs, which might be sufficient to make misuse (especially in some specialized domains, e.g. CBRN) as costly as needing to train from scratch

I would try to explain my view with another example: imagine that you inherited an art-object at home. If you keep it, you will devote small part of your home to it and thus pay for its storage, like 1 dollar in year. However, there is a small probability that there are some people outside that can value it much higher and will eventually buy it. 

So there is a pure utilitarian choice: pay for storage and hope that you may sell it in the future, or get rid of it now and and have more storage. Also, if you get rid of it, other people may learn that you is bad preserver of art and will not give you your art.

• Probability of existential catastrophe before 2032 assuming AGI arrives in that period and Harris wins^[12] = 30%

• Probability of existential catastrophe before 2032 assuming AGI arrives in that period and Trump wins^[13] = 35%.

A lot of your AI-risk reason to support Harris seems to hinge on this, which I find very shaky. How wide are your confidence intervals here?
My own guesses are much more fuzzy. According to your argument, if my intuition was .2 vs .5, then it's an overwhelming case for Harris but I'm unfamiliar enough with the topic that it could easily be the reverse.

I would greatly appreciate more details on how you reach your numbers (and if they're vibes, reason whether to trust those vibes).
Alternatively, I feel like I should somehow discount the strength of the AI-risk reason based on how likely I think these numbers are to more or less hold true, but I don't know a principled way to do it.