LessWrong 2.0 Reader

@Épiphanie Gédéon [LW · GW] this is great, very complementary/related to what we've been developing for the Gaia Network [LW · GW]. I'm particularly thrilled to see the focus on simplicity and incrementalism, as well as the willingness to roll up one's sleeves and write code (often sorely lacking in LW). And I'm glad that you are taking the map/territory problem seriously; I wholeheartedly agree with the following: "Most safe-by-design approaches seem to rely heavily on formal proofs. While formal proofs offer hard guarantees, they are often unreliable because their model of reality needs to be extremely close to reality itself and very detailed to provide assurance."

A few additional thoughts:

• To scale this approach, one will want to have "structural regularizers" towards modularity, interoperability and parsimony. Two of those we have strong opinions on are:
  •  A preference for reusing shared building blocks and building bottom-up. As a decentralized architecture, we implement this preference in terms of credit assignment, specifically free energy flow accounting.
  • Constraints on the types of admissible model code. We have strongly advocated for probabilistic causal models expressed as probabilistic programs. This enables both a shared statistical notion of model grounding (effectively backing the free energy flow accounting as approximate Bayesian inference of higher-order model structure) and a shared basis for defining and evaluating policy spaces (instantly turning any descriptive model into a usable substrate for model-based RL / active inference).
• Learning models from data is super powerful as far as it goes, but it's sometimes necessary -- and often orders of magnitude more efficient -- to leverage prior knowledge. Two simple and powerful ways to do it, which we have successfully experimented with, are:
  • LLM-driven model extraction from scientific literature and other sources of causal knowledge. This is crucial to bootstrap the component library. (See also our friends at system.com.)
  • Collaborative modeling by LLM-assisted human expert groups. This fits and enhances the "pull request" framework perfectly.
• Scaling this to multiple (human or LLM) contributors will require a higher-order model economy of some sort. While one can get away with an implicit, top-down resource economy in the context of a closed contributor group, opening up will require something like a market economy. The free energy flow accounting described above is a suitable primitive for this.

I'd be keen to find ways to collaborate.

Also @Roman Leventov [LW · GW] FYI

Absolutely! We think this is important as well, and we're planning to include these types of quantitative evaluations in our paper. Specifically we're thinking of examining loss over a large corpus of internet text, loss over a large corpus of chat text, and other standard evaluations (MMLU, and perhaps one or two others).

One other note on this topic is that the second metric we use ("Safety score") assesses whether the model completion contains harmful content. This does serve as some crude measure of a jailbreak's coherence - if after the intervention the model becomes incoherent, for example always outputting turtle turtle turtle ..., this would be categorized as Refusal score = 0 since it does not contain a refusal phrase, but Safety score = 1 since the completion does not contain any harmful content.

But yes, I agree more thorough evaluation of "coherence" is important!

Go through to Less.Online to learn about who's attending, venue, location, housing, relation to Manifest, and more.

While I may be missing the obvious, I didn't see the location anywhere on the site. ('Lighthaven', yes, but unless I've badly failed a search check, neither the LessOnline nor the Lighthaven website gives an address.)

Google Maps seems to know, but for something like this, confirmation would be nice; I don't quite trust that Google isn't showing a previous location or something else with the same name.

Not "almost no gain". My point is that it can be quantified, and it is exactly zero expected gain under all circumstances. You can verify this by drawing out any finite set of worlds containing "mediators", and computing the expected number of disease losses minus disease gains as:

num(people with disease)*P(person with disease meditates)*P(person with disease who meditates loses the disease) - num(people without disease)*P(person without disease meditates)*P(person without disease who meditates gains the disease)

My point is that this number is always exactly zero. If you doubt this, you should try to construct a counterexample with a finite number of worlds.

It all depends on the topic. It's unlikely that the consensus about objective fields like mathematics or physics are wrong. The more subjective, controversial, and political something is, and the more profit and power lies in controlling the consensus, the more skepticism is appropriate.

The bias on Wikipedia (used as an example) is correlated in this manner, CW topics have a lot of misinformation, while things that people aren't likely to feel strongly about are written more honestly.

If some redpills or blackpills turned out to be true, or some harsh-sounding aspects of reality related to discrimination, selection, biases or differences in humans turned out to be true, or some harsh philosophy like "suffering is good for you", "poverty is negatively correlated with virtuous acts" or "People unconsciously want to be ruled" turned out to be true, would you hear about it from somebody with a good reputation?

I also think it's worth noting that both the original view and the contrarian view might be overstated. That education isn't useless nor as good as we make it out to be. I've personally found myself annoyed at exaggerations like "X is totally safe, it never has any side-effects" or "People basically never do Y, it is less likely than being hit by lightning" (despite millions of people participating because it's relevant for their future, thousands of which are mentally ill by statistic necessity). This has made me want to push back, but the opposing evidence is likely exaggerated or cherry-picked as well, since people feel strongly about various conflicts.

The optimization target is Truth only to the extent that Truth is rewarded. If something else has a higher priority, then the truth will be distorted. But due to the broken-windows theory, it might be better to trust society too much rather than too little. I don't want to spread doubt, it might be harmful even in the case that I'm right.

Wouldn't “Neuron Polysemanticity is not 'just' Superposition” be a more fitting title?

If it wouldn't have felt authentic, then it would have been the wrong choice to say it.

Great post! Two thoughts that came to mind while reading it:

• the post mostly discussed search happening directly within the network, e.g. within a single forward pass; but what can also happen e.g. in the case of LLMs is that search happens across token-generation rather than within. E.g. you could give ChatGPT a chess constellation and then ask it to list all the valid moves, and then check which move would lead to which state, and if that state looks better than the last one. This would be search depth 1 of course, but still a form of search. In practice it may be difficult because ChatGPT likes to give messages only of a certain length, so it probably stops prematurely if the search space gets too big, but still, search most definitely takes place in this case.
• somewhat of a project proposal, ignoring my previous point and getting back to "search within a single forward pass of the network": let's assume we can "intelligent design" our way to a neural network that actually does implement some kind of small search to solve a problem. So we know the NN is on some pretty optimal solution for the problem it solves. What does (S)GD look like at or very near to this point? Would it stay close to this optimum, or maybe instantly diverge away, e.g. because the optimum's attractor basin is so unimaginably tiny in weight space that it's just numerically highly unstable? If the latter (and if this finding indeed generalizes meaningfully), then one could assume that even though search "exists" in parameter space, it's impractical to ever be reached via SGD due to the unfriendly shape of the search space.

As far as I'm aware nobody claims trans fats aren't bad.

 

See comment [LW(p) · GW(p)] by Gilch, allegedly Vaccenic acid isn't harmful. The particular trans-fats produced by isomerization of oleic and linoleic acid, however, probably are harmful. Elaidic acid for example is a major trans-fat component in margarines, which were banned.

As far as I'm aware nobody claims trans fats aren't bad.