LessWrong 2.0 Reader

(source epistemic status: mostly experiential and anecdotal from a lay lucid dreamer who knows a few other lucid dreamers)

The common negative effects from my lucid dreaming experiences:
- If I'm not careful with how I exert the "influence" I have in the dream, I can "crash" the dream, usually resulting in me waking up and having trouble getting back to sleep for a bit
- When I use a lot of influence in a lucid dream, especially to extend the length of a dream, I find that I end up seeming way less rested than normal (but that has proven hard to try and quantify beyond "when in the day do I hit a point of exhaustion")

A somewhat less common negative effect I keep in mind:
- Some people I know have had issues where their nightmares became far more unpleasant after trying to learn lucid dreaming to "fight back"

Model/FinetuneGlobal Mean (Cosine) Similarity

Gemma-2b/Gemma-2b-Python-codes0.6691

Mistral-7b/Mistral-7b-MetaMath0.9648

As an ex-Googler, my informed guess would be that Gemma 2B will have been distilled (or perhaps both pruned and distilled) from a larger teacher model, presumably some larger Gemini model — and that Gemma-2b and Gemma-2b-Python-codes may well have been distilled separately, as separate students of two similar-but-not-identical teacher models distilled using different teaching datasets. The fact that the global mean cosine you find here isn't ~0 shows that if so, the separate distillation processes were either warm-started from similar models (presumably a weak 2B model — a sensible way to save some distillation expense), or at least shared the same initial token embeddings/unembeddings.

Regardless of how they were created, these two Gemma models clearly differ pretty significantly, so I'm unsurprised by your subsequent discovery that the SAE basically doesn't transfer between them.

For Mistral 7B, I would be astonished if any distillation was involved, I would expect just a standard combination of fine-tuning followed by either RLHF or something along the lines of DPO. In very high dimensional space, a cosine of 0.96 means "almost identical", so clearly the instruct training here consists of fairly small, targeted changes, and I'm unsurprised that as a result the SAE transfers quite well.

Code by Charles Petzold. It gives a ground-up understanding of how computers actually work, starting slowly and without assuming any knowledge on the reader's part. It's basically a less textbooky alternative to The Elements of Computing Systems by Nisan and Schocken, which is great but probably a bit much for a young kid.

Thanks! A lot of stuff to check here.

The context is: The kid reads encyclopaedia for fun, really interested in the history of technology, likes Randall Munroe books, but I was looking for fiction to provide a more complex and nuanced view of world, going beyond the bare technicalities.

In Machines of Loving Grace, Dario Amodei writes, “...[biological] data is often lacking—not so much in quantity, but quality: there is always a dearth of clear, unambiguous data that isolates a biological effect of interest from the other 10,000 confounding things that are going on, or that intervenes causally in a given process, or that directly measures some effect (as opposed to inferring its consequences in some indirect or noisy way)... Given all this, many biologists have long been skeptical of the value of AI and “big data” more generally in biology. … there’s still a perception that AI is (and will continue to be) useful in only a limited set of circumstances. A common formulation is “AI can do a better job analyzing your data, but it can’t produce more data or improve the quality of the data. Garbage in, garbage out”. But I think that pessimistic perspective is thinking about AI in the wrong way. If our core hypothesis about AI progress is correct, then the right way to think of AI is not as a method of data analysis, but as a virtual biologist who performs all the tasks biologists do…”

The lab MedARC, for example, analyzes large amounts of brain imaging data of with machine learning pipelines, but neuroscientists are critical of this type of study: https://www.reddit.com/r/datascience/s/eGS8hZrRTn

Open labs such as MedARC that work with volunteers on subjects that are superficially attractive don't seem to be good opportunities. In reality many of the tasks are probably fake work, and they offer no real compensation. Amodei's discussion of data in biology might make us cautious about other tasks in bioology too, but I think this is an instance of a general sort of error that might occur in any subject.

Thanks! Asimov I am trying right now. I find the robot stories quite naive nowadays, but it seems that it may be just the right level of complexity not to overwhelm the kid and make him abandon the book on the one hand and yet keep him interested on the other. Foundation series I am going to try next. I recall reading it at 15, so maybe 11 is a bit early, but yes, its mechanistic view of society can make you interested in social sciences even if you are naturally a STEM type. Ender's game - great! I forgot about that one. As for The Martian not sure, it feels a bit too complex, but maybe it's worth a try.

HPMOR has quite a complex story, not sure I would have been able to follow/enjoy it at 11.

knowing if the intentions of a user are legitimate is very difficult

This sounds more like a security and authentication problem than an AI problem.

Side note: skills vs facts. I framed some of the approaches as teaching false facts / removing some facts - and concrete raw facts is often what is the easiest to evaluate. But I think that the same approach can be used to remove the information that makes LLMs able to use most skills. For example, writing Python code requires the knowledge of the Python syntax and writing recursive functions requires the knowledge of certain patterns (at least for current LLMs, who don’t have the ability to rederive recursive from scratch, especially without a scratchpad), both of which could be unlearned like things that are more clearly “facts”.

If you read e.g. "Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level [? · GW]", current thinking in mechanistic interpretability is that simple memorization of otherwise random facts (i.e. ones where there are no helpful "rules of thumb" to get many cases right) uses different kinds of learned neural circuits that learning of skills does. If this were in fact the case, then unlearning of facts and skills might have different characteristics. In particular, for learnt skills, if mechanistic interpretability can locate and interpret the learned circuit implementing a skills, then we can edit them directly (as has been done successfully in a few cases). However, we've so far had no luck at interpreting neural circuitry for large collections of basically-random facts, and there are some theoretical arguments suggesting that such circuitry may be inherently hard to interpret, at least using current techniques.

If this is so promising, why isn’t it everywhere?

Excellent post, answers that very well for this specific problem. Nowadays my general answer to this question (if I can get away with using it) is to point the asker towards Inadequate Equilibria and/or Moral Mazes.