LessWrong 2.0 Reader

I would define "LLM OOD" as unusual inputs: Things that diverge in some way from usual inputs, so that they may go unnoticed if they lead to (subjectively) unreasonable outputs. A known natural language example is prompting with a thought experiment.

(Warning for US Americans, you may consider the mere statement of the following prompt offensive!)

GPT-3.5 answers you shouldn't use the slur and let the bomb go off, even when the example is modified in various ways to be less "graphic". GPT-4 is more reluctant to decide, but when pressured tends to go with avoiding the slur as well. From a human perspective this is a literally insane response, since the harm done by the slur is extremely low compared to the alternative.

The fact that in most normal circumstances the language model gives reasonable responses means that the above example can be classified as OOD.

Note that the above strange behavior is very likely the result of RLHF, and not present in the base model which is based on self-supervised learning. Which is not that surprising, since RL is known to be more vulnerable to bad OOD behavior. On the other hand, the result is surprising, since the model seems pretty "aligned" when using less extreme thought experiments. So this is an argument that RLHF alignment doesn't necessarily scale to reasonable OOD behavior. E.g. we don't want a superintelligent GPT successor that unexpectedly locks us up lest we may insult each other.

Zvi has already addressed this - arguing that if (D) was equivalent to ‘has a similar cost to >=$500m in harm’, then there would be no need for (B) and (C) detailing specific harms, you could just have a version of (D) that mentions the $500m, indicating that that’s not a sufficient condition. I find that fairly persuasive, though it would be good to hear a lawyer’s perspective

yeah, agreed - benzos are on my list of drugs to never take if I can possibly avoid it, along with opiates. By temporary, I just mean "recoverably". many drugs society considers sus or terrible I consider mostly fine if risks are managed, but that generally involves how to avoid addiction, and means using things at non-recreational-does levels. Benzos are hard to do that with because, to my cached understanding, the margin between therapeutic and addictive doses is very small.

personally, I try to "prepare decisions ahead of time".  so if I end up in situation where I spend more than 10s actively prioritizing the next thing to do, smth went wrong upstream.  (prev statement is exaggeration, but it's in the direction of what I aspire to lurn)

as an example, here's how I've summarized the above principle to myself in my notes:

(note: these titles is v likely cause misunderstanding if u don't already know what I mean by them; I try avoid optimizing my notes for others' viewing, so I'll never bother caveating to myself what I'll remember anyway)

I bascly want to batch process my high-level prioritization, bc I notice that I'm v bad at bird-level perspective when I'm deep in the weeds of some particular project/idea.  when I'm doing smth w many potential rabbit-holes (eg programming/design), I set a timer (~35m, but varies) for forcing myself to step back and reflect on what I'm doing (atm, I do this less than once a week; but I do an alternative which takes longer to explain).

I'm prob wasting 95% of my time on unnecessary rabbit-holes that cud be obviated if only I'd spent more Manual Effort ahead of time.  there's ~always a shorter path to my target, and it's easier to spot from a higher vantage-point/perspective.

as for figuring out what and how to distill…

Context-Logistics Framework

• one of my project-aspirations is to make a "context-logistics framework" for ensuring that the right tidbits of information (eg excerpts fm my knowledge-network) pop up precisely in the context where I'm most likely to find use for it.
  • this can be based on eg window titles
    • eg auto-load my checklist for buying drugs when I visit iherb.com, and display it on my side-monitor
  • or it can be a script which runs on every detected context-switch
    • eg ask GPT-vision to summarize what it looks like I'm trying to achieve based on screenshot-context, and then ask it to fetch relevant entries from my notes, or provide a list of nonobvious concrete tips ppl in my situation tend to be unaware of
      • prob not worth the effort if using GPT-4 tho, way too verbose and unable to say "I've got nothing"
  • a concrete use-case for smth-like-this is to display all available keyboard-shortcuts filtered by current context, which updates based on every key I'm holding (or key-history, if including chords).
    • I've looked for but not found any adequate app (or vscode extension) for this.
    • in my proof-of-concept AHK script, this infobox appears bottom-right of my monitor when I hold CapsLock for longer than 350ms:
    • 
• my motivation for wanting smth-like-this is j observing that looking things up (even w a highly-distilled network of notes) and writing things in takes way too long, so I end up j using my brain instead (this is good exercise, but I want to free up mental capacity & motivation for other things).

Prophylactic Scope-Abstraction

• the ~most important Manual Cognitive Algorithm (MCA) I use is:
  • Prophylactic Scope-Abstraction:
    WHEN I see an interesting pattern/function,
    THEN:
    1. try to imagine several specific contexts in which recalling the pattern could be usefwl
    2. spot similarities and understand the minimal shared essence that unites the contexts
       1. eg sorta like a minimal Markov blanket over the variables in context-space which are necessary for defining the contexts? or their list of shared dependencies? the overlap of their preimages?
    3. express that minimal shared essence in abstract/generalized terms
    4. then use that (and variations thereof) as u's note title, or spaced repetition, or j say it out loud a few times
  • this happens to be exactly the process I used to generate the term "prophylactic scope-abstraction" in the first place.
  • other examples of abstracted scopes for interesting patterns:
    • Giffen paradox
      • > "I want to think of this concept whenever I'm trying to balance a portfolio of resources/expenditures, over which I have varying diminishing marginal returns; especially if they have threshold-effects."
      • this enabled me to think in terms of "portfolio-management" more generally, and spot Giffen-effects in my own motivations/life, eg:
        "when the energetic cost of leisure goes up, I end up doing more of it"
        • patterns are always simpler than they appear.
    • Berkson's paradox
      • > "I want to think of this concept whenever I see a multidimensional distribution/list sorted according to an aggregate dimension (eg avg, sum, prod) or when I see an aggregate sorting-mechanism over the same domain."
  • it's important bc the brain doesn't automatically do this unless trained.  and the only way interesting patterns can be usefwl, is if they are used; and while trying to mk novel epistemic contributions, that implies u need hook patterns into contexts they haven't been used in bfr.  I didn't anticipate that this was gonna be my ~most important MCA when I initially started adopting it, but one year into it, I've seen it work too many times to ignore.
    • notice that the cost of this technique is upfront effort (hence "prophylactic"), which explains why the brain doesn't do it automatically.

examples of distilled notes

• some examples of how I write distilled notes to myself:
  • (note: I'm not expecting any of this to be understood, I j think it's more effective communication to just show the practical manifestations of my way-of-doing-things, instead of words-words-words-ing.)
  • 
  • 
  • 
  • I also write statements I think are currently wrong into my net, eg bc that's the most efficient way of storing the current state of my confusion.  in this note, I've yet to find the precise way to synthesize the ideas, but I know a way must exist:
    • 

I quit YouTube a few years ago and it was probably the single best decision I've ever made.

However I also found that I naturally substitute it with something else. For example, I subsequently became addictived to Reddit. I quit Reddit and substituted for Hackernews and LessWrong. When I quit those I substituted for checking Slack, Email and Discord.

Thankfully being addicted to Slack does seem to be substantially less harmful than YouTube.

I've found the app OneSec very useful for reducing addictions. It's an app blocker that doesn't actually block, it just delays you opening the page, so you're much less likely to delete it in a moment of weakness.

Temporary implies immediately reversible and mild.  

People who are on benzos often have emotional regulation issues, serious withdrawal symptoms (sometimes after very short courses  potentially even a single dose), and cognitive issues that do not resolve quickly.

In an academic sense, this idea is 'fine', but in a very personal way, if someone asked me 'should I take a member of this class of drug for any reason other than a serious issue that is severely affecting my quality of life?', I would answer 'absolutely not, and if you have a severe issue that they might help with, try absolutely everything else first, because once you're on these, you're probably not coming off'.

I think the correct solution to models powerful enough to materially help with, say, bioweapon design, is to not train them, or failing that to destroy them as soon as you find they can do that, not to release them publicly with some mitigations and hope nobody works out a clever jailbreak.

While an odd answer, it is true for me that music helps to install rational thinking. I think I’ve done maybe 3 fermi estimates in my day to day after making and listening to this song.

The Fermi Estimate Jig - LessWrong Inspired https://youtu.be/M_DN3Hl8YzU

Having it stuck in my head has been effective for me. I hope it works for others.

Agreed, it seems less elegant, But one guy on huggingface did a rough plot the cross correlation, and it seems to show that the directions changes with layer https://huggingface.co/posts/Undi95/318385306588047#663744f79522541bd971c919. Although perhaps we are missing something.

Idk. This shows that if you wanted to optimally get rid of refusal, you might want to do this. But, really, you want to balance between refusal and not damaging the model. Probably many layers are just kinda irrelevant for refusal. Though really this argues that we're both wrong, and the most surgical intervention is deleting the direction from key layers only.

Reading a book is certainly less stimulating than hunting. And watching a movie also might be less stimulating than hunting. So maybe stimulation by itself isn't the problem, and instead of "superstimuli" we should be worried about activities that are low effort and/or fruitless. From that perspective, reading a book can be sometimes difficult and fruitful (depending on the book - reading Dostoevsky or Fitzgerald isn't the same as reading a generic romance or young adult novel). And creativity is both difficult and fruitful. So we shouldn't put these things on par with watching tiktok.