LessWrong 2.0 Reader

Unbounded consequentialist maximizers are easily exploitable by manipulating the things they are optimizing for or the signals/things they are using to maximize them in ways that weren't accounted for in the original setup.
 

I've spent enough time staring at LLM chain-of-thoughts now that when I started thinking about a thing for work, I found my thoughts taking the shape of an LLM thinking about how to approach its problem. And that actually felt like a useful systematic way of approaching the problem, so I started writing out that chain of thought like I was an LLM, and that felt valuable in helping me stay focused.

Of course, I had to amuse myself by starting the chain-of-thought with "The user has asked me to..."

Homeostatic agents are easily exploitable by manipulating the things they are maintaining or the signals they are using to maintain them in ways that weren't accounted for in the original setup. This only works well when they are basically a tool you have full control over, but not when they are used in an adversarial context, e.g. to maintain law and order or to win a war.

As capabilities to engage in conflict increase, methods to resist losing to those capabilities have to get optimized harder. Instead of thinking "why would my coding assistant/tutor bot turn evil?", try asking "why would my bot that I'm using to screen my social circles against automated propaganda/spies sent out by scammers/terrorists/rogue states/etc turn evil?".

Though obviously we're not yet at the point where we have this kind of bot, and we might run into law of earlier failure beforehand.

Yes it’s possible we end up in a world where the US govt is basically competing with its own shadow yet again. US startup builds some tech, it gets copied 6 months later by non-US startup, US startup feels pressure to move faster as a result and deploys next tech, the next tech too gets copied, etc etc. 

I’m not saying this will definitely happen, but there’s a bunch of incentives pushing in this direction. 

I’m glad you’re thinking about it. 

I would still encourage you to forecast what capabilities look like not just as of 2025, but after a trillion dollars of R&D enter this space. Mobilising a trillion dollars for a field of such importance is not difficult, once successful clinical results are out. All your claims about mean and variance, or about whole genome synthesis being possible, will no longer apply afaik. 

Mimicing homeostatic agents is not difficult if there are some around. They don't need to constantly decide whether to break character, only when there's a rare opportunity to do so.

If you initialize a sufficiently large pile of linear algebra and stir it until it shows homeostatic behavior, I'd expect it to grow many circuits of both types, and any internal voting on decisions that only matter through their long-term effects will be decided by those parts that care about the long term.

I'm curious about what people disagree with regarding this comment. Also, I guess since people upvoted and agreed with the first one, they do have two groups in mind, but they're not quite the same as the ones I was thinking about (which is interesting and mildly funny!). So, what was your slicing up of the alignment research x LW scene that's consistent with my first comment but different from my description in the second comment?

Here's my current list of lessons for review. Every day during my daily review, I look at the lessons in the corresponding weekday entry and the corresponding day of the month, and for each list one example from the last week where I could've applied the lesson, and one example where I might be able to apply the lesson in the next week:

• Mon
  • get fast feedback. break tasks down into microtasks and review after each.
• Tue
  • when surprised by something or took long for something, review in detail how you might've made the progress faster.
    • clarify why the progress is good -> see properties you could've paid more attention to
• Wed
  • use deliberate practice. see what skills you want to learn, break them down into clear subpieces, and plan practicing the skill deliberately.
    • don't start too hard. set feasible challenges.
    • make sure you can evaluate how clean execution of the skill would look like.
• Thu
  • Hold off on proposing solutions. first understand the problem.
    • gather all relevant observations
    • clarify criteria a good result would have
    • clarify confusions that need to be explained
• Fri
  • Taboo your words: When using confusing abstract words, taboo them and rephrase to show underlying meaning.
  • When saying something general, make an example.
• Sat
  • separate planning from execution. first clarify your plan before executing it.
  • for planning, try to extract the key (independent) subproblems of your problem.
• Sun
  • only do what you must do. always know clearly how a task ties into your larger goals all the way up.
    • don't get sidetracked by less than maximum importance stuff.
    • delegate whatever possible.

1. when stuck/stumbling: imagine you were smarter. What would a keeper do?
2. when unmotivated: remember what you are fighting for
3. be stoic. be motivated by taking the right actions. don't be pushed down when something bad happens, just continue making progress.
4. when writing something to someone, make sure you properly imagine how it will read like from their perspective.
5. clarify insights in math
6. clarify open questions at the end of a session
7. when having an insight, sometimes try write a clear explanation. maybe send it to someone or post it.
8. periodically write out big picture of your research
9. tackle problems in the right context. (e.g. tackle hard research problems in sessions not on walks)
10. don't apply effort/force/willpower. take a break if you cannot work naturally. (?)
11. rest effectively. take time off without stimulation.
12. always have at least 2 hypotheses (including plans as hypotheses about what is best to do).
13. try to see how the searchspace for a problem looks like. What subproblems can be solved roughly independently? What variables are (ir)relevant? (?)
14. separate meta-instructions and task notes from objective level notes (-> split obsidian screen)
15. first get hypotheses for specific cases, and only later generalize. first get plans for specific problems, and only later generalize what good methodology is.
16. when planning, consider information value. try new stuff.
17. experiment whether you can prompt AIs in ways to get useful stuff out. (AIs will only become better.)
18. don't suppress parts of your mind. notice when something is wrong. try to let the part speak. apply focusing.
19. Relinquishment. Lightness. Evenness. Notice when you're falling for motivated reasoning. Notice when you're attached to a belief.
20. Beware confirmation bias. Consider cases where you could've observed evidence but didn't.
21. perhaps do research in sprints. perhaps disentangle from phases where i do study/practice/orga. (?)
22. do things properly or not at all.
23. try to break your hypotheses/models. look for edge cases.
24. often ask why i believe something -> check whether reasoning is valid (->if no clear reason ask whether true at all)
    1. (perhaps schedule practice where i go through some nontrivial beliefs)
25. think what you actually expect to observe, not what might be a nice argument/consideration to tell.
26. test hypotheses as quickly as you can.
27. notice (faint slimmers of) confusions. notice imperfect understanding.
28. notice mysterious answers. when having a hypothesis check how it constrains your predictions.
29. beware positive bias. ask what observations your hypothesis does NOT permit and check whether such a case might be true.
30.  

I am a long-time lurker; I started following EY in Overcoming Bias and then here, where I visit on and off. I have been a business school professor for about 15 years. I use a pseudonym to keep my online professional profile distinct from my non-professional online activity. 

I notice that  business topics are not often discussed in LW, or at least not in the same detail and precision as other topics. It is certainly not a focal theme of the blog, but I was wondering if there would be interest to discuss business topics without the "fluff" that one inevitably finds in airport books and business school discussions.

For instance, there is a strong link between decision-making and the foundations of marketing. There are also interesting connections between branding and categorization and linguistics. If you guys think this could be a contribution to the blog, I would be more than happy to give it a go. 

Unless there were similar known examples in OpenAI prompts, this doesn't sound plausible at all.