LessWrong 2.0 Reader

I really like it! One remark, though: two upper tracks must be swapped, otherwise it's possible to precommit by staying in place and not running to the lever.

TBH one of the things I always wonder about is not so much the "sidesism" as the "both." How are people deciding what should count as a side, and why there should be two? And when should something no longer count as a side? 

I mean, I get it in practice, there's nothing this self-reflective going on at all and it's all decided by inertia, FPTP voting, and revenue. I still would naively have expected more people on the audience side to have the realization that:

this didn’t seem to fit into the Headmaster’s list; and it occurred to Hermione that there might be a lot more viewpoints on the subject than just four.

No, it was a lot of words that describe why your strategy of modelling stuff as more/less "dangerous" and then trying to calibrate to how much to be scared of "dangerous" stuff doesn't work.

The better strategy, if you want to pursue this general line of argument, is to make the strongest argument you can for what makes e.g. Bitcoin so dangerous and how horrible the consequences will be. Then since your sense of danger overestimates how dangerous Bitcoin will be, you can go in and empirically investigate where your intuition was wrong by seeing what predictions of your intuitive argument failed and what obstacles caused them to fail.

That was a lot of words to say "I don't think anything can be learned here".

Personally, I think something can be learned here.

You shouldn't use "dangerous" or "bad" as a latent variable because it promotes splitting. MAD and Bitcoin have fundamentally different operating principles (e.g. nuclear fission vs cryptographic pyramid schemes), and these principles lead to a mosaic of different attributes. If you ignore the operating principles and project down to a bad/good axis, then you can form some heuristics about what to seek out or avoid, but you face severe model misspecification, violating principles like realizability which are required for Bayesian inference to get reasonable results (e.g. converge rather than oscillate, and be well-calibrated rather than massively overconfident).

Once you understand the essence of what makes a domain seem dangerous to you, you can debug by looking at what obstacles this essence faced that stopped it from flowing into whatever horrors you were worried about, and then try to think through why you didn't realize those obstacles ahead of time. As you learn more about the factors relevant in those cases, maybe you will learn something that generalizes across cases, but most realistically what you learn will be about the problems with the common sense.

I think the issue here is not so much the disagreement or criticism as it is the mockery and ostracism. Unlike in, say, venture capital, there's much less opportunity in science for someone to try something different and exciting, get enough funding to see if it really works out, and then, if it doesn't but you were doing a good enough job trying, still be part of the community and get funding to try something else. (Yes, I know it doesn't always work that way either, but I think the odds are much better than in science)

I feel that this post would benefit from having the math spelled out. How is inserting a trader a way to do feedback? Can you phrase classical RL like this?

MAD is obviously governed by completely different principles than crypto is

Maybe this is obvious to you.  It is not obvious to me. I am genuinely confused what is going on here.  I see what seems to be a pattern: dangerous domain -> basically okay.  And I want to know what's going on.

I appreciate the discussion, but I can't help but be distracted by the specifics of the example scenario. In this case, it just seems obvious to me that the correct answer is to bury the waste and then invest in developing better processing solutions. There's no such thing as waste that can't be safely processed, even in principle, with a century of lead time to prepare. When I read the first few sentences, I actually thought the counterargument was going to be about uncertainty in long term impact projections.

• Here are the other GDM mech interp papers missed:
  • https://arxiv.org/abs/2307.15771
  • https://arxiv.org/abs/2404.16014
  • https://arxiv.org/abs/2407.14435
• We have some blog posts of comparable standard to the Anthropic circuit updates listed:
  • https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/full-post-progress-update-1-from-the-gdm-mech-interp-team [AF · GW]
  • https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall [AF · GW]
• You use a very wide scope for the "enhancing human feedback" (basically any post-training paper mentioning 'align'-ing anything). So I will use a wide scope for what counts as mech interp and also include:
  • https://arxiv.org/abs/2401.06102
  • https://arxiv.org/abs/2304.14767
  • There are a few other papers from the PAIR group as well as Mor Geva and also Been Kim, but mostly with Google Research affiliations so it seems fine to not include these as IIRC you weren't counting pre-GDM merger Google Research/Brain work