LessWrong 2.0 Reader

Thank you for writing this up! I've purchased some subscriptions and plan to block out time to play around with some of these services and get familiar with them.

criticism in LW comments is why he stopped writing Sequences posts

I wasn't aware of this and would like more information. Can anyone provide a source, or report their agreement or disagreement with the claim?

No. But I’m skeptical: seems hard to imagine provable safety, much less competitive with the default path to powerful AI, much less how post-hoc evals are relevant.

Out of curiosity, do you have any thoughts on the importance / feasibility of formal verification / mathematically "provable" safety based approaches in these evals you mention?

Note to readers; it is an obligatory warning on any post like this that you should not run random scripts downloaded from the internet without reading them to see what they do, because there are many harmful things they could be doing.

I'm curious what Geoff's relationship to this project is. Does he know about it? Is he contributing? Is he in favor of this? Opposed?

I would argue that the AI equivalent of these tiny organisms are "features," which are just beginning to be defined in a structured, mathematical way.

Gosh that was really beautifully written

[habryka] The way humans think about the question of "preferences for weak agents" and "kindness" feels like the kind of thing that will come apart under extreme optimization, in a similar way to how I expect the idea of "having a continuous stream of consciousness with a good past and good future is important" to come apart as humans can make copies of themselves and change their memories, and instantiate slightly changed versions of themselves, etc.

I think there will be options that are good under most of the things that "preferences for weak agents" would likely come apart into under close examination. If you're trying to fulfill the preferences of fish, you might argue about whether the exact thing you should care about is maximizing their hedonic state vs ensuring that they exist in an ecological environment which resembles their niche vs minimizing "boundary-crossing actions"... but you can probably find an action that is better than "kill the fish" by all of those possible metrics.

I think that some people have an intuition that any future agent must pick exactly one utility function over the physical configuration of matter in the universe, and that any agent that has a deontological constraint like "don't do any actions which are 0.00001% better under my current interpretation of my utility function but which are horrifyingly bad to every other agent " will be outcompeted in the long term. I personally don't see it, and particularly I don't see how there's an available slot for an arbitrary outcome-based utility function that is not "reproduce yourself at all costs" but there isn't an available slot for process-based preferences like "and don't be an asshole for miniscule gains while doing that".

Nice post. It prompts two questions, which you may or may not be the right person to answer:

1. How do you find good obsessions? Is it "just" a matter of being curious and widely-read? What is the combination of life practice and psychological orientation that leads a person to become obsessed with one or more ideas in the way that you became obsessed with progress studies and with Fieldbook?
2. On your path to world-class status, how do you avoid the "middle-competence trap" (analogy to the middle-income trap)? How do you handle having something you love that you've gotten damn good at, better than most people will ever get, but can't seem to break through to the level of the achievers who really make their mark on the field? Maybe this is more of an issue for me than for others-- maybe for example it is "just" a matter of being willing to burrow deep into something to the exclusion of your other interests in life, and I'm too much of a generalist to do that-- but it's been a problem for me twice now, and I really wonder if it might be a common failure mode of this kind of questing process.