LessWrong 2.0 Reader

oh super helpful, thanks!

I believe it's 2 hours of sun exposure. So unless you are spending all day outside, you should only need to apply it once. I personally apply it once before going to work. 

So regarding things that involve active prioritizing of compute resources, I think that would fairly clearly fall no longer under epistemic rationality. Because "spending compute resources on this rather than that" is an action, which are only part of instrumental rationality. So in that sense it wouldn't be part of intelligence. Which makes some sense given that intuitively smart people often concentrate their mental efforts on things that are not necessarily very useful to them.

This relates also to what you write about level 1 and 2 compared to level 3. In the first two cases you mention actions, but not in the third. Which makes sense if level 3 is about epistemic rationality. Assuming level 1 and 2 are about instrumental rationality then, this would be an interesting difference to my previous conceptualization: On my picture, epistemic rationality was a necessary but not sufficient condition for instrumental rationality, but on your picture, instead level 1 and 2 (~instrumental rationality) are a necessary but not sufficient condition for level 3 (~epistemic rationality). I'm not sure what we can conclude from these inversed pictures.

I think of generality of intelligence as relatively conceptually trivial. At the end of the day, a system is given a sequence of data via observation, and is now tasked with finding a function or set of functions that both corresponds to plausible transition rules of the given sequence, and has a reasonably high chance of correctly predicting the next element of the sequence

Okay, but terminology-wise I wouldn't describe this as generality. Because the narrow/general axis seems to have more to to with instrumental rationality / competence than with epistemic rationality / intelligence. The latter can be described as a form of prediction, or building causal models / a world model. But generality seems to be more about what a system can do overall in terms of actions. GPT-4 may have a quite advanced world model, but at its heart it only imitates Internet text, and doesn't do so in real time, so it can hardly be used for robotics. So I would describe it as a less general system than most animals, though more general than a Go AI.

Regarding an overall model of cognition, a core part that describes epistemic rationality seems to be captured well by a theory called predictive coding or predictive processing. Scott Alexander has an interesting article about it. It's originally a theory from neuroscience, but Yann LeCun also sees it as a core part of his model of cognition. The model is described here on pages 6 to 9. Predictive coding is responsible for the part of cognition that he calls the world model.

Basically, predictive coding is the theory that an agent constantly does self-supervised learning (SSL) on sensory data (real-time / online) by continuously predicting its experiences and continuously updating the world-model depending on whether those predictions were correct. This creates a world model, which is the basis for the other abilities of the agent, like creating and executing action plans. LeCun calls the background knowledge created by this type of predictive coding the "dark matter" of intelligence, because it includes fundamental common sense knowledge, like intuitive physics.

The current problem is that currently self-supervised learning only really works for text (in LLMs), but not yet properly for things like video. Basically the difference is that with text we have a relatively small number of discrete tokens with quite low redundancy, while for sensory inputs we have basically continuous data with a very large amount of redundancy. It makes no computational sense to predict probabilities of individual frames of video data like it makes sense for an LLM to "predict" probabilities for the next text token. Currently LeCun tries to make SSL work for these types of sensory data by using his "Joint Embedding Predictive Architecture" (JEPA), described in the paper above.

To the extent that creating a world model is handled by predictive coding, and if we call the ability to create accurate world models "epistemic rationality" or "intelligence", we seem to have a pretty good grasp of what we are talking about. (Even though we don't yet have a working implementation of predictive coding, like JEPA.)

But if we talk about a general theory of cognition/competence/instrumental rationality, the picture is much less clear. All we have is things like LeCun's very coarse model of cognition (pages 6ff in the paper above), or completely abstract models like AIXI. So there is a big gap in understanding what the cognition of a competent agent even looks like.

Yeah, many people, like the majority of users on this forum, have decided to not build AGI.

Not to build AGI yet. 
Many of us would love to build it as soon as we can be confident we have a realistic and mature plan for alignment, but that's a problem that's so absurdly challenging that even if aliens landed tomorrow and handed us the "secret to Friendly AI", we would have a hell of a time trying to validate that it actually was the real thing.

If one is faced with a math problem where you could be staring at the answer and know no way to unambiguously verify said answer, you are likely not capable of solving the problem until you somehow close the inferential distance separating you from understanding. Assuming the problem is solvable at all.

It says to reapply every two hours but I... do not do that 😅 I put it on in the morning and would reapply if I was spending time outside. I don't know how important the "every 2 hours" thing is

How long does the Elta MD sunscreen last?

I think all criticism, all shaming, all guilt tripping, all punishments and rewards directed at children - is for the purpose of driving them to do certain actions. If your children do what you think is right, there's no need to do much of anything.

A more general and correct statement would be "Pain is for the sake of change, and all change is painful". But that change is for the sake of actions. I don't think that's too much of a simplification to be useful.

I think regret, too, is connected here. And there's certainly times when it seems like pain is the problem rather than an attempt to solve it, but I think that's a misunderstanding. And while chronic pain does reduce agency, it's a constant pain and a constant reduction of agency (not cumulative). The pain persists until the problem is solved, even if the problem does not get worse. So it's the body telling the brain "Hey, do something about this, the importance is 50 units of pain", then you will do anything to solve it as long there's a path with less than 50 units of pain which leads to a solution.

The pain does limit agency, but not because it's a real limitation. It's an artificial one that the body creates to prevent you from damaging yourself. So all important agency is still possible. If the estimated consequences of avoiding the task is more painful than doing the task, you do it. But it's again the body is just estimating the cost/benefit of tasks and choosing the optimal action by making it the least painful action.

My explanation and yours are almost identical, but there's some important differences. In my view, 
suffering is good, not bad. I really don't want humanity to misunderstand this one fact, it has already had profound negative consequences. It's phantom damage created to avoid real damage. An agent which is unable to feel physical pain and exhaustion would destroy itself, therefore physical pain and exhaustion are valuable and not problems to be solved. Emotions like suffering, exhaustion, annoyance, etc. function the same as physical pain, and once they get over a certain threshold they coerce you into taking an action. Physical pain comes from nerves, but emotional pain comes from your interpretation of reality. Your brain relies on you to tell what ought to be painful (so if you overestimate risk, it just believes you). And you don't get to choose all your goals yourself, your brain wants you to fulfill your needs (prioritized by the hierarchy of needs). In short, the brain makes inaction painful, while keeping actions that it deems risky painful, and then messes with the weights/thresholds according to need. Just like with hunger (not eating is painful, but if all you have is stale or even moldy bread, then you need to be very hungry before you eat, and you will eat iff pain(hunger)>pain(eating the bread)).

An increase in power/agency feels a lot like happiness though, even according to Nietzsche who I'm not confident to argue against, so I get why you'd basically think that opposite of happiness is the opposite of agency (sorry if this summary does injustice to your point)

There's a lot of overlap between alignment researchers and the EA community, so I'm wondering how that was handled.

Agree that there is inherent/unavoidable overlap. As noted in the post, we were generally cautious about excluding participants from either sample for reasons you mention and also found that the key results we present here are robust to these kinds of changes in the filtration of either dataset (you can see and explore this for yourself here). 

With this being said, we did ask in both the EA and the alignment survey to indicate the extent to which they are involved in alignment—note the significance of the difference here:

From alignment survey:

From EA survey:

This question/result serves both as a good filtering criterion for cleanly separating out EAs from alignment researchers and also gives a pretty strong evidence that we are drawing on completely different samples across these surveys (likely because we sourced the data for each survey through completely distinct channels).

Regarding the support for various cause areas, I'm pretty sure that you'll find the support for AI Safety/Long-Termism/X-risk is higher among those most involved in EA than among those least involved. Part of this may be because of the number of jobs available in this cause area.

Interesting—I just tried to test this. It is a bit hard to find a variable in the EA dataset that would cleanly correspond to higher vs. lower overall involvement, but we can filter by number of years one has been involved involved in EA, and there is no level-of-experience threshold I could find where there are statistically significant differences in EAs' views on how promising AI x-risk is. (Note that years of experience in EA may not be the best proxy for what you are asking, but is likely the best we've got to tackle this specific question.)

Blue is >1 year experience, red is <1 year experience:

Blue is >2 years experience, red is <2 years experience:

Thoughtdump on why I'm interested in computational mechanics:

• one concrete application to natural abstractions from here [LW · GW]: tl;dr, belief structures generally seem to be fractal shaped. one major part of natural abstractions is trying to find the correspondence between structures in the environment and concepts used by the mind. so if we can do the inverse of what adam and paul did, i.e. 'discover' fractal structures from activations and figure out what stochastic process they might correspond to in the environment, that would be cool
  • ... but i was initially interested in reading compmech stuff not with a particular alignment relevant thread in mind but rather because it seemed broadly similar in directions to natural abstractions.
• re: how my focus would differ from my impression of current compmech work done in academia: academia seems faaaaaar less focused on actually trying out epsilon reconstruction in real world noisy data. CSSR is an example of a reconstruction algorithm. apparently people did compmech stuff on real-world data, don't know how good, but effort-wise far too less invested compared to theory work
  • would be interested in these reconstruction algorithms, eg what are the bottlenecks to scaling them up, etc.
• tangent: epsilon transducers seem cool. if the reconstruction algorithm is good, a prototypical example i'm thinking of is something like: pick some input-output region within a model, and literally try to discover the hmm model reconstructing it? of course it's gonna be unwieldly large. but, to shift the thread in the direction of bright-eyed theorizing ...
• the foundational Calculi of Emergence paper talked about the possibility of hierarchical epsilon machines, where you do epsilon machines on top of epsilon machines and for simple examples where you can analytically do this, you get wild things like coming up with more and more compact representations of stochastic processes (eg data stream -> tree -> markov model -> stack automata -> ... ?)
  • this ... sounds like natural abstractions in its wildest dreams? literally point at some raw datastream and automatically build hierarchical abstractions that get more compact as you go up
  • haha but alas, (almost) no development afaik since the original paper. seems cool
• and also more tangentially, compmech seemed to have a lot to talk about providing interesting semantics to various information measures aka True Names [LW · GW], so another angle i was interested in was to learn about them.
  • eg crutchfield talks a lot about developing a right notion of information flow - obvious usefulness in eg formalizing boundaries?
  • many other information measures from compmech with suggestive semantics—cryptic order? gauge information? synchronization order? check ruro1 and ruro2 for more.

citric acid and a polymer