LessWrong 2.0 Reader

Rather, they didn't foresee the possibility that Microsoft might want to invest. And they didn't consider that capped-for-profit was a path to billions of dollars.

Do you think wants that arise from conscious thought processes are equally valid to wants that arise from feelings? How do you think about that?

Reminds me of "Self-Integrity and the Drowning Child [LW · GW]" which talks about another kind of way that people in EA/rat communities are liable to hammer down parts of themselves. 

Napoleon is merely an argument for "just because you strongly believe it, even if it is a statement about you, does not necessarily make it true".

We will probably disagree on this, but the only reason I care about trans issues is that some people report significant suffering (gender dysphoria) from their current situation, and I am in favor of people not suffering, so I generally try not to be an asshole.

Unfortunately, for every person who suffers from something, there are probably dozen people out there who cosplay their condition... because it makes them popular on Twitter I guess, or just gives them another opportunity to annoy their neighbors. I have no empathy for those. Play your silly games, if you wish, but don't expect me to play along, and definitely don't threaten me to play along. Also, the cosplayers often make the situation more difficult for those who genuinely have the condition, by speaking in their name, and often saying things that the people who actually have the condition would disagree with... and in the most ironic cases, the cosplayers get them cancelled. So I don't mind being an asshole to the cosplayers, because from my perspective, they started it first.

The word "deadnaming" is itself hysterical. (Who died? No one.)

Gender essentialism? I don't make any metaphysical claim about essences. People simply are born with male or female bodies (yes, I know that some are intersex), and some people are strongly unhappy about their state. I find it plausible that there may be an underlying biological reason for that; and hormones seem like a likely candidate, because that's how body communicates many things. I don't have a strong opinion on that, because I have never felt a desire to be one sex or the other, just like I have never felt a strong desire to have a certain color of eyes, or hair, or skin, whether it would be the one I have or some that I have not.

I expect that you will disagree with a lot of this, and that's okay; I am not trying to convince you, just explaining my position.

The key is recognizing that the preference itself is completely independent from rationality or intelligence.

The orthogonality thesis is also for human beings.

I don't see how your question contradicts my statement, nor that link. People absolutely develop in their desires over time, and can change them, but that is not the same as being able to decide, in the moment, that you do not like the taste of pizza if your tongue is having the sensory experience of enjoying it.

Feels connected to his distrust of "quick, bright, standardized, mental processes", and the obsession with language. It's like his mind is relentlessly orienting to the territory, refusing to accept anyone else's map. Which makes it harder to be a student but easier to discover something new. Reminds me of Geoff Hinton's advice to not read the literature before engaging with the problem yourself.

while this paradigm of 'training a model that's an agi, and then running it at inference' is one way we get to transformative agi, i find myself thinking that probably WON'T be the first transformative AI, because my guess is that there are lots of tricks using lots of compute at inference to get not quite transformative ai to transformative ai.

my guess is that getting to that transformative level is gonna require ALL the tricks and compute, and will therefore eek out being transformative BY utilizing all those resources.

one of those tricks may be running millions of copies of the thing in an agentic swarm, but i would expect that to be merely a form of inference time scaling, and therefore wouldn't expect ONE of those things to be transformative AGI on it's own.

and i doubt that these tricks can funge against train time compute, as you seem to be assuming in your analysis.  my guess is that you hit diminishing returns for various types of train compute, then diminishing returns for various types of inference compute, and that we'll get to a point where we need to push both of them to that point to get tranformative ai

testing markdown editor

hello

Here's the content restructured as a numbered list with inline links:

1. Logical Induction and Embedded Agency - Explore Scott Garrabrant and Abram Demski's cornerstone work including the Embedded Agents sequence [LW · GW] and research on Logical Induction [LW · GW].

2. Learning-Theoretic Approach - Study Vanessa Kosoy's agenda for creating a foundational mathematical theory of agency, detailed in her interview on theoretical research [LW · GW].

3. Selection Theorems - Explore John Wentworth's research agenda on Natural Abstraction Hypothesis and system modularity through his research overview.

4. Decision Theory - Read about idealized decision theory and reflective oracles in agent foundations work [LW · GW].

5. Cartesian Frames - Study Scott Garrabrant's mathematical framework for agent-environment distinctions in the embedded agency guide.

6. Additional Research Areas - Discover various technical alignment approaches through the comprehensive overview [AF · GW] of the field.

It was gluten.