LessWrong 2.0 Reader

But anyway, it sometimes seems to me that you often advocate a morality regarding AI relations that doesn't benefit anyone who currently exists, or, the coalition that you are a part of. This seems like a mistake. Or worse.

I dispute this, since I've often talked explicitly about how I think hastening the arrival AI could benefit people who currently exist. Indeed, that's one of the best arguments for accelerating AI. The argument is that, by ensuring AI arrives sooner, we can accelerate the pace of medical progress, among other useful technologies. This could ensure that currently-existing old people who would otherwise die without AI will be saved and live a longer and healthier life than the alternative.

(Of course, this must be weighed against concerns about AI safety. I am not claiming that there is no tradeoff between AI safety and acceleration. Rather, my point is that, despite the risks, accelerating AI could still be the preferable choice.)

However, I do think there is an important distinction here to make between the following groups:

1. The set of all existing humans
2. The human species itself, including all potential genetic descendants of existing humans

Insofar as I have loyalty towards a group, I have much more loyalty towards (1) than (2). It's possible you think that I should see myself as belonging to the coalition comprised of (2) rather than (1), but I don't see a strong argument for that position. 

To the extent it makes sense to think of morality as arising from game theoretic considerations, there doesn’t appear to be much advantage for me in identifying with the coalition of all potential human descendants (group 2) rather than with the coalition of currently existing humans (group 1) plus potential future AIs. If we are willing to extend our coalition to include potential future beings, then I would seem to have even stronger practical reasons to align myself with a coalition that includes future AI systems. This is because future AIs will likely be far more powerful than any potential biological human descendants.

I want to clarify, however, that I don't tend to think of morality as arising from game theoretic considerations. Rather, I mostly think of morality as simply an expression of my personal preferences about the world.

Thought on short timelines. Opinionated.

I think that AGI timelines might be very short based on an argument taken from a different side of things.

We all can agree that humans have general intelligence. If we look at how our general intelligence evolved from simpler forms of specific intelligence typical for animals - it wasn't something that came from complex interactions and high evolutional pressure. Basically there were two aspects of that progress. The first one is the ability to pass on knowledge through generations (culture). Something that we share with some other animals including our cousins chimpanzee. The second one is intersexual selection - at some moment in the past, our species started to have sexual preferences based on the ability to gossip and talk. It is still there, even if we are not 100% aware of that - our courtship, known as dating, is based mostly on meeting together and talking. People who are not talkative and introverts, even if successful, have a hard time dating.
These two things seem to be major drivers for us to both develop more sophisticated language and better general intelligence.

It seems to me that this means that there are not many pieces missing from using current observations and some general heuristics like animals do, to have full-fledged general intelligence.

It also suggests that you need some set of functions or heuristics, possibly a small set, together with a form of external memory, to tackle any general problem by dividing it into smaller bits and rejoining sub-solutions into a general solution. Like a processor or Turing machine that has a small set of basic operations, but can in principle run any program.

Oh, my bad, yeah. When I was writing the comment, I flipped the direction of advantage for longer time controls (longer time controls are actually better for humans in odds matches of course), but this way I agree it's unclear a priori whether 200 elo drop would be enough to account for longer time controls.

I think this comment is either incredibly stupid or extremely insightful and borderline genius. I honestly can't tell (although I'm leaning towards the former, and the karma seems to agree with me), and that scares me just a little.

Thank you for another beautiful essay at real thinking! This time about the mental stance itself.

But I’ll describe a few tags I’m currently using, when I remind myself to “really think.” Suggestions/tips from readers would also be welcome.

I think there is a strong conceptual overlap with what John Vervaeke describes as Relevance Realisation and wisdom. 

I'll attempt a summary of my understanding of John Vervaeke's Relevance Realisation. 

A key capability of any agentic/living beings is to prune the exponentially exploding space of possibilities when making any decision or thought. We are computationally bounded, and how we deal with that is crucial. Vervaeke terms the process that does this Relevance Realisation.

There is a lot of detail to his model, but let's jump to how some of this plays out in human thinking: A core aspect in how we are agentic is our use of memeplexes that form an "agent-arena-relationship" - we combine a worldview with an agent that is suited to that world and then tie our identity to that agent. We build toy versions all the time (many games are like this), but --according to Vervaeke's theses in the "Awakening from the Meaning Crisis" lectures-- modern western culture has somewhat lost track of the cultural structures which allow individuals to coordinate on and (healthily) grow their mind's agent-arena-relationship. We have institutions of truth (how to reach good factual statements rather than falsehoods), but not of wisdom (how to live with a healthy stance towards reality rather than bullshit. Historically religious institutions played that role but they now successfully do that for fewer and fewer people)

Inhabiting a functional such relationship feels alive and vibrant ("inhabit" = tying one's identity into these memes), whereas the lack of a functional agent-arena-relationship feels dream-like (zombies are a good representation; maybe "NPC" is a more recent meme that points at this).

A related thing is people having a spiritual experience: this involves glimpsing a new agent-arena-relationship which then sometimes gets nourished into a re-structuring of one's self-concept and priorities.

tying this back to "real thinking"

Although the process I described is not the same thing as real thinking, I do think that there are important similarities.

Regarding how to do this well, one important point of Vervaeke's is that humans necessarily enter the world with very limited concepts of "self", "agent" or "arena". This perspective makes it clear that a core part of what we do while growing up is refining these concepts. A whole lot of our nature is about doing this process of transcending our previous self-concept. Verveake loves the quote "the sage is to the adult as the adult is to the child" to point at what the word wisdom means.

The process according to his recommendations, involves 

• a Community of Practice (a community of people who share the goal of re-orienting themselves. One such is forming with and around him at awakentomeaning.com),
• an Ecology of Practices (meditation, reflecting on philosophy, embodiment practices, etc. are all contributing factors to reducing self-delusion; the term practice emphasizes that it is about a regular activity rather than about fixed beliefs). A good way to get a feel for his perspective might be the first episode(s) of his After Socrates youtube series

and from my impression, a lot of this is practicing the ability of inhabiting/changing the currently active agent-arena-perspective, exploring its boundaries, not flinching away from noticing its limitations, engaging with the perspectives which others have built and so on. Generally, a kind of fluidity in the mental motions which are involved in this.

I hope my descriptions are sufficiently right to give an impression of his perspective and whether there are some ideas that are valuable to you :)

It would mean that R1 is actually more efficient and therefore more advanced that o1, which is possible but not very plausible given its simple RL approach.

I think that is very plausible. I don't think o1 or even r1 for that matter is anywhere near as efficient as LLMs can be. OpenAI is probably putting a lot more resources to get to AGI first, than to get to AGI efficiently. Deepseek v3 is already miles better than GPT 4o while being cheaper.

I think it's more likely that o1 is similar to R1-Zero (rather than R1), that is, it may mix languages which doesn't result in reasoning steps that can be straightforwardly read by humans. A quick inference time fix for this is to do another model call which translates the gibberish into readable English, which would explain the increased CoT time.

I think this is extremely unlikely. Here's some questions that demonstrate why.
Do you think OpenAI is using a model to first translate to English, and then another model to generate a summary? Is this conditional on showing the translated CoTs being a feature going forwards? If so, do you expect OpenAI to do this for all CoTs or just the CoTs they intend to show? If the latter, don't you think there will be a significant difference in the "thinking" time between the responses where the translated CoT is visible and the responses where only the summary is visible?

Thank you for being patient with me, I tend to live in my own head a bit with these things :/ Let me know if this explanation is clearer using the examples you gave:

Let me build on the discussion about optimization slack and sharp left turns by exploring a concrete example that illustrates the key dynamics at play.

Think about the difference between TD-learning and Monte Carlo methods in reinforcement learning. In TD-learning, we update our value estimates frequently based on small temporal differences between successive states. The "slack" - how far we let the system explore/optimize between validation checks - is quite tight. In contrast, Monte Carlo methods wait until the end of an episode to make updates, allowing much more slack in the intermediate steps.

This difference provides insight into the sharp left turn problem. When we allow more slack between optimization steps (like in Monte Carlo methods), the system has more freedom to drift from its original utility function before course correction. The divergence compounds particularly when we have nested optimization processes - imagine a base model with significant slack that then has additional optimization layers built on top, each with their own slack. The total divergence potential multiplies.

This connects directly to your point about GPT-style models versus AlphaZero. While the gradient descent itself may have tight feedback loops, the higher-level optimization occurring through prompt engineering or fine-tuning introduces additional slack. It's similar to how cultural evolution, with its long periods between meaningful corrections, allowed for the emergence of inner optimizers that could significantly diverge from the original selection pressures.

I'm still working to formalize precisely what mathematical structure best captures this notion of slack - whether it's best understood through the lens of utility boundaries, free energy, or some other framework.

Therefore, it's very likely that OpenAI is sampling the best ones from multiple CoTs (or CoT steps with a tree search algorithm), which are the ones shown in the screenshots in the post.

I find this unlikely. It would mean that R1 is actually more efficient and therefore more advanced that o1, which is possible but not very plausible given its simple RL approach. I think it's more likely that o1 is similar to R1-Zero (rather than R1), that is, it may mix languages which doesn't result in reasoning steps that can be straightforwardly read by humans. A quick inference time fix for this is to do another model call which translates the gibberish into readable English, which would explain the increased CoT time. The "quick fix" may be due to OpenAI being caught off guard by the R1 release.

If a spy slips a piece of paper to his handler, and then the counter-espionage officer arrests them and gets the piece of paper, and the piece of paper just says “85”, then I don’t know wtf that means, but I do learn something like “the spy is not communicating all that much information that his superiors don’t already know”.

By the same token, if you say that humans have 25,000 genes (or whatever), that says something important about how many specific things the genome designed in the human brain. For example, there’s something in the brain [LW · GW] that says “if I’m malnourished, then reduce the rate of the (highly-energy-consuming) nonshivering thermogenesis process”. It’s a specific innate (not learned) connection between two specific neuron groups in different parts of the brain, I think one in the arcuate nucleus of the hypothalamus, the other in the periaqueductal gray of the brainstem (two of many hundreds or low-thousands of little idiosyncratic cell groups in the hypothalamus and brainstem). There’s nothing in the central dogma of molecular biology, and there’s nothing in the chemical nature of proteins, that makes this particular connection especially prone to occurring, compared to the huge number of superficially-similar connections that would be maladaptive (“if I’m malnourished, then get goosebumps” or whatever). So this connection must be occupying some number of bits of DNA—perhaps not a whole dedicated protein, but perhaps some part of some protein, or whatever. And there can only be so many of that type of thing, given a mere 25,000 genes for the whole body and everything in it.

That’s an important thing that you can learn from the size of the genome. We can learn it without expecting aliens to be able to decode DNA or anything like that. And Archimedes’s comment above doesn’t undermine it—it’s a conclusion that’s robust to the “procedural generation” complexities of how the embryonic development process unfolds.

links 01/29/25: https://roamresearch.com/#/app/srcpublic/page/01-29-2025

• https://marginalrevolution.com/marginalrevolution/2025/01/its-time-to-build-the-peptidome.html  link of a piece I've already read by Maxwell Tabarrok about the value of building a dataset of peptides and their sequences, structures, and properties, for training models that can discover peptides that can be used pharmacologically (for instance, as antimicrobials).
• https://www.biorxiv.org/content/10.1101/2025.01.24.634830v1 using modified T-regulatory cells can be turned against the macrophage foam cells that produce atherosclerosis plaque, reducing heart disease (in mice).
  • how do they do it?
    • oxidized LDL is a marker of inflammation and atherosclerosis. It's also a protein (the acronym is for Low Density Lipoprotein) so you can target it with an antibody!
    • regulatory T cells (Treg) genetically modified to contain one of these anti-OxLDL antibodies prevents the OxLDL from getting into macrophages and turning them into foam cells.
    • inject them into live mice on a high-fat diet and they also get less atherosclerosis than control mice
    • interestingly they make Tregs from CD4 (helper) T-cells by getting them to express FOXP3. this will be useful for human clinical application, since natural Tregs are rare and it can be difficult to extract them in sufficient quantity.
• https://www.wired.com/story/elon-musk-lackeys-office-personnel-management-opm-neuralink-x-boring-stalin/
• https://talentmarket.org/cato-psych-policy-analyst/  ugh. they basically have already written the bottom line on what they want a new hire to think about the "psychology of progress", and it's mostly talking points that have been made ad infinitum already.
  • surely if you were serious about overcoming psychological barriers in the general public that make them ill-disposed to objectively beneficial economic/technological changes, you'd want to look for new ideas (since clearly past ones haven't worked). you'd also want to focus on starting with empathy and common ground, if the hope is to actually change minds. this is a job description that, ironically, guarantees stasis, not progress.
• https://stephenmalina.com/post/2023-11-04-biologizing-the-stack/ I'm glad he admits this is a contrarian position because the straightforward lesson of the past several decades is that you want as few things to be biomanufactured as possible, living things make incredibly cost-inefficient factories and should be a last resort.
• https://unstableontology.com/2022/05/02/on-the-paradox-of-tolerance-in-relation-to-fascism-and-online-content-moderation/ I like this post, but on the general topic, how could you possibly implement Popper's standard of "we must not tolerate speech that incites people to not listen to arguments"? That would, itself, require an extensive censorship apparatus.
  • 1st Amendment Law seems much more practical in protecting all political opinions and having exceptions only for "true threats" or narrow "incitement to violence". Saying "don't read this book, it's by a fascist" is intolerance by Popper's definition, but it's straightforwardly protected speech under US law and I think it should be; we don't trust a government agency to adjudicate questions of epistemic vice.
  • 1st Amendment law has almost a refreshingly nihilistic attitude to discussion -- all "opinion", including iirc all discussion on social media, is assumed to be neither true or false, so it can't be considered defamatory. basically, under the law, "this is all just yapping, people get to yap, call me when they make a false factual claim that costs you money, or make an actual plan to physically hurt someone". sometimes, in addition to being a safer standard for one's government to hold, this is a healthy attitude to adopt oneself!
• Stephen Wolfram on machine learning: https://writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/
  • first, we visualize how values at the nodes of a neural network (during inference) change based on the inputs. complicated!
  • then we look at how weights on edges change as a neural network is trained. also complicated! though you can see the changes getting smaller as the network approaches convergence.
  • a mesh neural net -- each node is only connected to its neighbors in a grid -- is a cellular automaton. Cellular automata are a special case of neural networks, in other words.
  • you can do an analogue of "training a model to fit a function" with something called a "rule array" -- you have a grid, and each square can have a local update rule relative to its neighbors, cellular-automata style, but the rule might be different for different squares. If your "input" is a black square at the top row, then "running" the automata rule repeatedly may propagate down a black-and-white pattern on the grid. you can then "adapt" the rule array iteratively to get a desired pattern, like "generate one that lasts for exactly 30 steps". or a function: you want to get something that gives certain "outputs" on the bottom row, given an "input" at the top row. then "mutate" cells at random, keeping only the results where the distance from the "training examples" doesn't change.
    • you can observe which cells mutate along the training runs; they seem to be the ones along the "ideal path" between x (at the top) and f(x) (at the bottom).
    • doing mutations at random is inefficient; you can do an equivalent of "steepest descent" in cellular automata too, but this tends to get stuck
    • there's an equivalent of "backpropagation" in automata-land too
    • in general, why translate to automata? he says they're easier to "inspect" because simpler, but I kinda don't get it.
    • he even builds a discrete analog of a transformer!
    • "It could have been that machine learning would somehow “crack systems”, and find simple representations for what they do. But that doesn’t seem to be what’s going on at all. Instead what seems to be happening is that machine learning is in a sense just “hitching a ride” on the general richness of the computational universe. It’s not “specifically building up behavior one needs”; rather what it’s doing is to harness behavior that’s “already out there” in the computational universe."
      • yep this is a convergent idea. the secret sauce in machine learning is just having a rich enough space of possible functions, and a means of variation and selection. you can get similar results using things like genetic algorithms or cellular automata that aren't "neural networks" at all.
  •  
• Stephen Wolfram on formal verification: https://writings.stephenwolfram.com/2025/01/who-can-understand-the-proof-a-window-on-formalized-mathematics/
  • Wolfram Language allows proof verification!
  • you can also visualize the dependency graph of lemmas.
    • sometimes the "high-degree nodes" are legible things like commutativity
    • but often they, and "commonly used lemmas" in randomly generated proofs, are weird elaborate non-human-interpretable things.
  • What about proof-to-proof equivalences?
    • this is where the "homotopy" metaphors come from. you could find a "path" from one proof to another...but what if there are "holes" in proof-space? "Then a “continuous deformation” of one proof into another will get stuck, and even if there is a much shorter proof, we’re liable to get “topologically stuck” before we find it."
    •  
• https://kmill.github.io/informalization/ucsc_cse_talk.pdf
  • nice explanation of formalization (in Lean), auto-formalization and auto-informalization
  • if you just throw ChatGPT at this, it sucks.
  • so let's do a little ontology hard-coding.
• https://moreisdifferent.blog/p/german-scientific-paternalism how Germany in the late 19th-early 20th century trained scientists
• https://kordinglab.com/about/  an approach to figuring out what neurons do by mapping between simulations, electrophysiology data, and psychophysics. "what algorithm is being implemented here?"
• https://www.hypothesisfund.org/ a Reid Hoffman project: funds breakthrough research, mostly life sciences.
  • this doesn't "smell" aggressive enough to me -- the projects look fine but i'm surprised they'd be unfundable elsewhere -- but maybe it's just an insufficiently pointed communication style.
• https://en.wikipedia.org/wiki/Robert_Duggan_(venture_capitalist)trange fellow. apparently he's a Scientologist and knows nothing about biology but has a knack for picking winners. picked up ibrutinib!!!