LessWrong 2.0 Reader

• I don't think it makes that much sense to just look at cortical neuron counts. Big bodies ask for many neurons, including cortical motor neurons. Do cetaceans have really big motor cortices? Visual cortices? Olfactory bulbs? Keyword "allometry". Yes, brains are plastic, but that doesn't mean orcas are actually ever doing higher mathematics with their brains.

See this comment.

• Scale matters, but I doubt it's very close to being the only thing! Humans likely had genetic adaptations for neuroanatomical phenotypes selected-for by some of: language; tool-making; persisting transient mental content; intent-inference; intent-sharing; mental simulation; prey prediction; deception; social learning; teaching; niche construction/expansion/migration. Orcas have a few of these. But how many, how much, for how long, in what range of situations and manifestations?

I already considered this. (I just posted a question about this [LW · GW].) I don't have good information on to what extent orcas have those, but my guesses are already reflected in my overall guess in the post. 
Why do you think orcas have few of those? For me it seems plausibe that orcas have everything except tool use and niche construction.

I do think there was some significant selection for some kind of intelligence in dolphins and orcas - the main question here is whether being optimized on tool use (IF that was a significant driver in what selected humans for intelligence) would be significantly more useful for having the brain potential generalize to doing science than if the brains were optimized because of social dynamics or hunting strategies.

But of course there are other considerations like "maybe you need fully recursive language to be able to have the abstract reasoning take off, and this might very well come from some adaptations that are not just about neoron counts, and maybe orcas don't have that".

I already took all my current uncertain consideration on this into account when I said "50% that they would be superhuman at science if they had similar quality and quantity of science education as scientists and were motivated for this".

Or do you think a cow brain scaled to 40 billion neurons would be superhuman?

I don't know what you're asking for here. I don't think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.

I'm not really excited about just smushing together more brain tissue without it having been optimized to work well together, but orca brains were optimized.

• Culture matters. The Greeks could be great philosophers... But could a kid living in 8000 BCE, who gets to text message with an advanced alien civilization of kinda dumb people, become a cutting edge philosopher in the alien culture? Even though almost everyone ze interacts with is preagricultural, preliterate? I dunno, maybe? Still seems kinda hard actually?

Yep that's why I'm only at like 15% that we get very significant results out of it in the next 30 years even if we tried hard. (aka 30% conditional on orcas being smart enough.)

• Superbabies is good. It would actually work. It's not actually that hard. There's lots of investment already in component science/tech. Orcas doesn't scale. No one cares about orcas. There's not hundreds of scientists and hundreds of millions in orca communications research. Etc. The sense of this plan being weird is a good sense to investigate further. It's possible for superficial weirdness to be wrong, but don't dismiss the weirdness out of hand.

I mean if Orcas are smarter they might be super vastly smarter so you wouldn't need that many.

Superbabies would work well given multiple generations but also only like 30% that we'd get +7std humans born within 10 years even if we tried similarly hard^[1], and I think it's pretty unlikely we have more than 40 years left without strong governance success. (E.g. afaik we still have problems cloning primates well (even though it's been a thing for long) and those are just sub-difficulties^[2] of e.g. creating superbabies through repeated embryo selection.)

1. ^{^}

   (e.g. 1 billon dollars and a few very smart geniuses going into trying to make communication with orcas work well)

2. ^{^}

   Though I don't know what much and wouldn't shock me if we somehow find a way around this.

I found your example problem very interesting, and started thinking about social dynamics that match "tell me if you do it, but don't do it".

The closest cultural anchor I could find is that of sin, confession, priest. Modelling the situation as a confession might be an apt anchor

When it comes to being up- or down-voted its a real gamble. There are of course the same mechanism at play here just as on other social (media) platforms, i.e., there are certain individuals, trends and thought patterns that are hailed and praised and vice versa without any real justification. But hey, that is what makes us human, these unexplainable things called feelings.

PS perhaps a new hashtag on X would be appropriate #stopthedownvoting

Perhaps if you did have the full solution, but it feels like that there are some things of a solution that you could figure out, such that that part of the solution doesn't tell you as much about the other parts of the solution.

I agree with that.

I don't follow this model yet.  I see why, under this model, a party would want the opponent's candidate to enrage people / have a big blind spot (and how this would keep the extremes on their side engaged), but I don't see why this model would predict that they would want their own candidate to enrage people / have a big blind spot.

This is an interesting project! I like the part where you ask people for feedback. Perhaps this is an opportunity to do some A/B testing... for example, if you suspect that "multiple learning styles" is nonsense, you could make a version that includes it, a version that does not, and compare the user satisfaction. This kind of experiment is probably never done with human teachers, because they wouldn't be so flexible at changing their pedagogical methods just because you told them to.

Teachers should use rewards and punishments to shape student behavior.

As an immediate reaction, I feel uncomfortable about the idea of an LLM using rewards and punishments to shape my behavior. Multiple reasons.

For starters, there are approaches even within behaviorism which say that punishments often do more harm than good. (Read Don't shoot the dog for more details.) One reason is that the rewards and punishments always apply to several things at once: if you reward/punish someone for giving you a correct/wrong answer, are you really rewarding/punishing the answer, or the fact that they speak clearly, or the fact that they communicate with you at all, or something else? For example, a strict teacher may congratulate themselves for punishing all the wrong answers, failing to consider that as a side effect the students now hate the teacher, hate the subject, hate the school, and perhaps hate the idea of learning things in general... because all these things are now strongly associated with the punishments in their minds.

Second, I do not approve the idea of being punished by a machine. Because this does not feel compatible with alignment, but also because the LLMs sometimes hallucinate.

...uhm, let's change it to "provide feedback" perhaps?

Social Constructivism [...] suggests that we learn best through working with others.

I guess there is not much the AI can do about this. This is something for the designers of the educational system to consider (and they could use the AI to advise them), but once the AI is talking 1:1 to the students, the idea that "it would be better to have more students there at the same time" is not actionable.

As a general remark, I would recommend making sure that we do not mistake the map for the territory. The educational theories are... well, theories. It is interesting to know how much the things the AI does align with the theories, but it is more important to know how much the things the AI does contribute to the actual learning. Perhaps the students taught by the AI should afterwards be tested and compared to alternative groups taught by an AI with a different prompt, by a human teacher, just reading a textbook alone, or doing nothing.

I also wonder whether trying all these principles at the same time might be too much. Like, if someone comes with an idea that scores high on some of those principles, but very low on one of them (for example: learning 1:1 with the AI tutor already scores low on the "learning in group with others" dimension), when should the idea be tried, and when should it be rejected? Maybe by requiring the solution to fit all the theories we discard some potentially good ideas? Perhaps we could separate the principles into "must be" and "nice to have".

Thanks; I love this description of the primordial thing, had not noticed this this clearly/articulately before, it is helpful.

Re: why I'm hopeful about the available levers here: 

I'm hoping that, instead of Susan putting primary focal attention on Robert ("how can he vote this way, what is he thinking?"), Susan might be able to put primary focal attention on the process generating the scissors statements: "how is this thing trying to trick me and Robert, how does it work?"

A bit like how a person watching a commercial for sugary snacks, instead of putting primary focal attention on the smiling person on the screen who seems to desire the snacks, might instead put primary focal attention on "this is trying to trick me."  

(My hope is that this can become more feasible if we can provide accurate patterns for how the scissors-generating-process is trying to trick Susan(/Robert).  And that if Susan is trying to figure out how she and Robert were tricked, by modeling the tricking process, this can somehow help undo the trick, without needing to empathize at any point with "what if candidate X is great."

How do we best model an irrational world rationally? I would assume we would need to understand at least how irrationality works?

A robust alignment scheme would likely be trivial to transform into an AGI recipe.

Perhaps if you did have the full solution, but it feels like that there are some things of a solution that you could figure out, such that that part of the solution doesn't tell you as much about the other parts of the solution.

And it also feels like there could be a book such that if you read it you would gain a lot of knowledge about how to align AIs without knowing that much more about how to build one. E.g. a theoretical solution to the stop button problem seems like it would not tell you that much about how to build an AGI compared to figuring out how to properly learn a world model of Minecraft. And knowing how to build a world model of minecraft probably helps a lot with solving the stop button problem, but it doesn't just trivially yield a solution.

If you had a system with “ENTITY 92852384 implies ENTITY 8593483" it would be a lot of progress, as currently in neural networks we don't even understand the interal structures.

I want to have an algorithm that creates a world model. The world is large. A world model is uninterpretable by default through it's sheer size, even if you had interpretable but low level abels. By default we don't get any interpretable labels. I think there are ways to have generic dataprocessing procedures that don't talk about the human mind at all, that would yield more interpretable world model. Similar to how you could probably specify some very general property about python programs, such that that program becomes easier to understand by humans. E.g. a formalism of what it means that the control flow is straightforward: Don't use goto in C.

But even if you wouldn't have this, understanding the system still allows you to understand what the structure of the knowledge would be. It seems plausible that one could simply by understanding the system very well, make it such that the learned datastrucutres need to take particular shapes, such that these shapes correspond some relevant alignment properties.

In any case, it seems that this is a problem that any possible way to build an intelligence runs into? So I don't think it is a case against the project. When building an AI with NN you might not even think about that the interal representations might be wierd and alien (even for an LLM trained on human text)^[1], but the same problem persists.

1. ^{^}

   I haven't looked into this, or thought about at all, though that's what I expect.