LessWrong 2.0 Reader

Playing ultimatum game against an agent that gives in to $9 from rocks but not from us is not in the fair problem class, as the payoffs depend directly on our algorithm and not just on our choices and policies.

https://arbital.com/p/fair_problem_class/

A simpler game is “if you implement or have ever implemented LDT, you get $0; otherwise, you get $100”.

LDT decision theories are probably the best decision theories for problems in the fair problem class.

(Very cool that you’ve arrived at the idea of this post independently!)

I don't think I'm the right person to look into this.

I just updated quickly via conservation of expected probability. (I agree though that I'd be a bit concerned about most people updating that quickly. If you think I've gone slightly psychotic please bet with me so I update harder if I notice you're right.)

(EDIT: actually it's sorta shitty because we might not get more evidence because I have even more important things to do and probably don't have time to look into it myself, but i'm happy to bet, though I'd probably want to revise my betting probability.)

I'm happy to bet on "By the end of 2034, does Tsvi think that it's >60% likely that orcas could do superhuman science if they had similar quality and quantity of science education as scientists and were motivated for this, conditional on Tsvi having talked to me for at least 2 hours about this {sometime in 2030-2034}/{when I might have more evidence}?"

I'd currently be at like 30%26% on this, though if you take more time to think about it I might adjust this estimate I am willing to bet on.

I'm happy to bet up to 200$ per bit. Aka if it's resolved "Yes", money flowing from you to me would be $200\$\cdot \left(log2\right(p_{me})-log2(p_{Tsvi}\left)\right)$ (and if it's resolved "No" it would be $200\$\cdot \left(log2\right(1-p_{me})-log2(1-p_{Tsvi}\left)\right)$ ). (Where negative money flow indicates flow into the other direction.)

I don't know what you mean by this:

The thing is, there's probably gonna be like ten other posts in the reference class of this post, and they just... don't leave much of a dent in things?

• I don't think it makes that much sense to just look at cortical neuron counts. Big bodies ask for many neurons, including cortical motor neurons. Do cetaceans have really big motor cortices? Visual cortices? Olfactory bulbs? Keyword "allometry". Yes, brains are plastic, but that doesn't mean orcas are actually ever doing higher mathematics with their brains.

See this comment.

• Scale matters, but I doubt it's very close to being the only thing! Humans likely had genetic adaptations for neuroanatomical phenotypes selected-for by some of: language; tool-making; persisting transient mental content; intent-inference; intent-sharing; mental simulation; prey prediction; deception; social learning; teaching; niche construction/expansion/migration. Orcas have a few of these. But how many, how much, for how long, in what range of situations and manifestations?

I already considered this. (I just posted a question about this [LW · GW].) I don't have good information on to what extent orcas have those, but my guesses are already reflected in my overall guess in the post. 
Why do you think orcas have few of those? For me it seems plausibe that orcas have everything except tool use and niche construction.

I do think there was some significant selection for some kind of intelligence in dolphins and orcas - the main question here is whether being optimized on tool use (IF that was a significant driver in what selected humans for intelligence) would be significantly more useful for having the brain potential generalize to doing science than if the brains were optimized because of social dynamics or hunting strategies.

But of course there are other considerations like "maybe you need fully recursive language to be able to have the abstract reasoning take off, and this might very well come from some adaptations that are not just about neoron counts, and maybe orcas don't have that".

I already took all my current uncertain consideration on this into account when I said "50% that they would be superhuman at science if they had similar quality and quantity of science education as scientists and were motivated for this".

Or do you think a cow brain scaled to 40 billion neurons would be superhuman?

I don't know what you're asking for here. I don't think organisms end up with 40 billion cortical neurons without either some strong selection for at least some sub-dimensions of intelligence, or being as big as Godzilla.

I'm not really excited about just smushing together more brain tissue without it having been optimized to work well together, but orca brains were optimized.

• Culture matters. The Greeks could be great philosophers... But could a kid living in 8000 BCE, who gets to text message with an advanced alien civilization of kinda dumb people, become a cutting edge philosopher in the alien culture? Even though almost everyone ze interacts with is preagricultural, preliterate? I dunno, maybe? Still seems kinda hard actually?

Yep that's why I'm only at like 15% that we get very significant results out of it in the next 30 years even if we tried hard. (aka 30% conditional on orcas being smart enough.)

• Superbabies is good. It would actually work. It's not actually that hard. There's lots of investment already in component science/tech. Orcas doesn't scale. No one cares about orcas. There's not hundreds of scientists and hundreds of millions in orca communications research. Etc. The sense of this plan being weird is a good sense to investigate further. It's possible for superficial weirdness to be wrong, but don't dismiss the weirdness out of hand.

I mean if Orcas are smarter they might be super vastly smarter so you wouldn't need that many.

Superbabies would work well given multiple generations but also only like 30% that we'd get +7std humans born within 10 years even if we tried similarly hard^[1], and I think it's pretty unlikely we have more than 40 years left without strong governance success. (E.g. afaik we still have problems cloning primates well (even though it's been a thing for long) and those are just sub-difficulties^[2] of e.g. creating superbabies through repeated embryo selection.)

1. ^{^}

   (e.g. 1 billon dollars and a few very smart geniuses going into trying to make communication with orcas work well)

2. ^{^}

   Though I don't know what much and wouldn't shock me if we somehow find a way around this.

I found your example problem very interesting, and started thinking about social dynamics that match "tell me if you do it, but don't do it".

The closest cultural anchor I could find is that of sin, confession, priest. Modelling the situation as a confession might be an apt anchor

When it comes to being up- or down-voted its a real gamble. There are of course the same mechanism at play here just as on other social (media) platforms, i.e., there are certain individuals, trends and thought patterns that are hailed and praised and vice versa without any real justification. But hey, that is what makes us human, these unexplainable things called feelings.

PS perhaps a new hashtag on X would be appropriate #stopthedownvoting

Perhaps if you did have the full solution, but it feels like that there are some things of a solution that you could figure out, such that that part of the solution doesn't tell you as much about the other parts of the solution.

I agree with that.

I don't follow this model yet.  I see why, under this model, a party would want the opponent's candidate to enrage people / have a big blind spot (and how this would keep the extremes on their side engaged), but I don't see why this model would predict that they would want their own candidate to enrage people / have a big blind spot.

This is an interesting project! I like the part where you ask people for feedback. Perhaps this is an opportunity to do some A/B testing... for example, if you suspect that "multiple learning styles" is nonsense, you could make a version that includes it, a version that does not, and compare the user satisfaction. This kind of experiment is probably never done with human teachers, because they wouldn't be so flexible at changing their pedagogical methods just because you told them to.

Teachers should use rewards and punishments to shape student behavior.

As an immediate reaction, I feel uncomfortable about the idea of an LLM using rewards and punishments to shape my behavior. Multiple reasons.

For starters, there are approaches even within behaviorism which say that punishments often do more harm than good. (Read Don't shoot the dog for more details.) One reason is that the rewards and punishments always apply to several things at once: if you reward/punish someone for giving you a correct/wrong answer, are you really rewarding/punishing the answer, or the fact that they speak clearly, or the fact that they communicate with you at all, or something else? For example, a strict teacher may congratulate themselves for punishing all the wrong answers, failing to consider that as a side effect the students now hate the teacher, hate the subject, hate the school, and perhaps hate the idea of learning things in general... because all these things are now strongly associated with the punishments in their minds.

Second, I do not approve the idea of being punished by a machine. Because this does not feel compatible with alignment, but also because the LLMs sometimes hallucinate.

...uhm, let's change it to "provide feedback" perhaps?

Social Constructivism [...] suggests that we learn best through working with others.

I guess there is not much the AI can do about this. This is something for the designers of the educational system to consider (and they could use the AI to advise them), but once the AI is talking 1:1 to the students, the idea that "it would be better to have more students there at the same time" is not actionable.

As a general remark, I would recommend making sure that we do not mistake the map for the territory. The educational theories are... well, theories. It is interesting to know how much the things the AI does align with the theories, but it is more important to know how much the things the AI does contribute to the actual learning. Perhaps the students taught by the AI should afterwards be tested and compared to alternative groups taught by an AI with a different prompt, by a human teacher, just reading a textbook alone, or doing nothing.

I also wonder whether trying all these principles at the same time might be too much. Like, if someone comes with an idea that scores high on some of those principles, but very low on one of them (for example: learning 1:1 with the AI tutor already scores low on the "learning in group with others" dimension), when should the idea be tried, and when should it be rejected? Maybe by requiring the solution to fit all the theories we discard some potentially good ideas? Perhaps we could separate the principles into "must be" and "nice to have".

Thanks; I love this description of the primordial thing, had not noticed this this clearly/articulately before, it is helpful.

Re: why I'm hopeful about the available levers here: 

I'm hoping that, instead of Susan putting primary focal attention on Robert ("how can he vote this way, what is he thinking?"), Susan might be able to put primary focal attention on the process generating the scissors statements: "how is this thing trying to trick me and Robert, how does it work?"

A bit like how a person watching a commercial for sugary snacks, instead of putting primary focal attention on the smiling person on the screen who seems to desire the snacks, might instead put primary focal attention on "this is trying to trick me."  

(My hope is that this can become more feasible if we can provide accurate patterns for how the scissors-generating-process is trying to trick Susan(/Robert).  And that if Susan is trying to figure out how she and Robert were tricked, by modeling the tricking process, this can somehow help undo the trick, without needing to empathize at any point with "what if candidate X is great."

How do we best model an irrational world rationally? I would assume we would need to understand at least how irrationality works?