Originality vs. Correctness
post by alkjash, habryka (habryka4) · 2023-12-06T18:51:49.531Z · LW · GW · 17 commentsContents
Intellectual generals vs supersoldiers Bayes optimality vs Discovering new hypothesis classes Correctness as defence against the dark arts None 17 comments
I talk with Alkjash about valuing original thinking vs getting things right. We discuss a few main threads:
- What are the benefits of epistemic specialisation? What about generalism?
- How much of the action in an actual human mind is in tweaking your distribution over hypotheses and how much over making sure you're considering good hypotheses?
- If your hope is to slot into an epistemic process that figures out what's correct in part by you coming up with novel ideas, will processes that are out to get you [LW · GW] make you waste your life?
Intellectual generals vs supersoldiers
Bayes optimality vs Discovering new hypothesis classes
Correctness as defence against the dark arts
17 comments
Comments sorted by top scores.
comment by Ben Pace (Benito) · 2023-12-06T20:09:51.182Z · LW(p) · GW(p)
The other mindset I have is something like, as long as I act in such a way that an alternative version of me who landed in the right community does great, that would be enough? Like, if I throw a dart at the dartboard and hit wrong, but I act in such a way that I would want everyone who happened to work on the right things to act, then I don't have to worry too much about getting good at darts.
There's a sense I'm getting that Alkjash wants to give up on having a great map of the communities that are around, and just get on with nailing the work of the one that he's in, so that if he's ended up in one of the good ones, then the payoff is large, and so that he isn't wasting most of his time trying to evaluate which one to be in (which is an adversarial game in which he basically expects to either lose or waste most of his resources on).
A key point to me that seemed not to be mentioned is that this takes the distribution of communities as static. Perhaps my life so far has seen the rise and fall of more communities than others get to experience, but I think one of the effects of modeling the communities and figuring out your own principles is not just that you figure out which is a good one to be part of right now, but that you set the standards and the incentives for what new ones can come into existence. If most people will basically go along with the status quo, then the competition for new games to play is weak.
I'll try to make up an example that points at the effect. Suppose you're a good academic who has harshly fought back against the bureaucratic attempts to make you into a manager. You've picked particular departments and universities and sub-fields where you can win this fight and actually do research. Nonetheless this has come with major costs for your career. Then suppose someone starts a new university and wants to attract you (a smart academic who is underpriced by the current system because of your standards) to join. Compared to the version of you that just did what the system wanted, where they just needed to offer you slightly more pay, they actually have a reason to build the sort of place that attracts people with higher standards. They can say "Look, the pay is lower, and we're just getting started with the department, but I will ensure that the majority of professors have complete control over the number of PhDs they take (including 0)." You doing the work of (a) noticing the rot in your current institution and (b) clearly signaling that you will not accept the rot, provides a clear communication to whoever builds the next institution that this is worth avoiding and furthermore they can attract good people by doing so.
This is a general heuristic that I've picked up that it's good to figure out what principles you care about and act according to them so people know what standards you will hold them to in future situations that you weren't thinking about and couldn't have predicted in advance.
I see this as an argument for the "broad map" over the "detailed map" side of the debate.
Replies from: alkjash↑ comment by alkjash · 2023-12-06T21:50:56.236Z · LW(p) · GW(p)
I don't have a complete reply to this yet, but wanted to clarify if it was not clear that the position in this dialogue was written with the audience (a particularly circumspect broad-map-building audience) in mind. I certainly think that the vast majority of young people outside this community would benefit from spending more time building broad maps of reality before committing to career/identity/community choices. So I certainly don't prescribe giving up entirely.
ETA: Maybe a useful analogy is that for Amazon shopping I have found doing serious research into products (past looking at purchase volume and average ratings) largely unhelpful. Usually if I read reviews carefully, I end up more confused than anything else as a large list of tail risks and second-order considerations are brought to my attention. Career choice I suspect is similar with much higher stakes.
comment by Steven Byrnes (steve2152) · 2023-12-06T20:00:35.024Z · LW(p) · GW(p)
"the tools of LW rationality are better at helping you converge towards the boundary of what is already known and worse at getting past that."
For what it’s worth, in my neuroscience work, I think of myself as doing a lot of new-hypothesis-generation (or new-theoretical-framework-generation) and not so much evaluation-of-existing-hypotheses. If you believe me in that characterization, then my perspective is: I agree that The Sequences’ emphasis on Bayes rule is not too helpful for that activity, but there is lots of other stuff in The Sequences, and/or in later rationality stuff like CFAR-handbook and Scout Mindset, and/or in LW cultural norms, that constitute useful tools for the activity of new-hypothesis-generation. Examples include the “I notice that I’m confused” thing, ass numbers and probabilities, ITTs, changing-one’s-mind being generally praiseworthy, “my model says…”, and probably lots more that I’ve stopped noticing because they’re so in-the-water.
I don’t think any of those things are a royal road to great hypothesis-generation, by any means, but I do think they’re helpful on the margin.
(Compare that to my own physics PhD where I learned no useful skills or tools whatsoever for constructing new good hypotheses / theories. …Although to be fair, maybe I just had the wrong PIs for that, or never bothered to ask them for advice, or something.)
Replies from: GuySrinivasan↑ comment by SarahNibs (GuySrinivasan) · 2023-12-07T03:18:32.996Z · LW(p) · GW(p)
I feel like alkjash's characterization of "correctness" is just not at all what the material I read was pointing towards.
The Sequences’ emphasis on Bayes rule
Maybe I'm misremembering. But for me, the core Thing this part of the Sequences imparted was "intelligence, beliefs, information, etc - it's not arbitrary. It's lawful. It has structure. Here, take a look. Get a feel for what it means for those sorts of things to 'have structure, be lawful'. Bake it into your patterns of thought, that feeling."
If a bunch of people are instead taking away as the core Thing "you can do explicit calculations to update your beliefs" I would feel pretty sad about that, I think?
Replies from: NicholasKross↑ comment by Nicholas / Heather Kross (NicholasKross) · 2023-12-08T18:39:09.138Z · LW(p) · GW(p)
Agreed. I think of it as:
You need your mind to have at least barely enough correctness-structure/Lawfulness to make your ideas semi-correct, or at least easy to correct them later.
Then you want to increase originality within that space.
And if you need more original ideas, you go outside that space (e.g. by assuming your premises are false, or by taking drugs; yes, these are the same class of thing), and then clawing those ideas back into the Lawfulness zone.
Reading things like this, and seeing how long it took them to remember "Babble vs Prune", makes me wonder if people just forgot the existence of the "create, then edit" pattern. So people end up rounding off to "You don't need [? · GW] to edit or learn more, because all of my creative ideas are also semi-correct in the first place [? · GW]". Or "You can't create good-in-hindsight ideas without editing tools X Y Z in your toolbelt".
The answer is probably closer to one of these than the other, and yadda yadda social engineering something something community beliefs, but man do people talk like they believe these trivially-false extreme cases.
comment by Ben Pace (Benito) · 2023-12-07T07:01:59.723Z · LW(p) · GW(p)
Habryka: ...if I expect someone to be in a position where it would be good for them to do something hard... I expect a person who is grounded in multiple worldviews to only really be compelled to do that when all their worldviews demand that action as good, which I think is just quite rare.
When I think of myself doing something like the "be in two cults" sort of move, I'm doing it differently:
- If something counts as a major win in one cult and is permissible (or slightly costly) in the other cult, this is a strong clue that I should do it.
- If something counts as a major failing in one cult and permissible (or slightly incentivized) in the other cult, this is a strong clue that I should avoid it.
Basically I think of both perspectives as generators for finding wins and avoiding failures.
comment by Nicholas / Heather Kross (NicholasKross) · 2023-12-08T18:43:19.607Z · LW(p) · GW(p)
Both people here are making conflation/failure-to-decouple mistakes. E.g. tying "community coordination" together with "how to generate and/or filter one's own ideas".
Tabooing most of the topic-words/phrases under discussion, I reckon, would have made this dialogue 3-10x better.
(Will have more thoughts, possibly a response post, once I'm done reading/thinking-through this.)
comment by Elizabeth (pktechgirl) · 2023-12-08T06:06:03.027Z · LW(p) · GW(p)
imagine two worlds, one where everyone spends a huge chunk of their time trying to find and lay claim to the next big heavy-tailed thing, and one where a small number of people do that and the rest all randomly assign themselves to some narrow craft to perfect. It seems to me that the second world results in the big heavy-tailed things being done better in the end, even if many individuals missed out by being captured in the wrong niche.
I agree with this, and it seems to be one of my biggest procedural disagreements with EA and rationality. I'd love to hear some really gears-based arguments against it.
Replies from: SaidAchmiz, kyle-ibrahim↑ comment by Said Achmiz (SaidAchmiz) · 2023-12-11T20:27:27.103Z · LW(p) · GW(p)
I can definitely see how a disagreement with EA can be based on this idea, but how does the rationality version work? Can you say more about that?
↑ comment by zrezzed (kyle-ibrahim) · 2023-12-08T19:59:03.323Z · LW(p) · GW(p)
Which do you agree would be better? I’m assuming the latter, but correct me if I’m wrong.
I haven’t thought this through, but a potential argument against: 1) agreement / alignment on what the heavy-tail problems are and their relative weight is a necessary condition for the latter to be a better strategy 2) neither this community, let alone the broader society have that thus 3) we should still focus on correctness overall.
That does reflect my own thinking about these things.
comment by Said Achmiz (SaidAchmiz) · 2023-12-07T21:57:26.420Z · LW(p) · GW(p)
This dialogue is very long and rambling (and is an example of why I am skeptical about the dialogues feature in general), so this comment only addresses a couple of bits of it. That said:
[alkjash] Of course the final goal is to produce thoughts that are original AND correct, but I find the originality condition more stringent and worth optimizing for. This might be a feature of working in mathematics where verifying correctness is cheap and reliable.
(Emphasis mine.)
Yes, I think that’s basically it. There doesn’t seem to be much more to say on the subject, actually; yes, if you work in math, where verifying correctness is cheap and reliably, then pursuing originality is obviously the way to go. Almost no other domain works like that, so pursuing originality over correctness is not the way to go in most other domains. End of discussion…?
One thing I’ll add from my own experience as a designer is that in design (UX design / web design in my case, though I strongly suspect—and what I’ve heard and read from others supports this—that it applies generally), if you try to do things correctly, and if you’re serious about this, you will find yourself doing original things, merely as an almost inevitable by-product of doing things correctly. This is partly because your assessment of what is correct may differ from others’, but mostly it’s because very few people are actually trying to do things correctly.
[habryka] … I think as people go I’ve been much more “heads-down” on building things than most other people in the LW community, and have had trouble finding other people to join me.
[habryka] But also, at the same time, one of the most common pieces of advice I give to young people in AI Alignment is to just try to write some blogposts that explain what you think the AI Alignment problem is, and where the hard parts are, and what you think might be ways to tackle it, in your own language and from your own perspective. And almost every time I’ve seen someone actually do that, I think they went on to do pretty great work. But almost no one does it, which is confusing.
This is consistent with my experience. As I wrote recently on a different forum, the number of people who are willing to actually do a technically simple thing, that they wish to see done, is very small. People mostly don’t do things, in general. If you assume that people just won’t take the initiative to do any given thing, you’ll usually predict outcomes correctly.
(This seems to be especially true among “rationalists”, unfortunately.)
Replies from: gwern↑ comment by gwern · 2023-12-11T22:26:12.986Z · LW(p) · GW(p)
This reminds me of the Replication Crisis. Psychologists placed a high value on novelty and surprising results, the sort of narratives you can build a TED Talk on, and thought that verifying correctness in psychology was cheap and reliable. You didn't have to eat your veggies, you could just skip straight to the dessert. Turns out, that is not the case: the effects are small, analyst flexibility far higher than known, required sample sizes at least 1 order of magnitude (and often 2) larger than used, and verification is expensive and extremely difficult. The areas of psychology that most prize novelty and cool stories, like social psychology, have turned out to be appallingly riddled with entire fake fields of study and now have a bad tummyache from trying to digest decades of dessert; meanwhile, the most boring, tedious, mathematical areas, like psychophysics, don't even know what 'the Replication Crisis' is.
comment by zrezzed (kyle-ibrahim) · 2023-12-08T04:35:29.190Z · LW(p) · GW(p)
And idk, maybe I am kind of convinced? Like, consistency checks are a really powerful tool and if I imagine a young person being like "I will just throw myself into intellectual exploration and go deep wherever I feel like without trying to orient too much to what is going on at large, but I will make sure to do this in two uncorrelated ways", then I do notice I feel a lot less stressed about the outcome
I worry this gives up too much. Being embedded in multiple communities/cultures with differing or even conflicting values and world views is exceedingly common. Noticing that, and explicitly playing with the idea of “holding multiple truths” in your head is less common, but still something perhaps even most people would recognize.
But most people would not recognize the importance of this dialogue. Navigating the tension between academics and politics does not seem sufficient to correctly orient humanity. The conflicting values of business and family do not seem to produce anything that resembles truth seeking.
I feel pretty strongly that letting go of correctness in favor of any heuristic means you will end up with the wrong map, not just a smaller or fuzzier one. I don’t think that’s advice that should be universally given, and I’m not even sure how useful it is at all.
Replies from: whitehatStoic↑ comment by MiguelDev (whitehatStoic) · 2023-12-09T08:15:50.654Z · LW(p) · GW(p)
I feel pretty strongly that letting go of correctness in favor of any heuristic means you will end up with the wrong map, not just a smaller or fuzzier one. I don’t think that’s advice that should be universally given, and I’m not even sure how useful it is at all.
I think correctness applies - until it reaches a hard limit. Understanding what an intellectual community like LessWrong was able to generate as clusters of valuable knowledge is the most correct thing to do but in order to generate novel solutions, one must accept with bravery[1] that the ideas in this forum might have some holes that original ideas will emerge.
- ^
I assume that many people will be scared of challenging what is considered to be normal / generally accepted principles in LessWrong, but this I think is necessary in tackling grand challenges like solving the alignment problem.
comment by riceissa · 2023-12-07T06:45:39.504Z · LW(p) · GW(p)
I feel like doing correctness right requires originality after a certain point, so the two don't feel too distinct to me. Early in one's intellectual development it might make sense to just "shop around" for different worldviews by reading widely, but after a while you are going to be routinely bumping into things that aren't on the collective map.
The casus belli example Habryka gives in the "Correctness as defence against the dark arts" strikes me as an example of ... how originality helps defend against the dark arts! (I am guessing here that Habryka did not just read some old rationalist blog post called "Casus belli, how people use it to manipulate each other, and how to avoid getting got", but that he formed this connection himself.) More generally but also personally, I feel like several times in my life (including now) I have been in bad situations where the world just does not seem to have a solution to my problem, where no amount of google-fu, reading books, seeking societally-established forms of help (therapists, doctors, etc.) has helped. The only way out seems to be to do original thinking.
I also want to highlight that the mental motions of Correctness reasoning seems to be susceptible to the dark arts (to be clear, I think Habryka himself is smart enough to avoid this kind of thing, but I want to highlight this for others). I feel this most whenever I go on Twitter. Like, I go on there thinking "ok, for whatever reason many people (even Wei Dai now, apparently) are discoursing on here now, so I better read it to not fall behind, I better enlarge my hypothesis space by taking in new ideas and increase the range of thoughts I can think!" (this is the kind of thing I mean by "mental motions of Correctness reasoning" -- I am mainly motivated by making my map bigger, not making it more detailed). But then after a while I feel like the social environment is tugging me to feel a certain way, value certain things, believe certain things (or else I'm a bad person) (maybe I only had this line of thought because Qiaochu tugged me in a certain direction!). I started out wanting to just explore and try to make my map Correct, but turns out the territory contained adversarial computations... This sort of thing, it seems to me, is even worse for the non-LessWrong population. Again it seems to me most healthy to mostly just be thinking for myself and then periodically check in on Twitter discourse to see what's up (this is aspirational).
comment by ZY (AliceZ) · 2024-10-26T18:44:13.540Z · LW(p) · GW(p)
Could you define what you mean by "correctness" in this context? I think there might be some nuances into this, in terms of what "correct" means, and under what context
comment by xpym · 2023-12-08T09:22:56.946Z · LW(p) · GW(p)
David Chapman [LW · GW] has been banging on for years now against "Bayesianism"/early LW-style rationality being particularly useful for novel scientific advances, and, separately, against utilitarianism being a satisfactory all-purpose system of ethics. He proposes another "royal road", something something Kegan stage 5 (and maybe also Buddhism for some reason), but, frustratingly, his writings so far are rich on expositions and problem statements but consist of many IOUs on detailed solution approaches. I think that he makes a compelling case that these are open problems, insufficiently acknowledged and grappled with even by non-mainstream communities like the LW-sphere, but is probably overconfident about postmodernism/himself having much useful to offer in the way of answers.