Posts
Comments
Agreed. My impression has been for a while that there's a super weak correlation (if any) between whether an idea goes into the right direction and how well it's received. Since there's rarely empirical data, one would hope for an indirect correlation where correctness correlates with argument quality, and argument quality correlates with reception, but second one is almost non-existent in academia.
Thanks! Sooner or later I would have searched until finding it, now you've saved me the time.
Well I don't remember anything in detail, but I don't believe so; I don't think you'd want to have a restriction on the training data.
I fully agree with your first paragraph, but I'm confused by the second. Where am I making an argument for camp #1?
I'm definitely a Camp 2 person, though I have several Camp 1 beliefs. Consciousness pretty obviously has to be physical, and it seems likely that it's evolved. I'm in a perpetual state of aporia trying to reconcile this with Camp 2 intuitions.
I wouldn't call those Camp #1 beliefs. It's true that virtually all of Camp #1 would agree with this, but plenty of Camp #2 does as well. Like, you can accept physicalism and be Camp #2, deny physicalism and be Camp #2, or accept physicalism and be Camp #1 -- those are basically the three options, and you seem to be in the first group. Especially based on your second-last paragraph, I think it's quite clear that you conceptually separate consciousness from the processes that exhibit it. I don't think you'll ever find a home with Camp #1 explanations.
I briefly mentioned in the post that the way Dennett frames the issue is a bit disingenuous since the Cartesian Theater has a bunch of associations that Camp #2 people don't have to hold.
Being a physicalist and Camp #2 of course leaves you with not having any satisfying answer for how consciousness works. That's just the state of things.
The synesthesia thing is super interesting. I'd love to know how strong the correlation is between having this condition, even if mild, and being Camp #2.
That's a relative rather than absolute claim. The article has pushback from camp 2
Yeah -- I didn't mean to imply that orthormal was or wasn't successful in dissolving the thought experiment, only that his case (plus that of some of the commenters who agreed with him) is stronger than what Dennett provides in the book.
I did remember reading, Why it's so hard to talk about Consciousness, and shrinking back from the conflict that you wrote as an example of how the two camps usually interact.
Thanks for saying that. Yeah hmm I could have definitely opened the post in a more professional/descriptive/less jokey way.
Since we seem to be unaware of the different sets of skills a human might possess, how they can be used, and how different they are 'processed', it kind of seems like Camp 1 and Camp 2 are fighting over a Typical Mind Fallacy - that one's experience is generalized to others, and this view seen as the only one possible.
I tend to think the camps are about philosophical interpretations and not different experiences, but it's hard to know for sure. I'd be skeptical about correlations with MBTI for that reason, though it would be cool.
(I only see two dots)
At this point, I've heard this from so many people that I'm beginning to wonder if the phenomenon perhaps simply doesn't exist. Or I guess maybe the site doesn't do it right.
I think the philosophical component of the camps is binary, so intermediate views aren't possible. On the empirical side, the problem that it's not clear what evidence for one side over the other looks like. You kind of need to solve this first to figure out where on the spectrum a physical theory falls.
I think this lacks justification why the entire approach is a good idea. Improving mathematical accuracy in LLMs seems like a net negative to me for the same reason that generic capability improvements are a net negative.
No. It would make a difference but it wouldn't solve the problem. The clearest reason is that it doesn't help with Inner Alignment at all.
Having not read the book yet, I'm kind of stumped at how different this review is to the one from Alexander. The two posts make it sound like a completely different book, especially with respect to the philosophical questions, and especially especially with respect to the expressed confidence. Is this book a neutral review of the leading theories that explicitly avoids taking sides, or is it a pitch for another I-solved-the-entire-problem theory? It can't really be both.
Downvote because I feel like this completely fails to engage with what rationality-as-used-in-this-community-and-in-the-sequences actually is. Especially since Eliezer devoted so much of his writing toward recognizing when you're wrong and changing your mind.
Does anyone know about an update to GPT-4 that was deployed to the https://chat.openai.com/ version in the last 1-3 weeks? It seems to have gotten significantly better.
Fair point, but there's still a strong correlation between established consensus and expert consensus. In most cases, they're probably gonna be similar.
One particular application of GPT-4 (or other LLMs) that seems really valuable to me is as a fact check on claims about what other people think (e.g., "what's the scientific consensus on whether exercise lowers the risk of dementia?") As long as the topic isn't about political correctness, I pretty much trust GPT-4 to represent the consensus fairly, and that's a pretty amazing tool to have. Like, it's not uncommon that people disagree about what the scientific consensus is, and we didn't really have a way to answer these questions before.
Sometimes I even feel like it should be an epistemic norm that you fact check important claims with GPT-4 when applicable. E.g., whenever you say "this is considered a fringe idea" or "these two books are the most widely known on the subject", or even "the argument this person makes is only xyz", if it's in the context of a serious post, you should check with GPT-4 whether that's actually true and perhaps link to a transcript. Maybe that's going too far but like, restricting the extent to which people can bend reality to fit their needs seems pretty great.
Tried it a bit, and this doesn't seem like a test that measures what we care about because the humans (at least some of them) are trying to fool you into thinking they're bots. Consequently, even if you have a question that would immediately and reliably tell a human-honestly-trying-to-answer apart from a bot, you can't with the game with this because humans won't play along.
To make this meaningful, all human players should be trying to make others think they're human.
Peano arithmetic is not complete. I think the claim from my post is correct except that it misses having a computable set of axioms as a second requirement (which in the post more or less follows from context, though I should still probably add it explicitly) actually nvm I did say this in the post. (See e.g. https://math.stackexchange.com/questions/1280043/every-complete-axiomatizable-theory-is-decidable)
I don't know any place where he wrote it up properly, but he's said enough to infer that he's confident that consciousness is about higher-order thoughts (i.e., self-reflection/meta-awareness/etc.) This explains the confidence that chickens aren't conscious, and it would extend to fish as well.
Pascal's mugging usually involves tiny probabilities though, and you need to have pretty darn confident philosophical beliefs to go below 1% here.
I generally don't and wouldn't expect people to increase understanding or epistemology based on meditation. I would expect productivity gains in some cases, depending on how they do it, and happiness gains in many cases.
More speculatively, I think the risk of degrading your epistemology is probably low if you go in with a sufficiently skeptical mindset, which you seem to have.
The claim "fish can suffer and this is morally important" is not uncontroversial. Also, not all possible points of disagreement are related to biology. I mean, the reaction you just got proved as much.
I'm pointing this out because you've written several posts about related topics before, and all of them assumed more background agreement than is actually there. I'm not saying that understanding these disagreements will cause you to change your mind (in fact I think that's very unlikely), but just as a matter of effectively communicating, it would be worth trying; otherwise, you'll keep being frustrated with people reacting strangely to your writing.
I think it kinda works here just because many people consider "fish can suffer and this is morally important" plausible enough to take seriously. As you pointed out, the quantitative case is very strong; even a small probability makes it a moral priority. But I would not write stuff like "But let’s be super conservative and say that there’s only a 50% chance that they can feel pain." when e.g. Eliezer would put way less than 10% on fish feeling pain in a morally relevant way.
I think it'd make most sense to post it as a link post here.
In your experience, if you just copy-paste a post you've written to GPT-4 and ask it to critique it, does it tell you anything useful?
Did you publish it? If so, can you link it?
If not, you can just publish it in a post.
Well, I don't have the money to help even if I wanted to.
That said, as a rule I think you should make sure that you can produce work that others appreciate at least to an extent before asking for funding. I mean, I don't actually know much about how these decisions are made, but just using common sense, it seems unreasonable to expect them to fund you otherwise. Perhaps you should just take an ordinary job.
Generally, if you have a good idea, you can also pursue it while working on the side (I've done this for about nine months). And then if you created something that people appreciate, that would be the time to think about transitioning to full-time alignment work.
Why is the reaction feature non-anonymous? It makes me significantly less likely to use them, especially negative ones.
I think the consideration makes sense because "lies are bad" is a much simpler norm than "lies are bad if they reduce the informational usefulness of the sentence below 0". The latter is so complex that if it were the accepted norm, it'd probably be so difficult to enforce and so open to debate that it'd lose its usefulness.
I'm emotionally very opposed to looking at drafts of anyone, though this is not a rationally thought out position. I don't have the same reaction toward votes because I don't feel like you have an expectation of privacy there. There are forums where upvotes are just non-anonymous by default.
I'm wondering where Biological Naturalism[1] falls within these two camps? It seems like sort of a "third way" in between them, and incidentally, is the explanation that I personally have found most compelling.
My take based on the summary is that it's squarely in Camp #2.
In particular, I think this part seals the deal
This means that while consciousness and other mental phenomena are rooted in the physical workings of the brain, they also have their own first-person ontology that is not captured by third-person descriptions of the brain's workings.
According to Camp #1, there's nothing ontologically special about consciousness, so as soon as you give it its own ontology, you've decided which camp you're in.
Alright, so I changed the paragraph into this:
Conversely, Camp #2 is convinced that there is an experience thing that exists in a fundamental way. There's no agreement on what this thing is – some postulate causally active non-material stuff, whereas others agree with Camp #1 that there's nothing operating outside the laws of physics – but they all agree that there is something that needs explaining. Moreover, even if consciousness is compatible with the laws of physics, it still poses a conceptual mystery relative to our current understanding. A complete solution (if it is even possible) may also have a nontrivial metaphysical component.
I think a lot of Camp #2 people want to introduce new metaphysics, which is why I don't want to take out the last sentence.
But note that all the controversy is about the descriptions. "Qualia" is a descriptor, "sensation" is a descriptor, etc. Even "illusionists" about qualia don't deny that people experience things.
I don't think this is true. E.g., Dennett has these bits in Consciousness Explained: 1, 2, 3, 4.
Of course, the issue is still tricky, and you're definitely not the only one who thinks it's just a matter of description, not existence. Almost everyone agrees that something exists, but Camp #2 people tend to want something to exist over and above the reports of that thing, and Dennett seems to deny this. And (as I mentioned in some other comment) part of the point of this post is that you empirically cannot nail down exactly what this thing is in a way that makes sense to everyone. But I think it's reasonable to say that Dennet doesn't think people experience things.
Also, Dennett in particular says that there is no ground truth as to what you experience, and this is arguably a pretty well-defined property that's in contradiction with the idea that the experience itself exists. Like, I think Camp #2 people will generally hold that, even if errors can come in during the reports of experience, there is still always a precise fact of the matter as to what is being experienced. And depending on their metaphysics, it would be possible to figure out what exactly that is with the right neurotech.
And another reason why I don't think it's true is because then I think illusionism wouldn't matter for ethics, but as I mentioned in the post, there are some illusionists who think their position implies moral nihilism. (There are also people who differentiate illusionism and eliminativism based on this point, but I'm guessing you didn't mean to do that.)
I think we need to be careful not to mush together metaphysics and epistemics. A conceptual mystery, a felt lack of explanation - these are epistemic problems. That's not sufficient reason to infer distinct metaphysical categories. Particular camp #2 philosophers sometimes have arguments that try to go from these epistemic premises, plus additional premises, to a metaphysical divide between mental and physical properties. Those arguments fail, but aside from that, it's worthwhile to distinguish their starting points from their conclusions.
Agreed; too tired right now but will think about how to rewrite this part.
Secondly, you imply that according to camp #2, statements like "I experienced a headache" cannot be mistaken. As TAG already pointed out, the claim of incorrigibility is not necessary. As soon as one uses a word or concept, one is risking error. Suppose you are at a new restaurant, and you try the soup, and you say, "this soup tastes like chicken." Your neighbor says, "no, it tastes like turkey." You think about it, the taste still fresh in your mind, and realize that she is right. It tastes (to you) like turkey, you just misidentified it.
I don't think I said that. I think I said that Camp #2 claims one cannot be wrong about the experience itself. I agree (and I don't think the post claims otherwise) that errors can come in during the step from the experience to the task of finding a verbalization of the experience. You chose an example where that step is particularly risky, hence it permits a larger error.
Note that for Camp #2, you can draw a pretty sharp line between conscious and unconscious modules in your brain, and finding the right verbalization is mostly an unconscious process.
Thanks!
I've been interested in consciousness through a 23 year career in computational cognitive neuroscience. I think making progress on bridging the gap between camp 1 and camp 2 requires more detailed explanations of neural dynamics. Those can be inferred from empirical data, but not easily, so I haven't seen any explanations similar to the one I've been developing in my head. I haven't published on the topic because it's more of a liability for a neuroscience career than an asset. Now that I'm working on AI safety, consciousness seems like a distraction. It's tempting to write a long post about it, since this community seems substantially better at engaging with the topic than neuroscientists are; but the time cost is still substantial. If I do write such a post, I'll cite this one in framing the issue.
If you do, and if you're interested in exchanging ideas, feel free to reach out. I've been thinking about this topic for several years now and am also planning to write more about it, though that could take a while.
Sounds about right. But just to be clear it doesn't mean that "consciousness" equals "talks about consciousness". It's just that by explaining a bigger thing (consciousness) we will also explain the smaller one (talks about consciousness) that depends on it. I expect consciousness to be related to many other stuff and talks about it being just an obvious example of a thing that wouldn't happen without consciousness.
Yes, this is also how I meant it. Never meant to suggest that the consciousness phenomenon doesn't have other functional roles.
Okay, now Camp 1 feels more like home. Yet, I notice that I'm confused. How can anyone in Camp 2 be a physicalist then? Can you give me an example?
So first off, using the word physicalist in the post was very stupid since people don't agree what it means, and the rewrite I made before my previous comment took the term out. So what I meant, and what the text now says now without the term, is "not postulating causal power in addition to the laws of physics".
With that definition, lots of Camp #2 people are physicalists -- and on LW in particular, I'd guess it's well over 80%. Even David Chalmers is an example; consciousness doesn't violate the laws of physics under his model, it's just that you need additional -- but non-causally-relevant -- laws to determine how consciousness emerges from matter. In general, you can also just hold that consciousness is a different way to look at the same process, which is sometimes called dual-aspect monism, and that's physicalist, too.
I was under the impression that your camps were mostly about whether a person thinks there is a Hard Problem of Consciousness or not. But now it seems that they are more about whether the person includes idealism in some sense into their worldview? I suppose you are trying to compress both these dimensions (idealism/non-idealism, HP/non-HP) into one. And if so, I'm afraid your model is going to miss a lot of nuances.
I mean, I don't think it's just about the hard problem; otherwise, the post wouldn't be necessary. And I don't think you can say it's about idealism because people don't agree what idealism means. Like, the post is about describing what the camps are, I don't think I can do it better here, and I don't think there's a shorter description that will get everyone on board.
In general, another reason why it's hard to talk about consciousness (which was in a previous version of this post but I cut it) is that there's so much variance in how people think about the problem, and what they think terms mean. Way back, gwern said about LLMs that "Sampling can prove the presence of knowledge but not the absence". The same thing is true about the clarity of concepts; discussion can prove that they're ambiguous, but never that they're clear. So you may talk to someone, or even to a bunch of people, and you'll communicate perfectly, and you may think "hooray, I have a clear vocabulary, communication is easy!". And then you talk to smn else the next day and you're talking way past each other. And it's especially problematic if you pre-select people who already agree with you.
Overall, I suspect the Camp #1/Camp #2 thing is the best (as in, the most consistently applicable and most informative) axis you'll find. Which is ultimately an empirical question, and you could do polls to figure it out. I suspect asking about the hard problem is probably pretty good (but significantly worse than the camps) and asking about idealism is probably a disaster. I also think the camps get at a more deeply rooted intuition compared to the other stuff.
Okay, that makes a lot of sense. I'm still pretty sure that you're a central example of what I meant by Camp #1, and that the problem was how I described them. In particular,
-
Solving consciousness = solving the Meta Problem: what I meant by "solving the meta problem" here entails explaining the full causal chain. So if you say "People talk about 'consciousness' because they possess some mental property that they call 'consciousness'", then this doesn't count as a solution until you also recursively unpack what this mental property is, until you've reduced it to the brain's physical implementation. So I think you agree with this claim as it was intended. The way someone might disagree is if they hold something like epiphenomenalism, where the laws of physics are not enough and additional information is required. Or, if they are physicalists, they might still hold that additional conceptual/philosophical/metaphysical work is required from our part.
-
hardcore physicalist accounts: I think virtually everyone in Camp #1 is a physicalist, whereas camp #2 is split. So this doesn't put you in camp #2.
-
getting your metaphysics right: well, this formulation was dumb since, as you say, needing to not bring strange metaphysics into the picture is also one way of getting it right. What I meant was that the metaphysics is nontrivial.
I've just rewritten the descriptions of the two camps. Ideally, you should now fully identify with the first. (Edit: I also rewrote the part about consciousness being fuzzy, since I think that was poorly phrased even if it didn't cause issues here.)
Thanks for that comment. Can you explain why you think you're Camp #2 according to the post? Because based on this reply, you seem firmly (in fact, quite obviously) in Camp #1 to me, so there must be some part of the post where I communicated very poorly.
( ... guessing for the reason here ...) I wrote in the second-last section that consciousness, according to Camp #1, has fuzzy boundaries. But that just means that the definition of the phenomenon has fuzzy boundaries, meaning that it's unclear when consciousness would stop being consciousness if you changed the architecture slightly (or built an AI with similar architecture). I definitely didn't mean to say that there's fuzziness in how the human brain produces consciousness; I think Camp #1 would overwhelmingly hold that we can, in principle, find a full explanation that precisely maps out the role of every last neuron.
Was that section the problem Or sth else?
I tend to think that, regardless of which camp is correct, it's unlikely that the difference is due to different experiences, and more likely that one of the two sides is making a philosophical error. Reason being that experience itself is a low-level property, whereas judgments about experience are a high-level property, and it generally seems to be the case that the variance in high-level properties is way way higher.
E.g., it'd be pretty surprising if someone claimed that red is more similar to green than to orange, but less surprising if they had a strange idea about the meaning of life, and that's pretty much true regardless of what exactly they think about the meaning of life. We've just come to expect that pretty much any high-level opinion is possible.
Yes; there are definitely people who disagree with most things Dennett says, including how exactly you can be wrong about your experience. Don't really want to get into the details here since that's not part of the post.
Yeah, I agree with both points. I edited the post to reflect it; for the whole brain vs parts thing I just added a sentence; for the kind of access thing I made it a footnote and also linked to your comment. As you said, it does seem like a refinement of the model rather than a contradiction, but it's definitely important enough to bring up.
Thanks!
First, I don't understand why IIT is still popular, Scott Aaronson showed its fatal shortcomings 10 years ago, as soon as it came out
Well, Scott constructed an example for which the theory gives a highly unintuitive result. This isn't obviously a fatal critique; you could always argue that a lot of theories give some unintuitive results. It's also the kind of thing you could maybe fix by tweaking the math,[1] rather than tossing out the entire approach.
I believe Tononi is on record somewhere biting the bullet on that point (i.e., agreeing that Scott's construction would indeed have high , and that that's okay). But I don't know where, and I think I already searched for it a few months ago (probably right after IIT4.0 was dropped) and couldn't find it.
Second, I do not see any difference between experiencing something and claiming to experience something, outside of intentionally trying to deceive someone.
Third, I don't know which camp I am in, beyond "of course consciousness is an emergent concept, like free will and baseball". Here by emergence I mean the Sean Carroll version: https://www.preposterousuniverse.com/blog/2020/08/11/the-biggest-ideas-in-the-universe-21-emergence/
I think this puts you firmly into Camp #1 (though you saying this proves that, at a minimum, the idea wasn't communicated as clearly as I'd hoped). Like, the introductory dialogue shows someone failing to communicate the difference, so if this difference isn't intuitively obvious to you, this would be a Camp #1 characteristic.
And like, since the whole point was that [trying to articulate what exactly it means for experience to exist independently of the report] is extremely difficult and usually doesn't work, I'm not gonna attempt it here.
Though as mentioned in another comment, I haven't actually read through the construction -- I always just trusted Scott here -- so maybe I'm wrong. ↩︎
I think a lot of Camp #2 people would agree with you that IIT doesn't make meaningful progress on the hard problem. As far as I remember, it doesn't even really try to; it just states that consciousness is the same thing as integrated information and then argues why this is plausible based on intuition/simplicity/how it applies to the brain and so on.
I think IIT "is Camp #2 stuff" in the sense that being in Camp #2 is necessary to appreciate IIT - it's definitely not sufficient. But it does seem necessary because, for Camp #1, the entire approach of trying to find a precise formula for "amount of consciousness" is just fundamentally doomed, especially since the math doesn't require any capacity for reporting on your conscious states, or really any of the functional capabilities of human consciousness. In fact, Scott Aaronson claims (haven't read the construction myself) here that
the system that simply applies the matrix W to an input vector x—has an enormous amount of integrated information Φ
So yeah, Camp #2 is necessary but not sufficient. I had a line in an older version of this post where I suggested that the Camp #2 memeplex is so large that, even if you're firmly in Camp #2, you'll probably find some things in there that are just as absurd to you as the Camp #1 axiom.
I don't get the downvotes, this post is just agreeing with the OP.
does it play better / make legal moves for longer this way?
LW is a known hotbed of compatibilism, so here's my question:
That's not been my impression. I would have summarized it more as "LW (a) agrees that LFW doesn't exist and (b) understands that debating compatibilism doesn't make sense because it's just a matter of definition"
Personally, I certainly don't consider myself a compatibilist (though this is really just a matter of preference since there are no factual disagreements). My brief answer to "does free will exist" is "no". The longer answer is the within-physics stick figure drawing.
No one will hear my counter-arguments to Sabien's propaganda who does not ask me for them privately.
uh, why? Why not make a top level post?
Was about to reread this, but
UPDATE IN 2023: I wrote this a long time ago and you should NOT assume that I still agree with all or even most of what I wrote here. I’m keeping it posted as-is for historical interest.
... are your updated thoughts written up anywhere?
This argument is based on drawing an analogy between
- Humans building an AI; and
- An AI improving itself
in the sense that both have to get their values into a system. But the two situations are substantially disanalogous because the AI starts with a system that has its values already implemented. it can simply improve parts that are independent of its values. Doing this would be easier with a modular architecture, but it should be doable even without that. It's much easier to find parts of the system that don't affect values than it is to nail down exactly where the values are encoded.
Fair. Ok, I edited the original post, see there for the quote.
One reason I felt comfortable just stating the point is that Eliezer himself framed it as a wrong prediction. (And he actually refers to you as having been more correct, though I don't have the timestamp.)
(Eliezer did think neural nets wouldn't work; he explicitly said it on the Lex Fridman podcast.)
Edit @request from gwern: at 11:30 in the podcast, Eliezer says,
back in the day I went around saying like, I do not think that just stacking more layers of transformers is going to get you all the way to AGI, and I think that GPT-4 is past where I thought this paradigm is going to take us, and I, you know, you want to notice when that happens, you want to say like "oops, I guess I was incorrect about what happens if you keep on stacking more transformer layers"
and then Fridman asks him whether he'd say that his intuition was wrong, and Eliezer says yes.
Why AI would want to align us or end us is something I still haven't figured out after reading about alignment so much.
Has your reading ever included anything related to Instrumental Convergence?