LessWrong 2.0 Reader

This seems pretty cool! The data augmentation technique proposed seems simple and effective. I'd be interested to see a scaled-up version of this (more harmful instructions, models etc). Also would be cool to see some interpretability studies to understand how the internal mechanisms change from 'deep' alignment (and compare this to previous work, such as https://arxiv.org/abs/2311.12786, https://arxiv.org/abs/2401.01967) 

As it often does when I write, this ended up being pretty long (and not especially well written by the standards I wish I lived up to).

I'm sure I did misunderstand part of what you are saying (that we do misunderstand easily was the biggest part of what we appear to agree on), but also, my disagreements aren't necessarily things you don't actually mention yourself. I think we disagree mostly on what outcomes the advice itself will give if adopted overly eagerly, because I see the bad way of implementing them as being the natural outcome. Again, I think your 8th point is basically the thrust of my criticism. There is no script you can actually follow to truly understand people, because people are not scripted.

Note: I like to think I am very smart and good at understanding, but in reality I think I am in some ways especially likely to misunderstand and to be misunderstood. (Possible reason: Maybe I think strangely as compared to other people?) You can't necessarily expect me to come at things from a similar angle as other people, and since advice is usually intended as implicitly altering the current state of things, I don't necessarily have a handle on that.

Importantly, since they were notes, I took them linearly, and didn't necessarily notice if my points were addressed sufficiently later.

Also, I view disagreements as part of searching for truth, not for trying to convince people you are right. Some of my distaste is that it feels like the advice is being given for persuasion more than truthseeking? (Honestly, persuasion feels a little dirty to me, though I try to ignore that since I believe there isn't actually anything inherently wrong with persuasion, and in many cases it is actually good.) Perhaps my writing would be better / make more sense if I was more interested in persuading people?

An important note on my motives for the comment is that I went through with posting it when I think I didn't do particularly well (there were obvious problems) in part to see how you would respond to it. I don't generally think my commenting actually helps so I mostly don't, but I've been trying out doing it more lately. There are also plenty of problems with this response I am making.

Perhaps it would have been useful for me to refer to what I was writing about by number more often.

Some of the points do themselves seem a bit disrespectful to me as well. (Later note: You actually mention changing this later and the new version on Karma is fine.) Like your suggestion for how to change the mind of religious people (though I don't actually remember what I found disrespectful about it at this moment). (I am not personally religious, but I find that people often try to act in these spaces like religious people are automatically wrong which grates on me.)

Watching someone else having a conversation is obviously very slow, but there is actually a lot of information in any given conversation.

Random take: The first video is about Karma, which I do have an opinion on. I believe that the traditional explanation of Karma is highly unlikely, but Karma exists if you think of it as "You have to live with who you are. If you suck, living with yourself sucks. If you're really good, living with yourself is pretty great." See also, "If you are in hell, it's a you thing. If you are in heaven, it's also a you thing." There are some things extreme enough where that isn't really true, like when being actively tortured, but your mind is what determines how your life goes even more than what events actually happen in the normal case, and it does still effect how you react to the worst (and best) things. (People sometimes use the story about a traveler asking someone what the upcoming town is like, and the person just asking the traveler what people in the previous place were like, while answering 'much the same' for multiple travelers with different outlooks and I do think this is somewhat true.)

Also, doing bad things can often lead to direct or indirect retaliation, and good to direct or indirect reward. Indirect could definitely 'feel' like Karma.

I think that the actual key to a successful conversation is to keep in mind what the person you are talking to actually wants from the conversation, and I would guess what people mostly want from a random conversation is for the other person to think they are important (whenever they don't have an explicit personal goal from the conversation). I pretty much always want to get at the truth as my personal goal because I'm obsessive that way, but I usually have that goal as an attempt at being helpful.

It seems to work for him getting his way, and nothing he does is too bad, but the conversational tactics seem a bit off to me. (Only a bit.) It seems like he is pushing his own ideas too much on someone else who is not ready for the conversation (though they are happy enough to talk).

No, I don't know any way to make sure your conversation partner is ready for the conversation. A lot of evidence for your position is not available in a synchronous thing like a conversation, and I believe that any persuasion should attempt to be through giving them things they can think through later when not under time pressure. He didn't exactly not do that, but he also didn't do that. "You must decide now" (before the end of the conversation) seemed to be a bit of an undercurrent to things. (A classic salesman tactic, but I don't like it. And sure, the salesman will pivot toward being willing to talk to you again later if you don't bite on that most of the time, but that doesn't mean they weren't pressuring you to decide quickly.)

The comparison between 'Karma' and 'Santa' seems highly disrespectful and unhelpful. They are very different things and the analogy seems designed to make things unclear more than clearing them up. In other words, I think it is meant to confuse the issue rather than give genuine insight. You could object that part of the Santa story is literally Karma (the naughty list) but I don't think that makes the analogy work.

I don't really get the impression he was actually willing to be convinced himself. He said at one point that he was willing to, and maybe in the abstract he is, but he never seemed to seek information against his own position. Note that I don't think I would necessarily be able to tell, and I actually disapprove of 'mindreading' quite strongly.

The fact that I am strongly against 'mindreading' and actually resort to it myself automatically is actually one of the points I was trying to make about how easy it is to misuse conversational tactics. I was genuinely trying to understand what he was doing, (in service of making a response based on it) and I automatically ended up thinking he was doing the opposite of what he claimed, just based on vibes without any real evidence.

You could argue I am so against it because I notice myself doing it, and maybe it is true, but I find it infuriating when others do it badly. I don't actually have any issues with them guessing what I'm doing correctly, though I'm unlikely to always be right about that either (just more than other people about me).

He also didn't seem entirely open that he was pushing for a specific position throughout the entire conversation, when he definitely was. This wasn't a case of just helping someone update on the information they have (though there was genuinely a large amount of that too.) (People do need help updating and it can be a valuable service, but for it to really be helpful, it needs to not be skewed itself.)

The second video (about convincing someone to support trans stuff) seems pretty strange. This video seems completely different from the previous one; more propaganda than anything. Clearly an activist (and I generally dislike activists of any stripe.). (Emotional response: Activists are icky.) Also an obviously hot culture war issue (which I have an opinion on though I don't think said opinion is relevant to this discussion). It's also very edited down which makes it feel even more different.

The main tactic seems like trying to force a person to have an emotional reaction through manipulative personal stories (though he claims otherwise and there are other explanations). But he seemed to do it over and over again, so this time I am pretty sure he isn't being entirely honest about that (even though I still disapprove of mindreading like I am doing here.). I feel like he is a bad person (though not unusually so for an activist.)

The alternate explanation, which does work, is just that people like to tell stories about themselves when talking about any subject. I clearly reference myself many times in this response and my original response. I'm not saying I'm being fair in my conclusions.

Do you really see those two videos as similar? While there are some similarities, they feel quite different to me! I didn't love it, but the first video was about talking through the other person's points and having a genuine conversation. The latter was about taking advantage of their conversation partner's points for the next emotional reaction. In other words, the latter video felt a lot more like tricking someone while the former was a conversation.

Moving past the videos to the rest of the response.
Yes, the switch to the longer way of rephrasing that includes explicitly accepting that you might be wrong seems much, much better. Obviously, it is best for the person to really believe they might be wrong, and saying it both helps an honest participant remember that, and should make it easier for the person they are talking with to correct them. Saying the words isn't enough, but I like it a lot better than before.

Obviously, I'm not rephrasing your points because that still isn't how I believe it should be done, but if there is a key point this way of asking about it can be very useful. Or, to rephrase, rephrasing is a tool to be used on points that seem to be particularly important to check your understanding of.

I don't remember exactly what you said in point 4 before you changed it, but I don't particularly read point 5 as being anti personal experience in the way my comment indicates. I have no idea why I would possibly write that about point 5 so I assume you are correct in your assumption.

Since I only vaguely remember it, my memory only contains the conclusion I came to which we both agree can be faulty. But the way I remember it, the old point 4 is very clearly a direct attack on personal experience in general rather than on distinguishing between faulty and reliable personal experience. From past experience, this could be attributable to many things, including just not reading a few words along the way.

I don't really have any issues with your new point 4 (and it is clearly taken from that first video.) That is very obviously a good approach for convincing people of things that doesn't rely on anything I find distasteful. It seems very clearly like what you are saying you are going for and I think it works very well.

For the record, I think 'working definition' is no more different from 'mathematical definition' than 'theoretical definition' is from 'mathematical definition' because I am using 'theoretical definition' in a colloquial way. I was definitely not saying mathematical definitions or formal definitions are useful when talking to a layperson. (Side note: I've been paying attention to 'rationalists' of this sort for about 20 years now, but I am not one. I tend to resist becoming part of groups.) I do generally think that unless you are in the field itself that 'formal definitions' are not helpful since they take far too much time and effort that could be used on other things (and formal definitions are often harder to understand even afterward in many cases), and mathematical definitions are unnatural for most people even after they understand them.

I do not want people spending more time on definitions in conversation unless it is directly relevant, but think remembering that there are different kinds of brief definitions seems important to me.

I perhaps overreacted to the mention of Bayes Rule. It's valid enough for describing things in probalistic circumstances, but people in this community try to stretch it to things that aren't probability theory related and it's become a bit of a pet peeve for me. I have never once used Bayes Rule to update my own beliefs (unless you include using it formally to decide what answer to give on a probability question a few times in school), but people act like that is how it is done.

In the paper on 'Erotectic' reasoning, ... includes a pretty weird bit of jargon on their very first example (first full paragraph of second page) which makes it hard to understand their point. And not only do they not explain, it isn't even something I could look up with a web search because all explanations are paywalled seemingly. They claim it is a well-known example, but it clearly isn't.

As best as I can tell, the example is really just them abusing language. Because 'or else' is closer to 'exclusive or' but they are pretending it is just 'or'. (It is a clear misuse to pretend it doesn't.) I don't know philosophy jargon well, but misstating the premise is not clever. In this case, every word of the premise mattered, and they intentionally picked incorrect ones. Their poor writing wasted a great deal of time. And yes, I am actually upset at them for it. I kept looping back around to being upset about their actions and wanting to yell at them rather than considering what they were writing about. (Which is an important point I suppose, if the person you are conversing with is upset with you, things are reasonably likely to go badly regardless of whether your points are good or bad.) I think it is the most upset I've been reading a formal paper (though I mostly have only really read a small number of AI and/or Math ones.)

In the end I could tell I wasn't going to stop if I kept reading, so I quit without understanding what they were writing about. (I can definitely be a bit overboard sometimes.) All I got was that they think there is some way to ask questions that works with the basic reasoning people normally use and leads to deductively valid reasoning. I have no idea what method of questioning they are in favor of or why they think it works. (I do think the example could have been trivially changed and not bothered me.) I do think my emotional reaction is a prime example of a reason resolving disagreements often doesn't work (and even why 'fake disagreements' where the parties actually agree on the matter can fail to be resolved).

To really stress point 8 it should be point 1. I was just saying it needed to be stressed more. I did notice you saying it was important and I was agreeing with you for the most part. Generally you evaluate points based on what came before, not based on what came after (though it does happen). It's funny, people often believe they are disagreeing when they are just focusing on things they actually agree on in a different manner.

On a related note, it's not like I'm ordering this stuff in order of how important I think it is. Sometimes things fit better in a different order than importance (this is obviously in order of what I am responding to.) (Also, revising this response on a global scale would take far too long given how long it already takes me to write comments. It might be worth writing shorter but better in the same amount of time, but I don't seem inclined to it.)

You know what, since I wrote that I had a lot of disagreements, I really should have pointed out that not all of the things I was writing were disagreements! I think my writing often comes off as more negative than I mean it (and not because other people are reading it badly).

On the note of it being a minimum viable product, I think those are very easy to badly. You aim for what you personally think is the minimum... when you already know the other stuff you are trying to say. It is then often missing many things that are actually necessary for it to work, which makes it just wrong. I get the idea, perfectionism is slow, a waste of resources, and even impossible, but aiming for the actual minimum is a bad idea. It is often useful advice for startups, but we do not want to accept the same rate of failure as a startup business! Virtually all of them fail. We should aim more for the golden mean in something like a formal post like you made. (A high rate of failure in comments/feedback seems fine though since even mostly failed comments can spark something useful.)

As far as quoting the first sentence of each thing I am responding to, that does sound like a useful idea, and I should do it, but I don't think I am going to anyway. For some reason I dread doing it (especially at this point). I also don't even know how to make a quote on lesswrong, much less a partial one. I know I don't necessarily signpost well exactly what I am responding to. (Plus, I actually write this in plaintext in notepad rather than the comment area. I am paranoid about losing comments written on a web interface since it takes me so long to write them.)

Thanks!  These are thoughtful points. See some clarifications below:
 

AGI could be very catastrophic even when it stops existing a year later.

You're right. I'm not even covering all the other bad stuff that could happen in the short-term, that we might still be able to prevent, like AGI triggering global nuclear war.

What I'm referring to is unpreventable convergence on extinction.
 

If AGI makes earth uninhabitable in a trillion years, that could be a good outcome nonetheless.

Agreed that could be a good outcome if it could be attainable.

In practice, the convergence reasoning is about total human extinction happening within 500 years after 'AGI' has been introduced into the environment (with very very little probability remainder above that).

In theory of course, to converge toward 100% chance, you are reasoning about going across a timeline of potentially infinite span.
 

I don't know whether that covers "humans can survive on mars with a space-suit",

Yes, it does cover that. Whatever technological means we could think of shielding ourselves, or 'AGI' could come up with to create as (temporary) barriers against the human-toxic landscape it creates, still would not be enough.
 

if humans evolve/change to handle situations that they currently do not survive under

Unfortunately, this is not workable. The mismatch between the (expanding) set of conditions needed for maintaining/increasing configurations of the AGI artificial hardware and for our human organic wetware is too great. 

Also, if you try entirely changing our underlying substrate to the artificial substrate, you've basically removed the human and are left with 'AGI'. The lossy scans of human brains ported onto hardware would no longer feel as 'humans' can feel, and will be further changed/selected for to fit with their artificial substrate. This is because what humans and feel and express as emotions is grounded in the distributed and locally context-dependent functioning of organic molecules (eg. hormones) in our body.

The way intuitive models work (I claim) is that there are concepts, and associations / implications / connotations of those concepts. There’s a core intuitive concept “carrot”, and it has implications about shape, color, taste, botanical origin, etc. And if you specify the shape, color, etc. of a thing, and they’re somewhat different from most normal carrots, then people will feel like there’s a question “but now is it really a carrot?” that goes beyond the complete list of its actual properties. But there isn’t, really. Once you list all the properties, there’s no additional unanswered question. It just feels like there is. This is an aspect of how intuitive models work, but it doesn’t veridically correspond to anything of substance.

Mhhhmhh. Let me see if I can work with the carrot example to where it fits my view of the debate.

A botanist is charged with filling a small field with plants, any plants. A chemist hands him a perfect plastic replica of a carrot, perfect in shape, color, texture, and (miraculously) taste. The botanist says that it's not a plant. The chemist, who has never seen plants other than carrots, points out the matching qualities to the plants he knows. The botanist says okay but those are just properties that a particular kind of plant happens to have, they're not the integral property of what makes something a plant. "The core intuitive concept 'plant' has implications about shape, color, texture, taste, et cetera", says the chemist. "If all those properties are met, people may think there's an additional question about the true plant-ness of the object, but [...]." The botanist points out that he is not talking about an intangible, immeasurable, or non-physical property but rather about the fact that this carrot won't grow and spread seeds when planted into the earth. The chemist, having conversed extensively with people who define plants primarily by their shape, color, texture, and taste (which are all those of carrots because they've also not seen other plants) just sighs, rolling his eyes at the attempt to redefine plant-ness to be entirely about this one obscure feature that also just happens to be the most difficult one to test.

Which is to say that I get -- or at least I think I get -- the sense that we're successfully explaining important features of consciousness and the case for linking it to anything special is clearly diminishing -- but I don't think it's correct. When I say that the hard meta problem of seeing probably contains ~90% of the difficulty of the hard meta problem of consciousness whereas the meta problem of free will contains 0% and the problem of awareness ~2%, then I'm not changing my model in response to new evidence. I've always thought Free Will was nonsense!

(The botanist separately points out that there in fact other plants with different shape, texture, and taste, although they all do have green leaves, to which the chemist replies that ?????. This is just to come back to the point that people report advanced meditative states that lose many of the common properties of consciousness, including Free Will, the feeling of having a self (I've experienced that one!) and even the presence of any information content whatsoever, and afaik they tend to be more "impressed", roughly speaking, with consciousness as a result of those experiences, not less.)

[seeing stuff]

Attempt to rephrase: the brain has several different intuitive models in different places. These models have different causal profiles, which explains how they can correspond to different introspective reports. One model corresponds to the person talking about smelling stuff. Another corresponds to the person talking about seeing stuff. Yet another corresponds to the person talking about obtaining vague intuitions about the presence and location of objects. The latter two are triggered by visual inputs. Blindsight turns off the second but not the third.

If this is roughly correct, my response to it is that proposing different categories isn't enough because the distinction between visually vivid experience and vague intuitions isn't just that we happen to call them by different labels. (And the analogous thing is true for every other sensory modality, although the case is the least confusing with vision.) Claiming to see a visual image is different from claiming to have a vague intuition in all the ways that it's different; people claim to see something made out of pixels, which can look beautiful or ugly, seems to have form, depth, spatial location, etc. They also claim to perceive a full visual image constantly, which presumably isn't possible(?) since it would contain more information than can actually be there, so a solution has to explain how this illusion of having access to so much information is possible. (Is awareness really a serial processor in any meaningful way if it can contain as much information at once as a visual image seems to contain?)

(I didn't actually intend to get into a discussion about any of this though, I was just using it as a demonstration of why I think the hard metaproblem of consciousness has at least one real subset and hence isn't empty.)

Hard Problem

Yeah, I mean, since I'm on board with reducing everything to the meta problem, the hard problem itself can just be sidestepped entirely.

But since you brought it up, I'll just shamelessly use this opportunity to make a philosophical point that I've never seen anyone else make, which is that imo the common belief that no empirical data can help distinguish an illusionist from a realist universe... is actually false! The reason is that consciousness is a high-level phenomenon in the illusionist universe and a low phenomenon in at least some versions of the realist universe, and we have different priors for how high-level vs. low-level phenomena behave.

The analogy I like is, imagine there's a drug that makes people see ghosts, and some think these ghosts tap into the fundamental equations of physics, whereas others think the brain is just making stuff up. One way you can go about this is to have a thousand people describe their ghosts in detail. If you find that the brightness of hallucinated ghosts is consistently proportional to their height, then you've pretty much disproved the "the brain is just making stuff up hypothesis". (Whereas if you find no such relationships, you've strengthened the hypothesis.) This is difficult to operationalize for consciousness, but I think determining the presence of absence of elegant mathematical structure within human consciousness is, at least in principle, an answer to the question of "[w]hat would progress on the 'breathes fire' question even look like".

Georgism for sure. When we build the political will to reclaim all titles to land and charge leases held for the public by the government, UBI is easily and appropriately funded. As you say though, the question is whether it will ever be enough to live on given other market forces.

My hope has been that a system like the one I described can get ahead of all of it by defining the minimum purchasing power as exactly what’s necessary to live on. In principle this only works when the economy has a high post scarcity multiplier, that is, one unit of work gives you four or more units of goods. As you mentioned before, automation may not be there yet, but as a hint of things to come, our current per capita GDP is double the current living wage floor.

Thanks for making this!

I found it a challenge to deduce strategies over many plays, rather than following the advice "not intended to be replayed". The first playthrough was pretty much meaningless for me, especially given the knowledge that both time and history could affect the results. I just viewed it as one step of information gathering for the real game.

The suboptimal zones weren't obviously suboptimal from a single pass, even Dragon Lake that always yields nothing. For all I knew, it could have yielded 5000 food with quite a low probability (and still be always optimal), or lesser amounts of food at specific combinations of time and day, or only when matching some rule based on the previous results of foraging in other zones.

After many runs I did settle on a strategy, and mentally scored myself by looking at the source to see whether there was anything that I should have spotted but didn't. As it happened, my final strategy was almost optimal though I stayed on the rats for a few more hours than ideal.

Curated!^[1] This thinks through a specific threat model with future AIs, and how to concretely mitigate it. I don't have a strong opinion on whether it works, but I really want to encourage more concrete analysis of this sort.

1. ^{^}

   "Curated", a term which here means "This just got emailed to 30,000 people, of whom typically half open the email, and it gets shown at the top of the frontpage to anyone who hasn't read it for ~1 week."

I'd missed that, thank you for pointing that out.

If "expected" effectively means what you're saying is that you're being offered a bet that is good by definition, that even at 50/50 odds, you take the bet, I suppose that's true. If the bet is static for a second flip, it wouldn't be a good deal, but if it dynamically altered such that it was once again a good bet by definition, I suppose you keep taking the bet.

If you're engaging with the uncertainty that people are bad at evaluating things like "expected utility" then at least some of the point is that our naive intuitions are probably missing some of the math, and costs, and the bet is likely a bad bet.

If I was trying to give credence to that second possibility, I'd say that the word "expected" is now doing a bunch of hidden heavy lifting in the payoff structure, and you don't really know what lifting it's doing.

I might as well check out the panel discussion. I didn't know about it.

I think I listened to the Hotz debate. The highlight of that one was when Hotz implied that he was using an LLM to drive a car, Yudkowsky freaks out a bit, and Hotz clarifies that he means the architecture for his learning algorithm is basically the same as an LLM.

I suspect the Destiny discussion is qualitatively similar to the Dwarkesh one.

At this point, maybe I should just read old MIRI papers.

Ok, I'm agreeing in principle to make the same bet as with RatsWrongAboutUAP.

("I commit to paying up if I agree there's a >0.4 probability something non-mundane happened in a UFO/UAP case, or if there's overwhelming consensus to that effect and my probability is >0.1.")