LessWrong 2.0 Reader

"harness" is doing a lot of work there. If incoherent search processes are actually superior then VNM agents are not the type of pattern that is evolutionary stable, so no "harnessing" is possible in the long term, more like a "dissolving into".

Unless you're using "VNM agent" to mean something like "the definitionally best agent", in which case sure, but a VNM agent is a pretty precise type of algorithm defined by axioms that are equivalent to saying it is perfectly resistant to being Dutch booked.

Resistance to Dutch booking is cool, seems valuable, but not something I'd spent limited compute resources on getting six nines of reliability on. Seems like evolution agrees, so far: the successful organisms we observe in nature, from bacteria to humans, are not VNM agents and in fact are easily Dutch booked.  The question is whether this changes as evolution progresses and intelligence increases.

If AI turns out to be very useful for cheaply writing formally verified code, what does that do for AI control? We can now request that the untrusted AI produce along with any code it writes a spec and certificate verifying that the code matches the spec.

How bad of a position does this put the untrusted model in, when it's trying to write code that does bad stuff? Some sub-questions towards answering that question:

• How tight are the constraints that the formal verification process, at its AI-driven heights, can put on the code? Seems like once something is formally verified at all this rules out large classes of vulnerabilities an AI might exploit. Can we push this further to remove almost all attack vectors?
• How easy is it to write a spec that looks good to humans, but actually contains a significant omission?
• How feasible is it for the spec writer to collude with the code writer?

A version of what ChatGPT wrote here prompted

What was the prompt?

Specifically, I assume you’re objecting to the first part “approximately everybody senses women’s cues whether they recognize it or not”, but you don’t seem to be noticing the distinction between “sensing the cue” and “being aware that you have sensed a cue, and having concluded that it counts as an a cue”. These are easy to conflate, but they’re wildly different things.

What is the distinction between “sensing the cue” and “being aware that you have sensed a cue”?

If, in “approximately everybody senses women’s cues whether they recognize it or not”, you mean by the phrase “sense women’s cues” something which can be done without even being aware that you have sensed anything, then what exactly are you saying? Is this claim falsifiable at all? How would you falsify it?

How would the world look different in the following two cases:

1. Everyone senses women’s cues, although some (many?) people do this without having any awareness at all that they have sensed anything whatsoever.

2. Some people do not sense women’s cues.

What observations would you expect to make in one of those scenarios but not in the other?

Heck, in high school a girl told my mom that she wanted to have my baby, and I was still telling my friends “You’re wrong, she’s not into me”. I get that.

Which is perfectly sensible, because I have in fact encountered cases where women would say things like that to men, but give no other indication of being interested in said men, and would react with bafflement to suggestions that they were interested in dating said men, etc. (Being a third party in these cases, I could observe in a disinterested way, and found these observations quite instructive.)

The same, by the way, can be said of this:

But if she holds unbroken eye contact, flat out doesn’t respond to anything he says in attempt to distract her or test her resolve, and leans all the way in until her lips are mere millimeters from his, waiting for minutes until he responds.… that’s yin, but not something that can be missed, you know?

I have absolutely known women who have done this with guys whom they had no intention whatsoever to kiss or do anything else with.

At the same time, I couldn’t help but respond to what she was actually doing. When she’d jump in front of me to try to stop me and get my attention, I could change direction and walk around her, but I had to sense her presence in order to do that. Sure, I had alternate explanations for why she was doing what she was doing, but that shows recognition of the thing to be explained.

As far as I can tell, you are using the phrase “sense a cue” in a way that I can only describe as completely useless. Obviously it is impossible to literally not perceive the the physical presence of a woman in front of you (assuming that you aren’t blind, it’s not dark, your eyes aren’t closed, etc.), but that is not what anyone means by “sense a cue”. What the phrase means is “recognize a specific action or behavior as a cue”.

“Recognition of the thing to be explained” is worth nothing.

For example, suppose I am walking down the street and I see a woman walking in the opposite direction toward me. Is this a “thing to be explained”? Taking the broad view, sure. What might the explanation be? Obviously it is “this woman happens to have some reason to be walking in that direction, just like any number of people who walk down streets every day”. What is the correct reaction? Nothing; no reaction is required or appropriate.

Or: I’m riding the subway and a woman brushes my arm as she walks past. Is this a “thing to be explained”? Sure. What might the explanation be? Obviously it is “it’s a crowded subway car and there’s no particular reason to take extreme care not to make physical contact with anyone, and this sort of accidental casual contact happens all the time”. What is the correct reaction? Ignore it; no reaction is required or appropriate.

Or: the woman in the cafe is smiling when she hands me my order. Is this a “thing to be explained”? Sure. What might the explanation be? Obviously it is “she smiles at everyone; she just smiled at the old lady before me, and now she’s smiling at the couple after me; being friendly is good for business, probably”. What is the correct reaction? Smile back, be polite, otherwise nothing; no special reaction is required or appropriate.

In each of these cases, I “recognized the thing to be explained”; and the explanation for thing to be explained was it was a totally mundane behavior that had nothing to do with anyone sending any cues.

In the cases of failing to notice cues, what happens is that a man “recognizes the thing to be explained” in the same way that I “recognized the thing to be explained” in my three examples above; he identifies the explanation for the thing as something mundane; he then does not react in any particular way, because generally in such mundane cases no special reaction is required or appropriate. But whoops! Actually the woman was strenuously sending all sorts of cues! But the man completely failed to perceive any of them, because the idea that these cues were somehow extremely obvious was just a fantasy in the woman’s head. All the man perceived was various behaviors which have various mundane explanations, just like the overwhelming majority of behaviors in which we engage every day.

So, back to the actually useful question: is it possible to not recognize a behavior or action as a deliberate cue that is being sent? Yes, absolutely it’s possible, it happens all the time and is definitely the explanation for approximately 100% of cases of “woman sends cues, guy doesn’t respond, woman does nothing more and doesn’t get date”, exactly as @johnswentworth wrote.

The problem with your argument is that it doesn’t at all explain all the cases where

I understand that I hadn't made this part very clear in the comments I made prior to this comment of yours. I have since addressed it in my latest response to johnswentworth.

But of course this is an absurd requirement. If she knows he’s going to be interested, of course that makes it vastly easier!

It's also not the real requirement, as I was cutting corners for sake of brevity at the cost of precision. The actual requirement is trickier to nail down both concretely and concisely, but it's closer to "she has to know he respects her" to be still a little inaccurate or "it has to be worth it, in expectation" to be tautological but vague.

It's completely possible for a woman to not know if a man is interested or not, and be absolutely terrified of how he might respond, and find the courage to find out anyway. It's also something that can be deliberately facilitated by the man, if he understands why she hasn't made herself more open to him as of yet.

… and yet, according to your own account, women still won’t say anything in that situation, despite having a guarantee of a positive response. What does that tell you?

Oh, no. Women will absolutely ask you out if you put them in a position where you guarantee a positive response and won't give one until they ask you out.

In the situations where they don't, it tells you that they don't have to. If a beautiful woman physically takes the distraction out of your hands, drags you to her bedroom, and undresses herself... does she really have to ask? Or does it maybe go without saying?
 

In fact there is no such as “forcing the man to contend with” anything. People (not just men) are, as it turns out, perfectly capable of totally ignoring a cue like this, and indeed of not even noticing it in the first place. A woman who thinks that leaning in and waiting to be kissed is somehow a guarantee that a man will correctly perceive the cue, is sadly, sadly mistaken.

You're not thinking of the situation I'm describing.

I'm referring to the situation I outlined in my first comment:

But if she holds unbroken eye contact, flat out doesn't respond to anything he says in attempt to distract her or test her resolve, and leans all the way in until her lips are mere millimeters from his, waiting for minutes until he responds.... that's yin, but not something that can be missed, you know?

It's not just "leaning in waiting to be kissed". It's actively taking steps to ensure that it cannot be missed. It's flat out refusing to engage with anything else -- and when that's not enough, holding her face close enough to his that her nose is touching his cheeks until he addresses it.

If this were to ever happen to you, you might not be able to make sense of it. You might not know what to do with it. You might rationalize that she's not interested just weird. You might choose to run away, rather than engage with it. You might even deny noticing anything strange has happened.

But what you can't do is fail to notice.

Hey, why don't you talk to Carla anymore? "Dude, she got weird. She won't respond to anything I say and always tries to press her face up against mine. I dunno, I guess we just drifted apart? Nothing out of the ordinary happened."

I'm not buying it.

Sorry, but this is empirically false.

With all due respect, I don't think you're in a position of being able to judge the veracity of claims like this. Before you can judge a claim true or false, you have to understand what the claim is -- and it doesn't look like you do.

Specifically, I assume you're objecting to the first part "approximately everybody senses women’s cues whether they recognize it or not", but you don't seem to be noticing the distinction between "sensing the cue" and "being aware that you have sensed a cue, and having concluded that it counts as an a cue". These are easy to conflate, but they're wildly different things.

There's essentially no limit to how far a man can explain away a woman's cues. Heck, in high school a girl told my mom that she wanted to have my baby, and I was still telling my friends "You're wrong, she's not into me". I get that.

At the same time, I couldn't help but respond to what she was actually doing. When she'd jump in front of me to try to stop me and get my attention, I could change direction and walk around her, but I had to sense her presence in order to do that. Sure, I had alternate explanations for why she was doing what she was doing, but that shows recognition of the thing to be explained.

Even on subtle things where it's like "The woman at the front desk smiles and says 'Hi'", I might not know that she's smiling a little more (or a little less) than her normal or what the exact significance is, but the woman I'm going to be responding to is the woman smiling as much as she's smiling -- and people respond differently to people who are warmer towards them than than those who are slightly less warm. She will still be able to steer my behavior on the margin by how inviting she is towards me, even if I don't have a moment of "Oh, THIS IS A CUE!".

People sense, with some noise, the degree to which people smile at them, touch them, laugh at their jokes, bend over in front of them, etc. People respond to what they sense, because it is the only thing to respond to. If they were presented with something different they would be responding to something different. This is all that subtle clues are intended to do. Whether these cues overcome priors, and whether the person in question allows themselves to admit when it does, is an entirely separate question.
 

It sounds like you're viewing the goal of thinking about DT as: "Figure out your object-level intuitions about what to do in specific abstract problem structures. Then, when you encounter concrete problems, you can ask which abstract problem structure the concrete problems correspond to and then act accordingly."

I think that approach has its place. But there's at least another very important (IMO more important) goal of DT: "Figure out your meta-level intuitions about why you should do one thing vs. another, across different abstract problem structures." (Basically figuring out our "non-pragmatic principles" as discussed here [LW · GW].) I don't see how just asking Claude helps with that, if we don't have evidence that Claude's meta-level intuitions match ours. Our object-level verdicts would just get reinforced without probing their justification. Garbage in, garbage out.

This is actually my primary focus. I believe it can be done through a complicated process that targets human psychology, but to explain it simply 

- Spread satisfaction & end suffering.
- Spread rational decision-making

To further simplify, if everyone was like us, and no one was on the chopping block if AGI doesn't get created, then the incentive to create AGI seizes and we effectively secure decades for AI-safety efforts.

This is a post I made on the subject.

https://www.lesswrong.com/posts/GzMteAGbf8h5oWkow/breaking-beliefs-about-saving-the-world [LW · GW]

Oh yeah I forgot about that, the bet is about the strategic implications of an AI market crash, not proving your opinion on AI economics.

Oops.

Hi Zvi, you misspelled my name as "Dei". This is a somewhat common error, which I usually don't bother to point out, but now think I should because it might affect LLMs' training data and hence their understanding of my views (e.g., when I ask AI to analyze something from Wei Dai's perspective). This search result contains a few other places where you've made the same misspelling.

spending tens of billions of dollars to build clusters that could train a GPT-6-sized model in 2028

Traditionally steps of GPT series are roughly 100x in raw compute (I'm not counting effective compute, since it's not relevant to cost of training). GPT-4 is 2e25 FLOPs. Which puts "GPT-6" at 2e29 FLOPs. To train a model in 2028, you would build an Nvidia Rubin Ultra NVL576 (Kyber) training system in 2027. Each rack holds 576 compute dies at about 3e15 BF16 FLOP/s per die^[1] or 1.6e18 FLOP/s per rack. A Blackwell NVL72 datacenter costs about $4M per rack to build, possibly a non-Ultra Rubin NVL144 datacenter will cost about $5M per rack, and a Rubin Ultra NVL576 datacenter might cost about $12M per rack^[2].

To get 2e29 BF16 FLOPs in 4 months at 40% utilization, you'd need 30K racks that would cost about $360B all-in (together with the rest of the training system). Which is significantly more than "tens of billions of dollars".

GPT-8 would require trillions

"GPT-8" is two steps of 100x in raw compute up from "GPT-6", at 2e33 FLOPs. You'd need to use 10000x more compute than what $360B buy in 2027. Divide it by how much cheaper that compute gets within a few years, let's say 8x cheaper. What we get is $450T, which is much more than merely "trillions", and also technologically impossible to produce at that time without transformative AI.