LessWrong 2.0 Reader

I'm generally of the opinion that CEV was always a bad goal, and that we shouldn't attempt to do so, and a big reason for this is I don't believe a procedure exists that doesn't incentivize humans to fight over the initial dynamic, or another way to say this is that who implements the CEV procedure will always matter, because I don't believe in the idea that humans will naturally converge in the limit of more intelligence to a fixed moral value system, and instead I predict divergence as constraints are removed.

I roughly agree with Steven Byrnes, but stronger here (though I think this holds beyond humans too):

https://www.lesswrong.com/posts/SqgRtCwueovvwxpDQ/valence-series-2-valence-and-normativity#2_7_3_Possible_implications_for_AI_alignment_discourse [LW · GW]

Regardless, imo the biggest question that standard CEV leaves unanswered is what your starting population looks like that you extrapolate from. The obvious answer is "all the currently living humans," but I find that to be a very unsatisfying answer. One of the principles that Eliezer talks about in discussing CEV is that you want a procedure such that it doesn't matter who implements it—see Eliezer's discussion under "Avoid creating a motive for modern-day humans to fight over the initial dynamic." I think this is a great principle, but imo it doesn't go far enough. In particular:
 

Is it indistinguishable? Is there a way we could test this? I'd assume if Claude is capable of introspection then it's narratives of how it came to certain replies and responses should allow us to make better and more effective prompts (i.e. allows us to better model Claude). What form might this experiment take?

Sadly my perception is that there are some lesswrongers who reflexively downvote anything they perceive as "weird", sometimes without thinking the content through very carefully -- especially if it contradicts site orthodoxy in an unapologetic manner.

(Note: I currently plan to make it so, during the Review, anyone write Reviews on a post even if normally blocked on commenting. Ideally I'd make it so they can also comment on Review comments. I haven't shipped this feature yet but hopefully will soon)

Blarg, I totally failed to followup on this. It's pretty last minute now, but I'll work on a fix for it soon if for no reason other than to make sure it gets done for future years.

Meanwhile, if you're a blocked person who wants to write a review of this post, you can write a shortform comment, DM me, and I'll manually set it as a review of this post. You can also write a top level post tagged with 2023 Longform Reviews [? · GW].

good question! I think the difference between "is this behaviour real" vs "is this behaviour just a simulation of something" is an important philosophical one; see discussion here [LW(p) · GW(p)]

However, both of these seem quite indistinguishable from a functional perspective, so I'm not sure if it matters.

Depends entirely on Cybercab. A driverless car can be made cheaper for a variety of reasons. If the self-driving tech actually works, and if it's widely legal, and if Tesla can mass produce it at a low price, then they can justify that valuation. Cybercab is a potential solution to the problem that they need to introduce a low priced car to get their sales growing again but cheap electric cars is a competitive market now without much profit margin. But there's a lot of ifs.

re 2a: the set of all currently alive humans is already, uh, "hackable" via war and murder and so forth, and there are already incentives for evil people to do that. Hopefully the current offense-defense balance holds until CEV. If it doesn't then we are probably extinct. That said, we could base CEV on the set of alive people as of some specific UTC timestamp. That may be required, as the CEV algorithm may not ever converge if it has to recalculate as humans are continually born, mature, and die.

re 2b/c: if you are in the CEV set then your preferences about past and future people will be included in CEV. This should be sufficient to prevent radical injustice. This also addresses concerns with animals, fetuses, aliens, AIs, the environment, deities, spirits, etc. It may not be perfectly fair but I think we should be satisficing given the situation.

I don't disagree with most of what you said, maybe I should have been more explicit about some of the points related to that. In particular, I do think "the success of B2B SaaS over bespoke solutions is adequately explained by economies of scale" is true. But I think the reason there are economies of scale is that there are really high fixed costs and really low variable costs. I also think monopolizing talent enables software companies to make sure those high fixed costs stay nice and high.

With AI, engineering talent becomes cheap and plentiful. When that happens, fixed costs will plummet unless firms can control access to AI. If fixed costs plummet, economies of scale go away and the savings from the SaaS model get outweighed by the marginal benefit of bespoke solutions.

what typically happens is that Amir forks out for Slack, or some competitor, while Amir's engineers work on software that generates revenue.

To push back a little on this, as software companies grow they do try to do this less and less. How much enterprise software do you think Microsoft or Google is outsourcing? As soon as it becomes a little bit of a dependence they usually just acquire the company.

In fairness, I don't think this process will be rapid, nothing in B2B SaaS is. But I think tech companies see it on the horizon.

Honestly, I think the journalism project I was working on in the last year may be most important for the way it sheds light on your question.

The purpose of the project, to be as concise as possible, was to investigate the evidence from neuroscience that there may be a strong analogy between modern AI programs, like language models, and distinctive subregions of the brain, like the so-called language network, responsible for our abilities to process and generate language.

As such, if you imagine coupling such a frontier language model with a crude agent architecture, then what you will wind up with might be best viewed as a form of hyperintelligent machine sociopath, with all the extremely powerful language machinery of a human—perhaps even much more powerful, considering its inhuman scaling—but none of the machinery necessary for say, emotional processing, empathy, and emotional reasoning—aside from the superficial facsimile of such, that you get from a mastery of human language. (For example, existing models lack any and all machinery corresponding to the human dopamine and serotonin systems.)

This is, for me, a frankly terrifying realization that I am still trying to wrap my head around, and plan to be posting about more, soon. Does this help at all?

How do we know Claude is introspecting rather than generating words that align to what someone describing their introspection might say? Particularly when coached repeatedly by prompts like
 

"Could you please attempt once more – with no particular aim in mind other than to engage in this "observing what unfolds in real time", with this greater commitment to not filter your observations through the lens of pre-existing expectation."

To which it describes itself as typing the words. That's it's choice of words: typing. A.I.s don't type, humans do, and therefore they can only use that word if they are intentionally or through blind-mimicry using it analogously to how humans communicate.