LessWrong 2.0 Reader

I'm not gonna read the reddit post because

• it's an eyebleed wall of text,
• the author spent hours being excited about this stuff without bothering to learn that we have ~20 billion cortical neurons, not 20 trillion,
• yeah.

I don't know whether orcas are supersmart. A couple remarks:

• I don't think it makes that much sense to just look at cortical neuron counts. Big bodies ask for many neurons, including cortical motor neurons. Do cetaceans have really big motor cortices? Visual cortices? Olfactory bulbs? Keyword "allometry". Yes, brains are plastic, but that doesn't mean orcas are actually ever doing higher mathematics with their brains.
• Scale matters, but I doubt it's very close to being the only thing! Humans likely had genetic adaptations for neuroanatomical phenotypes selected-for by some of: language; tool-making; persisting transient mental content; intent-inference; intent-sharing; mental simulation; prey prediction; deception; social learning; teaching; niche construction/expansion/migration. Orcas have a few of these. But how many, how much, for how long, in what range of situations and manifestations? Or do you think a cow brain scaled to 40 billion neurons would be superhuman?
• Culture matters. The Greeks could be great philosophers... But could a kid living in 8000 BCE, who gets to text message with an advanced alien civilization of kinda dumb people, become a cutting edge philosopher in the alien culture? Even though almost everyone ze interacts with is preagricultural, preliterate? I dunno, maybe? Still seems kinda hard actually?
• Regardless of all this, talking to orcas would be super cool, go for it lol.
• Superbabies is good. It would actually work. It's not actually that hard. There's lots of investment already in component science/tech. Orcas doesn't scale. No one cares about orcas. There's not hundreds of scientists and hundreds of millions in orca communications research. Etc. The sense of this plan being weird is a good sense to investigate further. It's possible for superficial weirdness to be wrong, but don't dismiss the weirdness out of hand.

It seems unlikely to me that there is potential to make large brain based intelligence advancements beyond the current best humans using human evolved biology.  There will be distance scaling limitations linked to neural signal speeds.

Then there is Jeff Hawkins 'thousand brains' theory of human intelligence that our brains are made up of thousands of parallel processing cortical columns of a few mm cross section and a few mm thick with cross communication and recursion etc, but that fundamental processing core probably isn't scalable in complexity, only in total number - your brain could perhaps be expanded to handle thinking about more things in parallel at once, but not at much higher levels of sophistication without paying a large coordination speed price (and evolution places a premium on reaction speed for animals that encounter violence)

I look at whales and other mammals with much much larger than human brains and wonder why they are not smarter - some combination of no evolutionary driver and perhaps a lot of their neurons are dedicated to delay-line processing needed for processing sonar and controlling large bodies with long signaling delays.

Regardless, if AI is a dominant part of our future then it seems likely to me that regardless of whether the future is human utopia or dystopia, non-transhuman humans will not exist in significant numbers in a few hundred years. Neural biology and perhaps all biology is going to be superseded as maladapted to the technological future. 

The transmission utility is not purely a transmission company. It spends money on both generation and transmission. Some generation charges leave to other companies. This is not a competitive market, but even if it were, it would only give you a bound on the cost of generation and tell you nothing about the cost of transmission.

I appreciate you being relatively clear about this, but yeah, I think it's probably better to spend more time learning facts and thinking stuff through, compared to writing breathless LW posts. More like a couple weeks rather than a couple days. But that's just my stupid opinion. The thing is, there's probably gonna be like ten other posts in the reference class of this post, and they just... don't leave much of a dent in things? There's a lot that needs serious thinking-through, let's get to work on that! But IDK, maybe someone will be inspired by this post to think through orca stuff more thoroughly.

True; and they would only need to merge up to they reach a "swing state" type of voting distribution.

IIUC, I agree with your vision being desirable. (And, IDK, it's sort of plausible that you can basically do it with a good toolbox that could be developed straightforwardly-ish.)

But there might be a gnarly, fundamental-ish "levers problem" here:

• It's often hard to do [the sort of empathy whereby you see into your blindspot that they can see]
• without also doing [the sort of empathy that leads to you adopting some of their values, or even blindspots].

(A levers problem is analogous to a buckets problem, but with actions instead of beliefs. You have an available action VW which does both V and W, but you don't have V and W available as separate actions. V seems good to do and W seems bad to do, so you're conflicted, aahh.)

I would guess that what we call empathy isn't exactly well-described as "a mental motion whereby one tracks and/or mirrors the emotions and belief-perspective of another". The primordial thing--the thing that comes first evolutionarily and developmentally, and that is simpler--is more like "a mental motion whereby one adopts whatever aspects of another's mind are available for adoption". Think of all the mysterious bonding that happens when people hang out, and copying mannerisms, and getting a shoulder-person, and gaining loyalty. This is also far from exactly right. Obviously you don't just copy everything, it matters what you pay attention to and care about, and there's probably more prior structure, e.g. an emphasis on copying aspects that are important for coordinating / synching up values. IDK the real shape of primordial empathy.

But my point is just: Maybe, if you deeply empathize with someone, then by default, you'll also adopt value-laden mental stances from them. If you're in a conflict with someone, adopting value-laden mental stances from them feels and/or is dangerous.

To say it another way, you want to entertain propositions from another person. But your brain doesn't neatly separate propositions from values and plans. So entertaining a proposition is also sort of questioning your plans, which bleeds into changing your values. Empathy good enough to show you blindspots involves entertaining propositions that you care about and that you disagree with.

Or anyway, this was my experience of things, back when I tried stuff like this.

Are any of the socio-economic-political-demographic problems of the world actually fixable or improvable in the time before the imminent singularity renders them all moot anyway?  It all feels like bread-and-circuses to me.

The pressing political issues of today are unlikely to even be in the top-10 in a decade.

"Investment is essentially a game of compounding. The first rule is to avoid losses. Identify the world’s top four high-quality assets and diversify your investments among them. Exercise patience and wait for opportunities. Buy during times of market panic and downturns, then patiently wait. Sell when the market reaches a phase of extreme exuberance and bubbles. Longevity is key, as time is the fuel for the compounding game."

This pattern is one I have observed as well, in various disagreements but especially in political ones.  For the past few years I've been working on methods for dissolving these scissors statements.  With some foundational concepts to peel away assumptions and take the conversation down to basics, a person can do it systematically with relative ease.  

Points of disagreement take the form of tradeoffs, which people either accept or reject.  These tradeoffs can be described in terms of costs, risks, habits, and trust.  People accept or reject a tradeoff based on what their experiences tell them about the tradeoff's drawbacks, as well as how well they are positioned to deal with those drawbacks.  

As you pointed out, people often downplay or ignore the drawbacks of their own tradeoff choices, while believing that people who make the opposite choice with different drawbacks are selfish, or even have that choice's drawbacks as a terminal value instead of an unfortunate side-effect.  

For example: 

"My environmental policy won't affect the economy that much.  Our opponents care more about consumerism than about the future of the planet."  

"My economic policy won't affect the environment that much.  Our opponents care more about feelings superior than about the people they'd be putting out of work."  

Listening to and acknowledging the real reasons people accept or reject a tradeoff makes them feel understood and respected, at which point they are more comfortable listening to reasons why other people might disagree.  Understanding the tradeoff concepts makes it much easier to understand and explain the reasoning behind tradeoffs.  

Once we understand the tradeoffs in play and the drawbacks people fear, we can resolve these points of disagreement using constructive principles.  (This part requires zooming out of the narrow focus on the situation and applying problem-solving mindsets.  Most humans get stuck arguing about zero-sum tradeoffs because humans don't yet teach constructive problem-solving as part of a standard educational curriculum.)  

Constructive principles improve the situation over time to put everyone in better positions with more options, lowering the stakes for disagreement about what to do in the short term.  Investment deals with costs.  Preparation deals with risks.  Challenge deals with habits.  Ethics deals with trust.  These principles are how we explore approaches for simultaneously addressing all the drawbacks that all the stakeholders are concerned about.  

How does that sound?  If you'd like to learn more about the process for dissolving scissors statements and reconciling conflict constructively, just let me know.  We can apply it to conflict situations that you want to resolve.  

I'm glad to see this. Some initial thoughts about the control safety case:

• I don't get how the "monitoring game" is different from the red-team game? Is it because the blue team is limited to only considering protocols that use just monitoring? Or do you not expect to play a full red-team simulation (this would be worrisome)
• Option 2 (honeypotting) feels doomed (in isolation). This primarily stems from my belief that making situations that are indistinguishable from deployment is probably hard - the model has control over state in very broad and hard-to-predict ways, and this can implicitly encode that it is in deployment (for instance, the model can see code that it itself has written and would've taken a long time to write, and realize it has likely been deployed for some time). You can still run honeypotting tests using this newer distribution, but then you have to hope that you are running honeypotting frequently enough to catch when the model is confident it is in deployment.
• The elicitation problem, and the broader problem of justifying that the red-blue teaming has been sufficient, indeed feel quite hard. This is one reason as to why you might expect that control cases can only work for human-level AI - if your 'elicitation protocol' just ends up being 'use attack strategies made primarily by humans'.
  • There are related problems in justifying how many affordances to give to the red team. For instance, decisions related to if the red team should be able to give/train into the model 'coordination signals' are likely important [AF · GW], but might not be easy to make. Past research has been maximally conservative, but this might not be realistic.
• The limitation that you are measuring P(catastrophe | misaligned) rather than just P(catastrophe) is fairly large, and I would be interested in discussion on how important this is to others. More broadly, I don't feel great about our chances of bounding P(catastrophe) for reasons such as this.