LessWrong 2.0 Reader

So... yeah, technically these things are connected, but sometimes the connection is real and strong, and sometimes it's just "both of them sound kinda sci-fi to me". 

As far as I remember (may be misremembering), to the extent that their memetic genealogy analysis holds water, there was little bit more common origin than I expected.

But ofc I agree that the entire thing is just an ideological carpet bombing via guilt by (often overblown) association.

What is the "industrial & philanthropic complex"? Doesn't it include... pretty much everything? I mean, most things are produced by industry, and most non-profits accept donations. Should we consider Bill Gates a part of TESCREAL? He had a software company, and he donated to charities.

I didn't mean all of industry or all of philanthropy. I was pointing at their perception that there is a cluster of futuristic privileged dudes, fractions constantly bickering, and having some views on AGI, industry-ing and philanthropy-ing.

(Possibly first time in my life I feel like I'm overcharitably steelmanning Torres & Gebru on TESCREAL. Is this even worse than I'm giving them credit for?)

People who use the term TESCREAL generally don't realize that science fiction authors often take the futures they write about seriously (if not literally). They will talk about "TESCREALists taking sci-fi books too seriously" without knowing Marvin Minsky, the AI pioneer whose "AI tasked to solve the Riemann hypothesis" thought experiment is effectively the origin of the paperclip-minimizer thought experiment, was the technical consultant for 2001: A Space Odyssey and was considered by Isaac Asimov to be one of the two smartest people he ever met (alongside cosmist Carl Sagan).

I think you're wrong to be psychoanalysing why people aren't paying attention to your work. You're overcomplicating it. Most people just think you're wrong upon hearing a short summary, and don't trust you enough to spend time learning the details. Whether your scenario is important or not, from your perspective it'll usually look like people are bouncing off for bad reasons.

For example, I read the executive summary. For several shallow reasons^[1], the scenario seemed unlikely and unimportant. I didn't expect there to be better arguments further on. So I stopped. Other people have different world models and will bounce off for different reasons.

Which isn't to say it's wrong (that's just my current weakly held guess). My point is just that even if you're correct, the way it looks a priori to most world views is sufficient to explain why people are bouncing off it and not engaging properly.

Perhaps I'll encounter information in the future that indicates my bouncing off was a mistake, and I'll go back.

1. ^{^}

   There are a couple of layers of maybes, so the scenario doesn't seem likely. I expect power to be more concentrated. I expect takeoff to be faster. I expect capabilities to have a high cap. I expect alignment to be hard for any goal. Something about maintaining a similar societal structure without various chaotic game-board-flips seems unlikely. The goals-instilled-in-our-replacements are pretty specific (institution-aligned), and pretty obviously misaligned from overall human flourishing. Sure humans are usually myopic, but we do sometimes consider the consequences and act against local incentives. 

   I don't know whether these reasons are correct, or how well you've argued against them. They're weakly held and weakly considered, so I wouldn't have usually written them down. They are just here to make my point more concrete.

There's a lot more complexity, obviously, but one thing that sticks out to me is this paragraph, from https://sinceriously.blog-mirror.com/net-negative/ :

I described how I felt like I was the only one with my values in a world of flesh eating monsters, how it was horrifying seeing the amoral bullet biting consistency of the rationality community, where people said it was okay to eat human babies as long as they weren’t someone else’s property if I compared animals to babies. How I was constantly afraid that their values would leak into me and my resolve would weaken and no one would be judging futures according to sentient beings in general. How it was scary Eliezer Yudkowsky seemed to use “sentient” to mean “sapient”. How I was constantly afraid if I let my brain categorize them as my “in-group” then I’d lose my values.

This is one among several top hypothesis-parts for something at the core of how Ziz, and by influence other Zizians, gets so far gone from normal structures of relating. It is indeed true that normal people (I mean, including the vast majority of rationalists) live deep in an ocean of {algorithm, stance, world, god}-sharing with people around them. And it's true that this can infect you in various ways, erode structures in you, erode values. So you can see how someone might think it's a good idea to become oppositional to many normal structures of relating; and how that can be in a reinforcing feedback loop with other people's reactions to your oppositionality and rejection.

(As an example of another top hypothesis-part: I suspect Ziz views betrayal of values (the blackmail payout thing) and betrayal of trust (the behavior of Person A) sort of as described here: https://sideways-view.com/2016/11/14/integrity-for-consequentialists/ In other words, if someone's behavior is almost entirely good, but then in some subtle sneaky ways or high-stakes ways bad, that's an extreme mark against them. Many would agree with the high-stakes part of (my imagined) Ziz's stance here, but many fewer would agree so strongly with the subtle sneaky part.)

If that's a big part of what was going on, it poses a general question (which is partly a question of community behavior and partly a question of mental technology for individuals): How to make it more feasible to get the goods of being in a community, without the bads of value erosion?

Like, what the fuck is "cosmism"? It doesn't even have a proper Wikipedia page

It does: https://en.wikipedia.org/wiki/Russian_cosmism 

Minimising some term like $\frac{\sigma \left(\delta \right(h^{\prime }\left)\right)}{\text{E}\left(\delta \right(h^{\prime }\left)\right)}$, with $\delta \left(h^{\prime }\right(t\left)\right):=\left|\right|h_t^{\prime }-h_{t+1}^{\prime }|{|}_2^2$, where the standard deviation $\sigma \left(\delta \right(h^{\prime }\left)\right)$ and expectation $\text{E}\left(\delta \right(h^{\prime }\left)\right)$ are taken over the batch.

Why does this make ${\delta }_t:=\left|\right|h_t-h_{t+1}|{|}_2^2$ tend to be small? Wouldn't it just encourage equally-sized jumps, without any regard for the size of those jumps?

To me this seems like a motte-and-bailey situation, where the motte is making true statements about various connections, and the bailey is making a conspiracy theory out of it.

A similar situation in politics is e.g. pointing out that something is "connected to Soros". Sure, millions of things are: Soros is a billionaire who gives tons of money to lots of organizations and charities; many of them distribute the money further... Like, maybe there is a playground near your house, which was built using money from some city fund, which at some moment has received money from some other fund, which has received money from Soros. So I could make a technically true point that the playground next to your house was built using money from Soros. OK, what does that imply? Rationally speaking, it just says that Soros spends money on lots of various things, some of them trickle down to some local playgrounds; that's it.

But a person who makes such statement on internet is typically using it to insinuate something. The full tweet will be like: "Yeah, of course Mateusz says X and opposes Y, that's no surprise for the people who actually know something about him. Even the playground next to his house was built using money from Soros!" And, you know, that kinda makes you an important part of the New World Order, and of course you are "skeptical" about chemtrails, that's what all people on his payroll would say.

If you know about the "six degrees of separation", that basically just means that in (some kind of) network, the longer connections you make, the exponentially more nodes you can reach, so at a surprisingly small number of steps you can reach practically everyone. (Maybe the actual number is greater than six, depending on how we define a "friend". That means that any person, including you or me, is probably within six steps distance from Donald Trump, or Vladimir Putin, or whoever else. If you are directly connected to some international network (such as Less Wrong), the number is probably much lower. There are some Less Wrong readers in Russia, some people in their families very likely work for the Russian government, add two more steps, and you are four degrees of separation away from Putin. What does that mean? Nothing specific. Until one day a journalist decides that he doesn't like your face for some reason, and he will describe this four-step chain, making you responsible for the latest atrocity in Ukraine in the mainstream narrative. "Mateusz, an online pal of Vanya, a nephew of Ivan the officer of FSB" -- if you put it this way, is a typical reader going to believe that this is all just a coincidence and it means nothing?

And in my opinion, TESCREAL is a similar thing. Yes, some parts are connected to others, but the entire thing is blown out of proportions. Like, what the fuck is "cosmism"? It doesn't even have a proper Wikipedia page, and I have never heard about it before Émile P. Torres [EA · GW] decided that it is an important component of... the thing that I am apparently a part of. I can't even make a coherent objection, because I have no idea what "cosmism" is, but... let's assume that it is a group of people who have read sci-fi books, and who assume that in future we will fly to stars, have intelligent robots, and that will create many ethical and economical dilemmas; or something like that. What does that mean? Nothing. But let's go further and assume that dozen "cosmists" actually post on Less Wrong (but for some reason have never mentioned "cosmism" in their posts). I just made that up -- but let's assume that it is true, just to steelman Torres. What would that mean? In my opinion, still nothing. In the world with internet, it makes sense that some of the people who like sci-fi would also find a website about artificial intelligence interesting.

Internet made the world feel really small. Or rather, organized by intellectual interests rather than geographical distances. There are people on my street that I don't know. There are people on the opposite side of the planet that I do -- because we share a hobby. One day, my wife's mother mentioned that when she listened to some Slovak radio, they were also talking about Scott Alexander. Another day, I found in Slate Star Codex article a reference on intelligence research done by my boss. And yet, my wife's mother or my former boss could in no reasonable way be considered members of the "TESCREAL"... unless the category becomes so wide that it practically includes all smart educated people in the Western civilization.

Of course, some connections in TESCREAL make a lot of sense. Like, yes, there is a connection between rationalists and effective altruists. You could hardly deny it, when their web forums literally run on two instances of the same software, administered by the same people! (Now that the SBF scandal put EA in a bad light, even Wikipedia freely admits that EA and R actually might have something in common.) But from my perspective, this makes the acronym even less valid, because it kinda suggests that the link between EA and R is comparable to the link between them and C, which to me just sounds absurd. But of course, if we just renamed it to REA, it wouldn't be so controversial.

So... yeah, technically these things are connected, but sometimes the connection is real and strong, and sometimes it's just "both of them sound kinda sci-fi to me". Also, the world is small. What is the "industrial & philanthropic complex"? Doesn't it include... pretty much everything? I mean, most things are produced by industry, and most non-profits accept donations. Should we consider Bill Gates a part of TESCREAL? He had a software company, and he donated to charities.

We could narrow it down to something like "people influenced by writings of Eliezer Yudkowsky and Scott Alexander". But even this is less narrow than in seems -- there are thousands, maybe even millions of people who read Scott, or at least are influenced by his writing through friends (and ironically it probably excludes the "cosmists", maybe, because I still don't know who they are). So it is wrong to insinuate that this is some small sinister group, when it probably includes half of the Silicon Valley.

What else is there? Historical links to eugenics? Really; who doesn't have them? People seem to forget that eugenics used to be very popular among progressives before WW2 associated it with Nazis. Threats to environment? There are millions of things outside tech that threaten environment, too. Some people are sexist and racist? If that is supposed to mean anything, one would need to prove that this exceeds the base rates in general population. (Otherwise it reduces to: another link connecting these groups is that they... dramatic music... are made of humans.)

The language of the Wikipedia article also annoys me: "Elon Musk has been described as sympathetic to some TESCREAL ideologies", "It has also been suggested that Peter Thiel is sympathetic to TESCREAL ideas", "Sam Altman and much of the OpenAI board has been described as supporting TESCREAL movements", " Nick Bostrom and Eliezer Yudkowsky [...] have also been described as leaders of the TESCREAL movement", "William MacAskill [...] has been described as a TESCREAList". Does it mean that any of these people actually uses the acronym, or even considers it meaningful? I mean, wouldn't it be funny if the supposed leaders of a movement didn't believe that it exists?

I'll conjecture the following in a VERY SPECULATIVE, inflammatory, riff-on-vibes statements:

• Gradient descent solves problem in the complexity class P^[1].  It is P-Complete.
• Learning theory (and complexity theory) have for decades been pushing two analogous bad narratives about the weakness of gradient descent (and P).
• These narratives dominate because it is easy prove impossibility results like "Problem X can't be solved by gradient descent" (or "Problem Y is NP-Hard").  It's academically fecund -- it's a subject aspiring academics can write a lot of papers about.  Results about what gradient descent (and polynomial time) can't do compose a fair portion of the academic canon
• In practice, these impossible results are corner cases cases don't actually come up.  The "vibes" of these impossibility results run counter to the "vibes" of reality
  • Example, gradient descent solves most problems, even though it theoretically it gets trapped in local minima.  (SAT is in practice fast to solve, even though in theory it's theoretical computer science's canonical Hard-Problem-You-Impossible-To-Solve-Quickly)
  • The vibe of reality is "local (greedy) algorithms usually work"

1. ^{^}

   Stoner-vibes based reason: I'm guessing you can reduce something like Horn Satisfiability^[2] to gradient descent.  My mental visualization for Singular Learning Theory is reminiscent to my mental visualization of the algorithm for Horn Satisfiability [EDIT less so after thinking about it for a minute but maybe].  Horn Satisfiability is a P-compete problem -- you can transform any polynomial-time decision problem in a Horn Satisfiability problem using a log-space transformation.  Therefore, gradient descent is "at least as big as P" (P-hard).  And I'm guessing you can your formalization of gradient descent in P as well (hence "P-Complete").

2. ^{^}

   Horn Satisfiability is about finding true/false values that satisfy a bunch of logic clauses of the form $a\wedge c\wedge d\to e$. or  $b\wedge e\to \perp$ (that second clause means "don't set both $b$ and $e$ to true -- at least one of them has to be false" ).  In the algorithm for solving it, you figure out a variable that must be set to true or false, then propagate that information forward to other clauses.  I bet you can do this with a loss function turning into a greedy search on a hypercube.

Or is this all just bad luck... that if you make a workshop, and a future murderer decides to go there, and they decide to use some of your keywords in their later manifesto... then it doesn't really matter what you do, even if you tell them to fuck off and call cops on them, you will forever be connected to them, and it's up to journalists whether they decide to spin it as: the murderer is just an example of everything that is wrong with this community.

I think this is a strange description of the mainstream media coverage when most of the articles talking about Zizian ideology are nearly entirely sourced from interviews with rationalists and talk about their conflict with mainline rationalists at length.

The lesson I can glean is probably that considering the high rate of cult creation in the community and the flashing warning signs about them, rationalists should have been far more proactive in adopting normal procedures of cult and terrorism prevention (e.g. surveilling their activities, preventing the isolation of potential recruits, and looking for any suspiciously missing person).

But isn't there almost always a possibility of a entity goodharting to change it's definition  of what consitutes a paperclip that is easier for it to maximize? How does it internally represent what is a paperclip? How broad is that definition? What power does it have over it's own "thinking" (sorry to anthropamorphize) does it have to change how it represents the things which that representation relies on?

Why is it most likely that it will have an immutable, unchanging, and unhackable terminal goal? What assumptions underpin that as more likely than fluid or even conflicting terminal goals which may cause radical self-modifications?

A terminal goal is a case of criteria according to which actions are chosen; "self-modify to change my terminal goal" is an action.

What does "a case of criteria" mean?