LessWrong 2.0 Reader

This is a good point. I don't think it should make that much of a difference given how young LessWrong is on average, but it can't hurt to try. 

My two problem are 1) finding SAT statistics for nationally representative samples, and not just seniors that take the SAT (the latter are obviously selected) is difficult, and 2) I’d need more detailed data than just the SAT averages—I'd have to adjust each person’s SAT z-score based on the year they took the test.

By this logic, if rationalists are selected based on IQ and not height, and the average rationalist height is +1.85 SD, then we’d have to assume that rationalists’ IQ is +9.25 SD (assuming an IQ-height correlation of 0.2), which is, of course, impossible.

For another example of why this logic doesn’t work, consider this: if you have a variable that is uncorrelated with IQ (r = 0), and rationalists are just slightly above average for that variable, then we'd be forced to conclude that rationalists are infinitely smart (or, if they're below average, infinitely dumb) depending on the direction of the deviation. This is clearly nonsensical.

For an explanation of why this logic doesn't work, see my reply to Unnamed's comment. And for the correct calculations, see my reply to faul_sname's comment.

If someone is wondering what prefilling means here, I believe Ted means ‘putting words in the model’s mouth’ by being able to fabricate a conversational history where the AI appears to have said things it didn’t actually say.

For instance, if you can start a conversation midway, and if the API can’t distinguish between things the model actually said in the history vs. things you’ve written in its behalf as supposed outputs in a fabricated history, this can be a jailbreak vector: If the model appeared to already violate some policy on turns 1 and 2, it is more likely to also violate this on turn 3, whereas it might have refused if not for the apparent prior violations.

(This was harder to clearly describe than I expected.)

Your argument assumes a uniform prior, but a Gaussian prior is more realistic in this case. In practice, IQ scores are distributed normally, so it's more likely that someone with a high SAT score comes from a more common IQ range than from a very high outlier. For example, say the median rationalist has an SAT score of +2 SD (chosen for ease of computation), and the SAT-IQ correlation is 0.80. The IQ most likely to produce an SAT of +2 SD is 137.5 (+2.5 SD). However, IQs of 137.5 are rare (99.4%-ile). While lower IQs are less likely to achieve such a high SAT score, there are more people in the lower IQ ranges, making it more probable that someone with a +2 SD SAT score falls into a lower IQ bracket.

This shift between the MLE (Maximum Likelihood Estimate) and MAP (Maximum A Posteriori) estimates is illustrated in the graph, where the MLE estimate would be +2.5 SD, but the MAP estimate, accounting for the Gaussian prior, is closer to +1.6 SD, as expected. (You may also be interested in my reply to faul_sname [LW · GW]'s comment.)

The below captures some of my thoughts on the "Jobs hits first" scenario.

There is probably something with the sort of break down I have in mind (though, whatever this may be, I have not encountered it, yet) w.r.t. Jobs hits first but here goes: the landscape of AI induced unemployment seems highly heterogeneous, at least in expectation over the next 5 years. For some jobs, it seems likely that there will be instances of (1) partial automation, which could mean either not all workers are no longer needed (even if their new tasks no longer fully resemble their previous tasks), i.e. repurposing of existing workers, or most workers remain employed but do more labor with help from AI and of (2) prevalent usage of AI across the occupational landscape but without much unemployment, with human work (same roles pre-automation) still being sought after, even if there are higher relative costs associated with the human employment (basically, the situation: human work + AI work > AI work, after considering all costs). Updating based on this more nuanced (but not critically evaluated) situation w.r.t. Jobs hits first, I would not expect the demand for a halt to job automation to be any less but perhaps a protracted struggle over what jobs get automated might be less coordinated if there are swaths of the working population still holding out career-hope, on the basis that they have not had their career fully stripped away, having possibly instead been repurposed or compensated less conditional on the automation.

The phrasing

...a protracted struggle...

nevertheless seems a fitting description for the near-term future of employment as it pertains to the incorporation of AI into work.

You're being downvoted and nobody's telling you why :-(, so I thought I'd give some notes.

• You're not talking to the right audience.  Few groups are more emotionally in favor of a glorious transhumanist future than people on LessWrong.  This is not technophobes who are afraid of change.  It's technophiles who have realized, in harsh constrat to the conclusion they emotionally want, that making a powerful AI would likely be bad for humanity.
• Yes, it's important to overly anthropomorphize AIs, and you are doing that all over the place in your argument.
• These arguments have been rehashed a lot.  It's fine to argue that the LessWrong consensus opinion is wrong, but you should indicate you're familiar with why the LessWrong consensus opinion is what it is.

(To think about what it might not settle on a cooperative post-enlightenment philosophy, read, I don't know, correct heaps [LW · GW]?)

Eric Neyman is right. They are both valid! 

In general, if we have two vectors $X$ and $Y$ which are jointly normally distributed, we can write the joint mean $\mu$ and the joint covariance matrix $\Sigma$ as

The conditional distribution for $Y$ given $X$ is given by $Y|X\sim N\left({\mu }_{Y|X},K_{Y|X}\right)$,
defined by conditional mean 

${\mu }_{Y|X}={\mu }_Y+K_{YX}{K_{XX}}^{-1}(X-{\mu }_X)$

and conditional variance 

$K_{Y|X}=K_{YY}{K_{XX}}^{-1}{K}_{XY}$

Our conditional distribution for the IQ of the median rationalist, given their SAT score is $N(0+(0.8\cdot 1\cdot (2.42-0)),1-(0.8\ast 1\ast 0.8\left)\right)=N(1.94,0.36)$ 
(That's a mean of 129 and a standard deviation of 9 IQ points.)

Our conditional distribution for the IQ of the median rationalist, given their height is $N(0+(0.2\ast 1\ast (1.85-0)),1-(0.2\ast 1\ast 0.2\left)\right)=N(0.37,0.96)$
(That's a mean of 106 and a standard deviation of 14.7 IQ points.)

Our conditional distribution for the IQ of the median rationalist, given their SAT score and height is $N(0+\left(\left[\begin{array}{cc}0.8& 0.2\end{array}\right]{\left[\begin{array}{cc}1& 0.16\\ 0.16& 1\end{array}\right]}^{-1}\left[\begin{array}{c}2.42\\ 1.85\end{array}\right]\right),1-\left(\left[\begin{array}{cc}0.8& 0.2\end{array}\right]{\left[\begin{array}{cc}1& 0.16\\ 0.16& 1\end{array}\right]}^{-1}\left[\begin{array}{c}0.8\\ 0.2\end{array}\right]\right))=N(2.04,0.35)$(That's a mean of 131 and a standard deviation of 8.9 IQ points)

Unfortunately, since men are taller than women, and rationalists are mostly male, we can't use the height as-is when estimating the IQ of the median rationalist (maybe normalizing height within each sex would work?). 

Capitalism is a force that has lifted billions out of poverty, where even poor remote villagers enjoy luxuries that would have been unimaginable to medieval kings. When someone takes a job, even the worst job, it’s because both parties expect mutual gain.

We often treat capitalism, freedom of labor, and free trade as inevitably going along with one another, but I'm not sure that's correct. In this post you're mostly talking about labor, not capital or trade. It may be worth looking into ways that these don't always go together.

(By "capitalism" I mean the allocation of capital to firms through markets. By "freedom of labor" I mean the allocation of jobs through voluntary arrangements; as opposed to jobs being assigned through slavery, government command, family trades, caste, etc.)

For example: Some of the historically earliest firms to be traded in capital markets — colonial ventures like the East India Companies, Hudson's Bay Company, etc. — employed slave labor and enjoyed legal monopolies on trade. So in those cases there was capitalism without either freedom of labor (for some, anyway) or freedom of trade.

Another example: Free-trade agreements such as NAFTA enable the free movement of goods and capital across borders, but not the free movement of labor. In contrast, political unions such as the European Union (or the US itself) enable the movement of labor as well as goods and capital among member states. So, similar policies on goods and capital can coexist with dissimilar policies on labor.

On our planet, free trade is only really possible with some military might defending it, because otherwise we get pirates or bandits taxing our trade and then it's not free anymore. Similarly, we can only ensure freedom of labor by actually enforcing rules against employers who might otherwise use unfree labor.

There are gradations of unfreedom of labor, too. If you can't leave a bad job because you don't have savings or credit to tide you over till you find a new job, that's a little bit of labor unfreedom. It makes the job market less efficient, because you can't switch jobs to one that's a better fit due to not being able to pay the search cost. If you can't take a day off from your current job to interview for a new job, that's a little bit of labor unfreedom. If the jobs available to you are strictly limited by your race, caste, or nationality, that's quite a bit of labor unfreedom. If the only way to get a job that you otherwise fit, is to tolerate violations of your human rights by your employer (like sexual assault), that's pretty significant labor unfreedom. If you are shackled in a basement and beaten for not peeling potatoes fast enough, that's a lot of labor unfreedom.

One kind of thing we might want to describe as "exploitation" is using someone's current form and degree of unfreedom to reduce their freedom even further; to deepen and maintain an asymmetry where you get more power and they get less and less. Making it hard for someone to seek a new job, in order to keep them in a job as it gets worse. Punishing workers for "disloyalty" if they try to negotiate for a raise or organize for collective bargaining. Blackballing through employer collusion ("you'll never work in this town again!"). Debt-slavery.

You might enjoy this story I wrote a few months ago, also about AI doom and also set in the future: https://www.lesswrong.com/posts/BgTsxMq5bgzKTLsLA/this-is-already-your-second-chance [LW · GW]

I don't think the intelligence consistently leads to self-annihilation hypothesis is possible. At least a few times it would amount to robust self-preservation.

Well.. I guess I think it boils down to the dark forest hypothesis. The question is whether your volume of space is likely to contain a certain number of berserkers, and the number wouldn't have to be large for them to suppress the whole thing.

I've always felt the logic of berserker extortion doesn't work, but occasionally you'd get a species that just earnestly wants the forest to be dark and isn't very troubled by their own extinction, no extortion logic required. This would be extremely rare, but the question is, how rare.