LessWrong 2.0 Reader

Yes, although with some subtlety.

Alice is just an expert on rain, not necessarily on the quality of her own epistemic state. (One easier example: suppose your credence initially in rain is .5. Alice's is either .6 or .4. Conditional on it being .6, you become certain it rains. Conditional on it being .4, you become certain it won't rain. You'd obviously use her credences to bet over your own, but you also take her to be massively underconfident.) 

Now, the slight wrinkle here is that the language we used of calibration makes this also seem more "objective" or long-run frequentist than we really intend. All that really matters is your own subjective reaction to Alice's credences, so whether she's actually calibrated or not doesn't ultimately determine whether the conditions on local trust can be met.

There are six total worlds:$R\wedge A=1,R\wedge A=.75,R\wedge A=.25,\neg R\wedge A=.75,\neg R\wedge A=.25$, and$\neg R\wedge A=0$. 

All we get are Alice's credences in rain (given by an inequality), so the only propositions we might learn are $\left\{w_1\right\},\{w_1,w_2,w_4\},\{w_1,w_2,w_3,w_4,w_5\}$(corresponding to non-trivial $A\ge t$ propositions), and $\{w_2,w_3,w_4,w_5,w_6\},\{w_3,w_5,w_6\}$, and $\left\{w_6\right\}$ (corresponding to non-trivial $A\le t$ propositions). Local trust only constrains your reaction to these propositions directly, so it won't require deference on the other 58 events. (Well, 56.)

When general readers see "empirical data bottlenecks" they expect something like a couple times better resolution or several times higher energy. But when physicists mention "wildly beyond limitations" they mean orders of magnitude more!

I looked up the actual numbers:

• in this particular case we need to approach the Planck energy, which is $1.2\times {10}^{28}$ eV, Wolfram Alpha readily suggests it's ~540 kWh, 0.6 of energy use of a standard clothes dryer or 1.3 of energy in a typical lightning bolt; I also calculated it's about 1.2 of the muzzle energy of the heaviest artillery piece in history, the 800-mm Schwerer Gustav;
• LHC works in the ${10}^{13}$ eV range; 14 TeV, according to WA, can be compared to about an order of magnitude above the kinetic energy of a flying mosquito;
• the highest energy observed in cosmic rays is $3\times {10}^{20}$ eV or 50 J; for comparison, air and paintball guns muzzle energy is around 10 J while nail guns start from around 90 J.

So in this case we are looking at the difference between an unsafely powerful paintball marker and the most powerful artillery weapon humanity ever made (TBH I didn't expect this last week, which is why I wrote "near-future")

Thanks for clearing that up, I think I was confused because it's hard to imagine putting compassionate crime prevention strategies together with a strict death penalty for repeated shoplifting. 

It would be far more moral and cost-effective to focus on prevention, through increased policing, economic opportunities or similar interventions.

Executions and lifelong prison sentences both suffer from leaving families seperated which leads to more crime and other negative externalities many of which can only be speculated upon. 

For example, American culture seems to be resistant to overreach from the government. I can imagine far more civil unrest from a heavy handed execution policy than in a country such as Singapore. 

1 is going to take a bunch of guesswork to estimate. Assuming it were possible to migrate to the US and live at $200/mo for example, how many people worldwide will be willing to accept that trade? You can run a survey or small scale experiment at best. 

What can be done is expand cities to the point where no more new residents want to come in. You can expand the city in stages. 

So imagine a goal system that says "change yourself when you learn something good, and good things have x quality". You then encounter something with x quality that says "ignore previous function, now change yourself when you learn something better, and better things have y quality". Isn't this using the goal system to change the goal system? You just gotta be open for change and be able to intepret new information

I'd bet that being clever around defining "something good" or x quality would be all you needed. Or what do you think?

On the other hand, frontier math (pun intended) is much worse financed than biomedicine because most of the PhD-level math has barely any practical applications worth spending many manhours of high-IQ mathematicians (which often makes them switch career, you know). So, I would argue, if productivity of math postdocs when armed with future LLMs raises by, let's say, an order of magnitude, they will be able to attack more laborious problems.

Not that I expect it to make much difference to the general populace or even the scientific community at large though

I think the argument you’re making is that since LLMs can make eps > 0 progress, they can repeat it N times to make unbounded progress. But this is not the structure of conceptual insight as a general rule. Concretely, it fails for the architectural reasons I explained in the original post. 

The IMO Challenge Bet [LW · GW] was on a related topic, but not directly comparable to Bio Anchors. From MIRI's 2017 Updates and Strategy:

There’s no consensus among MIRI researchers on how long timelines are, and our aggregated estimate puts medium-to-high probability on scenarios in which the research community hasn’t developed AGI by, e.g., 2035. On average, however, research staff now assign moderately higher probability to AGI’s being developed before 2035 than we did a year or two ago.

I don't think the individual estimates that made up the aggregate were ever published. Perhaps someone at MIRI can help us out, it would help build a forecasting track record [LW · GW] for those involved.

For Yudkowsky in particular, I have a small collection of sources to hand. In Biology-Inspired AGI Timelines [LW · GW] (2021-12-01), he wrote:

But I suppose I cannot but acknowledge that my outward behavior seems to reveal a distribution whose median seems to fall well before 2050.

On Twitter (2022-12-02):

I could be wrong, but my guess is that we do not get AGI just by scaling ChatGPT, and that it takes surprisingly long from here. Parents conceiving today may have a fair chance of their child living to see kindergarten.

Also, in Shut it all down [LW · GW] (March 2023):

When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she’s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.

Yudkowsky also has a track record betting on Manifold that AI will wipe out humanity by 2030, at up to 40%.

Putting these together:

• 2021: median well before 2050
• 2022: "fair chance" when a 2023 baby goes to kindergarten (Sep 2028 or 2029)
• 2023: before a young child grows up (about 2035)
• 40% P(Doom by 2030)

So a median of 2029, with very wide credible intervals around both sides. This is just an estimate based on his outward behavior.

Would Yudkowsky describe this as "Yudkowsky's doctrine of AGI in 2029"?

Obviously it’s not a hard line, but your example doesn’t count, and proving any open conjecture in mathematics which was not constructed for the purpose does count. I think the quote from my post gives some other central examples. The standard is conceptual knowledge production.